diff options
Diffstat (limited to 'doc/administration/geo/replication')
9 files changed, 262 insertions, 172 deletions
diff --git a/doc/administration/geo/replication/container_registry.md b/doc/administration/geo/replication/container_registry.md new file mode 100644 index 00000000000..b425e5dcc0d --- /dev/null +++ b/doc/administration/geo/replication/container_registry.md @@ -0,0 +1,167 @@ +--- +stage: Systems +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: howto +--- + +# Container Registry for a secondary site **(PREMIUM SELF)** + +You can set up a Container Registry on your **secondary** Geo site that mirrors the one on the **primary** Geo site. + +## Supported container registries + +Geo supports the following types of container registries: + +- [Docker](https://docs.docker.com/registry/) +- [OCI](https://github.com/opencontainers/distribution-spec/blob/main/spec.md) + +## Supported image formats + +The following container image formats are support by Geo: + +- [Docker V2, schema 1](https://docs.docker.com/registry/spec/manifest-v2-1/) +- [Docker V2, schema 2](https://docs.docker.com/registry/spec/manifest-v2-2/) +- [OCI (Open Container Initiative)](https://github.com/opencontainers/image-spec) + +In addition, Geo also supports [BuildKit cache images](https://github.com/moby/buildkit). + +## Supported storage + +### Docker + +For more information on supported registry storage drivers see +[Docker registry storage drivers](https://docs.docker.com/registry/storage-drivers/) + +Read the [Load balancing considerations](https://docs.docker.com/registry/deploying/#load-balancing-considerations) +when deploying the Registry, and how to set up the storage driver for the GitLab integrated +[Container Registry](../../packages/container_registry.md#use-object-storage). + +### Registries that support OCI artifacts + +The following registries support OCI artifacts: + +- CNCF Distribution - local/offline verification +- Azure Container Registry (ACR) +- Amazon Elastic Container Registry (ECR) +- Google Artifact Registry (GAR) +- GitHub Packages container registry (GHCR) +- Bundle Bar + +For more information, see the [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec). + +## Configure Container Registry replication + +You can enable a storage-agnostic replication so it +can be used for cloud or local storage. Whenever a new image is pushed to the +**primary** site, each **secondary** site pulls it to its own container +repository. + +To configure Container Registry replication: + +1. Configure the [**primary** site](#configure-primary-site). +1. Configure the [**secondary** site](#configure-secondary-site). +1. Verify Container Registry [replication](#verify-replication). + +### Configure **primary** site + +Make sure that you have Container Registry set up and working on +the **primary** site before following the next steps. + +We need to make Container Registry send notification events to the +**primary** site. + +1. SSH into your GitLab **primary** server and login as root: + + ```shell + sudo -i + ``` + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + registry['notifications'] = [ + { + 'name' => 'geo_event', + 'url' => 'https://<example.com>/api/v4/container_registry_event/events', + 'timeout' => '500ms', + 'threshold' => 5, + 'backoff' => '1s', + 'headers' => { + 'Authorization' => ['<replace_with_a_secret_token>'] + } + } + ] + ``` + + NOTE: + Replace `<example.com>` with the `external_url` defined in your primary site's `/etc/gitlab/gitlab.rb` file, and + replace `<replace_with_a_secret_token>` with a case sensitive alphanumeric string + that starts with a letter. You can generate one with `< /dev/urandom tr -dc _A-Z-a-z-0-9 | head -c 32 | sed "s/^[0-9]*//"; echo` + + NOTE: + If you use an external Registry (not the one integrated with GitLab), you must add + these settings to its configuration yourself. In this case, you also have to specify + notification secret in `registry.notification_secret` section of + `/etc/gitlab/gitlab.rb` file. + + NOTE: + If you use GitLab HA, you also have to specify + the notification secret in `registry.notification_secret` section of + `/etc/gitlab/gitlab.rb` file for every web node. + +1. Reconfigure the **primary** node for the change to take effect: + + ```shell + gitlab-ctl reconfigure + ``` + +### Configure **secondary** site + +Make sure you have Container Registry set up and working on +the **secondary** site before following the next steps. + +The following steps should be done on each **secondary** site you're +expecting to see the container images replicated. + +Because we need to allow the **secondary** site to communicate securely with +the **primary** site Container Registry, we need to have a single key +pair for all the sites. The **secondary** site uses this key to +generate a short-lived JWT that is pull-only-capable to access the +**primary** site Container Registry. + +For each application and Sidekiq node on the **secondary** site: + +1. SSH into the node and login as the `root` user: + + ```shell + sudo -i + ``` + +1. Copy `/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key` from the **primary** to the node. + +1. Edit `/etc/gitlab/gitlab.rb` and add: + + ```ruby + gitlab_rails['geo_registry_replication_enabled'] = true + + # Primary registry's hostname and port, it will be used by + # the secondary node to directly communicate to primary registry + gitlab_rails['geo_registry_replication_primary_api_url'] = 'https://primary.example.com:5050/' + ``` + +1. Reconfigure the node for the change to take effect: + + ```shell + gitlab-ctl reconfigure + ``` + +### Verify replication + +To verify Container Registry replication is working, on the **secondary** site: + +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Geo > Nodes**. + The initial replication, or "backfill", is probably still in progress. + +You can monitor the synchronization process on each Geo site from the **primary** site's **Geo Nodes** dashboard in your browser. diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index acd27d5feed..40b71684bac 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -57,6 +57,8 @@ verification methods: | Blobs | Pipeline artifacts _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | Pages _(file system)_ | Geo with API | SHA256 checksum | | Blobs | Pages _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | CI Secure Files _(file system)_ | Geo with API | SHA256 checksum | +| Blobs | CI Secure Files _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | - (*1*): Redis replication can be used as part of HA with Redis sentinel. It's not used between Geo sites. - (*2*): Object storage replication can be performed by Geo or by your object storage provider/appliance @@ -185,8 +187,8 @@ successfully, you must replicate their data using some other means. |Feature | Replicated (added in GitLab version) | Verified (added in GitLab version) | GitLab-managed object storage replication (added in GitLab version) | GitLab-managed object storage verification (added in GitLab version) | Notes | |:--------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------|:---------------------------------------------------------------------------|:--------------------------------------------------------------------|:----------------------------------------------------------------|:------| |[Application data in PostgreSQL](../../postgresql/index.md) | **Yes** (10.2) | **Yes** (10.2) | N/A | N/A | | -|[Project repository](../../../user/project/repository/) | **Yes** (10.2) | **Yes** (10.7) | N/A | N/A | | -|[Project wiki repository](../../../user/project/wiki/) | **Yes** (10.2) | **Yes** (10.7) | N/A | N/A | | +|[Project repository](../../../user/project/repository/index.md) | **Yes** (10.2) | **Yes** (10.7) | N/A | N/A | | +|[Project wiki repository](../../../user/project/wiki/index.md) | **Yes** (10.2) | **Yes** (10.7) | N/A | N/A | | |[Group wiki repository](../../../user/project/wiki/group.md) | [**Yes** (13.10)](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | No | N/A | N/A | Behind feature flag `geo_group_wiki_repository_replication`, enabled by default. | |[Uploads](../../uploads.md) | **Yes** (10.2) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Replication is behind the feature flag `geo_upload_replication`, enabled by default. Verification was behind the feature flag `geo_upload_verification`, removed in 14.8. | |[LFS objects](../../lfs/index.md) | **Yes** (10.2) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696).<br /><br />Replication is behind the feature flag `geo_lfs_object_replication`, enabled by default. Verification was behind the feature flag `geo_lfs_object_verification`, removed in 14.7. | @@ -194,7 +196,8 @@ successfully, you must replicate their data using some other means. |[Project snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | N/A | N/A | | |[CI job artifacts](../../../ci/pipelines/job_artifacts.md) | **Yes** (10.4) | **Yes** (14.10) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Verification is behind the feature flag `geo_job_artifact_replication`, enabled by default in 14.10. | |[CI Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/pipeline_artifact.rb) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Persists additional artifacts after a pipeline completes. | -|[Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | No | No | Disabled by default. See [instructions](docker_registry.md) to enable. | +|[CI Secure Files](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/secure_file.rb) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [No](object_storage.md#verification-of-files-in-object-storage) | Verification is behind the feature flag `geo_ci_secure_file_replication`, enabled by default in 15.3. | +|[Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | No | No | Disabled by default. See [instructions](container_registry.md) to enable. | |[Infrastructure Registry](../../../user/packages/infrastructure_registry/index.md) | **Yes** (14.0) | **Yes** (14.0) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Behind feature flag `geo_package_file_replication`, enabled by default. | |[Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | N/A | N/A | Designs also require replication of LFS objects and Uploads. | |[Package Registry](../../../user/packages/package_registry/index.md) | **Yes** (13.2) | **Yes** (13.10) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Behind feature flag `geo_package_file_replication`, enabled by default. | @@ -202,9 +205,10 @@ successfully, you must replicate their data using some other means. |[External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Replication is behind the feature flag `geo_merge_request_diff_replication`, enabled by default. Verification was behind the feature flag `geo_merge_request_diff_verification`, removed in 14.7.| |[Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [**Yes** (14.2)](https://gitlab.com/groups/gitlab-org/-/epics/2810) | N/A | N/A | Verification was implemented behind the feature flag `geo_snippet_repository_verification` in 13.11, and the feature flag was removed in 14.2. | |[GitLab Pages](../../pages/index.md) | [**Yes** (14.3)](https://gitlab.com/groups/gitlab-org/-/epics/589) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Behind feature flag `geo_pages_deployment_replication`, enabled by default. Verification was behind the feature flag `geo_pages_deployment_verification`, removed in 14.7. | +|[Project-level Secure files](../../../ci/secure_files/index.md) | **Yes** (15.3) | **Yes** (15.3) | **Yes** (15.3) | [No](object_storage.md#verification-of-files-in-object-storage) | | |[Incident Metric Images](../../../operations/incident_management/incidents.md#metrics) | [Planned](https://gitlab.com/gitlab-org/gitlab/-/issues/362561) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/362561) | No | No | | |[Alert Metric Images](../../../operations/incident_management/alerts.md#metrics-tab) | [Planned](https://gitlab.com/gitlab-org/gitlab/-/issues/362564) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/362564) | No | No | | |[Server-side Git hooks](../../server_hooks.md) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/1867) | No | N/A | N/A | Not planned because of current implementation complexity, low customer interest, and availability of alternatives to hooks. | |[Elasticsearch integration](../../../integration/elasticsearch.md) | [Not planned](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | No | Not planned because further product discovery is required and Elasticsearch (ES) clusters can be rebuilt. Secondaries use the same ES cluster as the primary. | |[Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [Not planned](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | No | Blocked by [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Replication of this cache is not needed for disaster recovery purposes because it can be recreated from external sources. | -|[Vulnerability Export](../../../user/application_security/vulnerability_report/#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | No | No | Not planned because they are ephemeral and sensitive information. They can be regenerated on demand. | +|[Vulnerability Export](../../../user/application_security/vulnerability_report/index.md#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | No | No | Not planned because they are ephemeral and sensitive information. They can be regenerated on demand. | diff --git a/doc/administration/geo/replication/disable_geo.md b/doc/administration/geo/replication/disable_geo.md index 5d4ae12c990..f0658ae45a2 100644 --- a/doc/administration/geo/replication/disable_geo.md +++ b/doc/administration/geo/replication/disable_geo.md @@ -28,7 +28,7 @@ To disable Geo, you need to first remove all your secondary Geo sites, which mea anymore on these sites. You can follow our docs to [remove your secondary Geo sites](remove_geo_site.md). If the current site that you want to keep using is a secondary site, you need to first promote it to primary. -You can use our steps on [how to promote a secondary site](../disaster_recovery/#step-3-promoting-a-secondary-site) +You can use our steps on [how to promote a secondary site](../disaster_recovery/index.md#step-3-promoting-a-secondary-site) to do that. ## Remove the primary site from the UI diff --git a/doc/administration/geo/replication/docker_registry.md b/doc/administration/geo/replication/docker_registry.md index 855e33d9a51..d0af6f2a66f 100644 --- a/doc/administration/geo/replication/docker_registry.md +++ b/doc/administration/geo/replication/docker_registry.md @@ -1,137 +1,11 @@ --- -stage: Systems -group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: howto +redirect_to: 'container_registry.md' +remove_date: '2022-10-29' --- -# Docker Registry for a secondary site **(PREMIUM SELF)** +This document was moved to [another location](container_registry.md). -You can set up a [Docker Registry](https://docs.docker.com/registry/) on your -**secondary** Geo site that mirrors the one on the **primary** Geo site. - -## Storage support - -Docker Registry currently supports a few types of storage. If you choose a -distributed storage (`azure`, `gcs`, `s3`, `swift`, or `oss`) for your Docker -Registry on the **primary** site, you can use the same storage for a **secondary** -Docker Registry as well. For more information, read the -[Load balancing considerations](https://docs.docker.com/registry/deploying/#load-balancing-considerations) -when deploying the Registry, and how to set up the storage driver for the GitLab -integrated [Container Registry](../../packages/container_registry.md#use-object-storage). - -## Replicating Docker Registry - -You can enable a storage-agnostic replication so it -can be used for cloud or local storage. Whenever a new image is pushed to the -**primary** site, each **secondary** site pulls it to its own container -repository. - -To configure Docker Registry replication: - -1. Configure the [**primary** site](#configure-primary-site). -1. Configure the [**secondary** site](#configure-secondary-site). -1. Verify Docker Registry [replication](#verify-replication). - -### Configure **primary** site - -Make sure that you have Container Registry set up and working on -the **primary** site before following the next steps. - -We need to make Docker Registry send notification events to the -**primary** site. - -1. SSH into your GitLab **primary** server and login as root: - - ```shell - sudo -i - ``` - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - registry['notifications'] = [ - { - 'name' => 'geo_event', - 'url' => 'https://<example.com>/api/v4/container_registry_event/events', - 'timeout' => '500ms', - 'threshold' => 5, - 'backoff' => '1s', - 'headers' => { - 'Authorization' => ['<replace_with_a_secret_token>'] - } - } - ] - ``` - - NOTE: - Replace `<example.com>` with the `external_url` defined in your primary site's `/etc/gitlab/gitlab.rb` file, and - replace `<replace_with_a_secret_token>` with a case sensitive alphanumeric string - that starts with a letter. You can generate one with `< /dev/urandom tr -dc _A-Z-a-z-0-9 | head -c 32 | sed "s/^[0-9]*//"; echo` - - NOTE: - If you use an external Registry (not the one integrated with GitLab), you must add - these settings to its configuration yourself. In this case, you also have to specify - notification secret in `registry.notification_secret` section of - `/etc/gitlab/gitlab.rb` file. - - NOTE: - If you use GitLab HA, you also have to specify - the notification secret in `registry.notification_secret` section of - `/etc/gitlab/gitlab.rb` file for every web node. - -1. Reconfigure the **primary** node for the change to take effect: - - ```shell - gitlab-ctl reconfigure - ``` - -### Configure **secondary** site - -Make sure you have Container Registry set up and working on -the **secondary** site before following the next steps. - -The following steps should be done on each **secondary** site you're -expecting to see the Docker images replicated. - -Because we need to allow the **secondary** site to communicate securely with -the **primary** site Container Registry, we need to have a single key -pair for all the sites. The **secondary** site uses this key to -generate a short-lived JWT that is pull-only-capable to access the -**primary** site Container Registry. - -For each application and Sidekiq node on the **secondary** site: - -1. SSH into the node and login as the `root` user: - - ```shell - sudo -i - ``` - -1. Copy `/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key` from the **primary** to the node. - -1. Edit `/etc/gitlab/gitlab.rb` and add: - - ```ruby - gitlab_rails['geo_registry_replication_enabled'] = true - - # Primary registry's hostname and port, it will be used by - # the secondary node to directly communicate to primary registry - gitlab_rails['geo_registry_replication_primary_api_url'] = 'https://primary.example.com:5050/' - ``` - -1. Reconfigure the node for the change to take effect: - - ```shell - gitlab-ctl reconfigure - ``` - -### Verify replication - -To verify Container Registry replication is working, on the **secondary** site: - -1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **Geo > Nodes**. - The initial replication, or "backfill", is probably still in progress. - -You can monitor the synchronization process on each Geo site from the **primary** site's **Geo Nodes** dashboard in your browser. +<!-- This redirect file can be deleted after <2022-10-29>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
\ No newline at end of file diff --git a/doc/administration/geo/replication/faq.md b/doc/administration/geo/replication/faq.md index bdf1771e8a8..7a67af1cfa2 100644 --- a/doc/administration/geo/replication/faq.md +++ b/doc/administration/geo/replication/faq.md @@ -65,10 +65,10 @@ connectivity between your sites, your hardware, and so on. That's totally fine. We use HTTP(s) to fetch repository changes from the **primary** site to all **secondary** sites. -## Is this possible to set up a Docker Registry for a **secondary** site that mirrors the one on the **primary** site? +## Is this possible to set up a Container Registry for a **secondary** site that mirrors the one on the **primary** site? -Yes. See [Docker Registry for a **secondary** site](docker_registry.md). +Yes. See [Container Registry for a **secondary** site](container_registry.md). -## Can you login to a secondary site? +## Can you log in to a secondary site? Yes, but secondary sites receive all authentication data (like user accounts and logins) from the primary instance. This means you are re-directed to the primary for authentication and then routed back. diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 082ecbbb208..26d192f62cd 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -129,7 +129,7 @@ http://secondary.example.com/ ``` To find more details about failed items, check -[the `gitlab-rails/geo.log` file](../../troubleshooting/log_parsing.md#find-most-common-geo-sync-errors) +[the `gitlab-rails/geo.log` file](../../logs/log_parsing.md#find-most-common-geo-sync-errors) ### Check if PostgreSQL replication is working @@ -191,7 +191,7 @@ If a replication slot is inactive, the `pg_wal` logs corresponding to the slot are reserved forever (or until the slot is active again). This causes continuous disk usage growth and the following messages appear repeatedly in the -[PostgreSQL logs](../../logs.md#postgresql-logs): +[PostgreSQL logs](../../logs/index.md#postgresql-logs): ```plaintext WARNING: oldest xmin is far in the past @@ -331,8 +331,7 @@ Be sure to restart PostgreSQL for this to take effect. See the This occurs when PostgreSQL does not have a replication slot for the **secondary** node by that name. -You may want to rerun the [replication -process](../setup/database.md) on the **secondary** node . +You may want to rerun the [replication process](../setup/database.md) on the **secondary** node . ### Message: "Command exceeded allowed execution time" when setting up replication? @@ -376,7 +375,7 @@ log data to build up in `pg_xlog`. Removing the unused slots can reduce the amou Slots where `active` is `f` are not active. - When this slot should be active, because you have a **secondary** node configured using that slot, - sign in to that **secondary** node and check the [PostgreSQL logs](../../logs.md#postgresql-logs) + sign in to that **secondary** node and check the [PostgreSQL logs](../../logs/index.md#postgresql-logs) to view why the replication is not running. - If you are no longer using the slot (for example, you no longer have Geo enabled), you can remove it with in the @@ -510,7 +509,7 @@ To solve this: 1. Back up [the `.git` folder](../../repository_storage_types.md#translate-hashed-storage-paths). -1. Optional. [Spot-check](../../troubleshooting/log_parsing.md#find-all-projects-affected-by-a-fatal-git-problem) +1. Optional. [Spot-check](../../logs/log_parsing.md#find-all-projects-affected-by-a-fatal-git-problem) a few of those IDs whether they indeed correspond to a project with known Geo replication failures. Use `fatal: 'geo'` as the `grep` term and the following API call: @@ -597,7 +596,7 @@ to start again from scratch, there are a few steps that can help you: gitlab-ctl stop geo-logcursor ``` - You can watch the [Sidekiq logs](../../logs.md#sidekiq-logs) to know when Sidekiq jobs processing has finished: + You can watch the [Sidekiq logs](../../logs/index.md#sidekiq-logs) to know when Sidekiq jobs processing has finished: ```shell gitlab-ctl tail sidekiq @@ -837,7 +836,7 @@ to transfer each affected repository from the primary to the secondary site. The following are possible error messages that might be encountered during failover or when promoting a secondary to a primary node with strategies to resolve them. -### Message: ActiveRecord::RecordInvalid: Validation failed: Name has already been taken +### Message: `ActiveRecord::RecordInvalid: Validation failed: Name has already been taken` When [promoting a **secondary** site](../disaster_recovery/index.md#step-3-promoting-a-secondary-site), you might encounter the following error message: @@ -869,11 +868,10 @@ or `gitlab-ctl promote-to-primary-node`, either: ``` - Upgrade to GitLab 12.6.3 or later if it is safe to do so. For example, - if the failover was just a test. A [caching-related - bug](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22021) was - fixed. + if the failover was just a test. A + [caching-related bug](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22021) was fixed. -### Message: ActiveRecord::RecordInvalid: Validation failed: Enabled Geo primary node cannot be disabled +### Message: `ActiveRecord::RecordInvalid: Validation failed: Enabled Geo primary node cannot be disabled` If you disabled a secondary node, either with the [replication pause task](../index.md#pausing-and-resuming-replication) (GitLab 13.2) or by using the user interface (GitLab 13.1 and earlier), you must first @@ -1127,12 +1125,6 @@ Geo secondary sites continue to replicate and verify data, and the secondary sit This bug was [fixed in GitLab 14.4](https://gitlab.com/gitlab-org/gitlab/-/issues/292983). -### GitLab Pages return 404 errors after promoting - -This is due to [Pages data not being managed by Geo](datatypes.md#limitations-on-replicationverification). -Find advice to resolve those error messages in the -[Pages administration documentation](../../../administration/pages/index.md#404-error-after-promoting-a-geo-secondary-to-a-primary-node). - ### Primary site returns 500 error when accessing `/admin/geo/replication/projects` Navigating to **Admin > Geo > Replication** (or `/admin/geo/replication/projects`) on a primary Geo site, shows a 500 error, while that same link on the secondary works fine. The primary's `production.log` has a similar entry to the following: @@ -1146,7 +1138,28 @@ Geo::TrackingBase::SecondaryNotConfigured: Geo secondary database is not configu On a Geo primary site this error can be ignored. -This happens because GitLab is attempting to display registries from the [Geo tracking database](../../../administration/geo/#geo-tracking-database) which doesn't exist on the primary site (only the original projects exist on the primary; no replicated projects are present, therefore no tracking database exists). +This happens because GitLab is attempting to display registries from the [Geo tracking database](../../../administration/geo/index.md#geo-tracking-database) which doesn't exist on the primary site (only the original projects exist on the primary; no replicated projects are present, therefore no tracking database exists). + +### Secondary site returns 400 error "Request header or cookie too large" + +This error can happen when the internal URL of the primary site is incorrect. + +For example, when you use a unified URL and the primary site's internal URL is also equal to the external URL. This causes a loop when a secondary site proxies requests to the primary site's internal URL. + +To fix this issue, set the primary site's internal URL to a URL that is: + +- Unique to the primary site. +- Accessible from all secondary sites. + +1. Enter the [Rails console](../../operations/rails_console.md) on the primary site. + +1. Run the following, replacing `https://unique.url.for.primary.site` with your specific internal URL. + For example, depending on your network configuration, you could use an IP address, like + `http://1.2.3.4`. + + ```ruby + GeoNode.where(primary: true).first.update!(internal_url: "https://unique.url.for.primary.site") + ``` ## Fixing client errors @@ -1158,6 +1171,23 @@ requests redirected from the secondary to the primary node do not properly send Authorization header. This may result in either an infinite `Authorization <-> Redirect` loop, or Authorization error messages. +### Error: Net::ReadTimeout when pushing through SSH on a Geo secondary + +When you push large repositories through SSH on a Geo secondary site, you may encounter a timeout. +This is because Rails proxies the push to the primary and has a 60 second default timeout, +[as described in this Geo issue](https://gitlab.com/gitlab-org/gitlab/-/issues/7405). + +Current workarounds are: + +- Push through HTTP instead, where Workhorse proxies the request to the primary (or redirects to the primary if Geo proxying is not enabled). +- Push directly to the primary. + +Example log (`gitlab-shell.log`): + +```plaintext +Failed to contact primary https://primary.domain.com/namespace/push_test.git\\nError: Net::ReadTimeout\",\"result\":null}" code=500 method=POST pid=5483 url="http://127.0.0.1:3000/api/v4/geo/proxy_git_push_ssh/push" +``` + ## Recovering from a partial failover The partial failover to a secondary Geo *site* may be the result of a temporary/transient issue. Therefore, first attempt to run the promote command again. diff --git a/doc/administration/geo/replication/tuning.md b/doc/administration/geo/replication/tuning.md index 670459624f3..755ab45a76c 100644 --- a/doc/administration/geo/replication/tuning.md +++ b/doc/administration/geo/replication/tuning.md @@ -30,7 +30,7 @@ However, this may not lead to more downloads in parallel unless the number of available Sidekiq threads is also increased. For example, if repository synchronization concurrency is increased from 25 to 50, you may also want to increase the number of Sidekiq threads from 25 to 50. See the -[Sidekiq concurrency documentation](../../operations/extra_sidekiq_processes.md#number-of-threads) +[Sidekiq concurrency documentation](../../sidekiq/extra_sidekiq_processes.md#number-of-threads) for more details. ## Repository re-verification diff --git a/doc/administration/geo/replication/upgrading_the_geo_sites.md b/doc/administration/geo/replication/upgrading_the_geo_sites.md index 30961de0381..ce1ff4fe6a5 100644 --- a/doc/administration/geo/replication/upgrading_the_geo_sites.md +++ b/doc/administration/geo/replication/upgrading_the_geo_sites.md @@ -28,12 +28,23 @@ and cause downtime. If you want to avoid downtime, consider using To upgrade the Geo sites when a new GitLab version is released, upgrade **primary** and all **secondary** sites: -1. **Optional:** [Pause replication on each **secondary** sites.](../index.md#pausing-and-resuming-replication) +1. Optional. [Pause replication on each **secondary** site](../index.md#pausing-and-resuming-replication) + to protect the disaster recovery (DR) capability of the **secondary** sites. 1. SSH into each node of the **primary** site. 1. [Upgrade GitLab on the **primary** site](../../../update/package/index.md#upgrade-using-the-official-repositories). +1. Perform testing on the **primary** site, particularly if you paused replication in step 1 to protect DR. [There are some suggestions for post-upgrade testing](../../../update/plan_your_upgrade.md#pre-upgrade-and-post-upgrade-checks) in the upgrade documentation. 1. SSH into each node of **secondary** sites. 1. [Upgrade GitLab on each **secondary** site](../../../update/package/index.md#upgrade-using-the-official-repositories). -1. If you paused replication in step 1, [resume replication on each **secondary**](../index.md#pausing-and-resuming-replication) +1. If you paused replication in step 1, [resume replication on each **secondary**](../index.md#pausing-and-resuming-replication). + Then, restart Puma and Sidekiq on each **secondary** site. This is to ensure they + are initialized against the newer database schema that is now replicated from + the previously upgraded **primary** site. + + ```shell + sudo gitlab-ctl restart sidekiq + sudo gitlab-ctl restart puma + ``` + 1. [Test](#check-status-after-upgrading) **primary** and **secondary** sites, and check version in each. ### Check status after upgrading diff --git a/doc/administration/geo/replication/version_specific_upgrades.md b/doc/administration/geo/replication/version_specific_upgrades.md index f0925bdf87e..350310c7076 100644 --- a/doc/administration/geo/replication/version_specific_upgrades.md +++ b/doc/administration/geo/replication/version_specific_upgrades.md @@ -178,11 +178,15 @@ GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab ## Upgrading to GitLab 13.7 -We've detected an issue with the `FetchRemove` call used by Geo secondaries. -This causes performance issues as we execute reference transaction hooks for -each upgraded reference. Delay any upgrade attempts until this is in the -[13.7.5 patch release.](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3002). -More details are available [in this issue](https://gitlab.com/gitlab-org/git/-/issues/79). +- We've detected an issue with the `FetchRemove` call used by Geo secondaries. + This causes performance issues as we execute reference transaction hooks for + each upgraded reference. Delay any upgrade attempts until this is in the + [13.7.5 patch release.](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3002). + More details are available [in this issue](https://gitlab.com/gitlab-org/git/-/issues/79). +- A new secret is generated in `/etc/gitlab/gitlab-secrets.json`. + In an HA GitLab or GitLab Geo environment, secrets need to be the same on all nodes. + Ensure this new secret is also accounted for if you are manually syncing the file across + nodes, or manually specifying secrets in `/etc/gitlab/gitlab.rb`. ## Upgrading to GitLab 13.5 @@ -243,8 +247,8 @@ the recommended procedure, see the ## Upgrading to GitLab 12.9 WARNING: -GitLab 12.9.0 through GitLab 12.9.3 are affected by [a bug that stops -repository verification](https://gitlab.com/gitlab-org/gitlab/-/issues/213523). +GitLab 12.9.0 through GitLab 12.9.3 are affected by +[a bug that stops repository verification](https://gitlab.com/gitlab-org/gitlab/-/issues/213523). The issue is fixed in GitLab 12.9.4. Upgrade to GitLab 12.9.4 or later. By default, GitLab 12.9 attempts to upgrade the embedded PostgreSQL server @@ -397,6 +401,6 @@ For the recommended procedure, see the ## Upgrading to GitLab 12.0 WARNING: -This version is affected by a [bug that results in new LFS objects not being -replicated to Geo secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). +This version is affected by a +[bug that results in new LFS objects not being replicated to Geo secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). The issue is fixed in GitLab 12.1. Be sure to upgrade to GitLab 12.1 or later. |