diff options
Diffstat (limited to 'doc/administration/geo')
| -rw-r--r-- | doc/administration/geo/disaster_recovery/bring_primary_back.md | 4 | ||||
| -rw-r--r-- | doc/administration/geo/disaster_recovery/index.md | 4 | ||||
| -rw-r--r-- | doc/administration/geo/disaster_recovery/planned_failover.md | 2 | ||||
| -rw-r--r-- | doc/administration/geo/index.md | 6 | ||||
| -rw-r--r-- | doc/administration/geo/replication/container_registry.md | 167 | ||||
| -rw-r--r-- | doc/administration/geo/replication/datatypes.md | 12 | ||||
| -rw-r--r-- | doc/administration/geo/replication/disable_geo.md | 2 | ||||
| -rw-r--r-- | doc/administration/geo/replication/docker_registry.md | 140 | ||||
| -rw-r--r-- | doc/administration/geo/replication/faq.md | 6 | ||||
| -rw-r--r-- | doc/administration/geo/replication/troubleshooting.md | 68 | ||||
| -rw-r--r-- | doc/administration/geo/replication/tuning.md | 2 | ||||
| -rw-r--r-- | doc/administration/geo/replication/upgrading_the_geo_sites.md | 15 | ||||
| -rw-r--r-- | doc/administration/geo/replication/version_specific_upgrades.md | 22 | ||||
| -rw-r--r-- | doc/administration/geo/secondary_proxy/index.md | 5 | ||||
| -rw-r--r-- | doc/administration/geo/setup/database.md | 224 | ||||
| -rw-r--r-- | doc/administration/geo/setup/index.md | 2 |
16 files changed, 385 insertions, 296 deletions
diff --git a/doc/administration/geo/disaster_recovery/bring_primary_back.md b/doc/administration/geo/disaster_recovery/bring_primary_back.md index 833b9a877e9..a2d4f35a7c3 100644 --- a/doc/administration/geo/disaster_recovery/bring_primary_back.md +++ b/doc/administration/geo/disaster_recovery/bring_primary_back.md @@ -41,8 +41,8 @@ To bring the former **primary** site up to date: NOTE: If you [changed the DNS records](index.md#step-4-optional-updating-the-primary-domain-dns-record) - for this site during disaster recovery procedure you may need to [block - all the writes to this site](planned_failover.md#prevent-updates-to-the-primary-site) + for this site during disaster recovery procedure you may need to + [block all the writes to this site](planned_failover.md#prevent-updates-to-the-primary-site) during this procedure. 1. [Set up database replication](../setup/database.md). In this case, the **secondary** site diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index f457cb4b0a2..0cae94fcec1 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -572,7 +572,7 @@ and after that you also need two extra steps. postgresql['md5_auth_cidr_addresses'] = ['<primary_site_ip>/32', '<secondary_site_ip>/32'] # Every secondary site needs to have its own slot so specify the number of secondary sites you're going to have - postgresql['max_replication_slots'] = 1 + # postgresql['max_replication_slots'] = 1 # Set this to be the number of Geo secondary nodes if you have more than one ## ## Disable automatic database migrations temporarily @@ -581,7 +581,7 @@ and after that you also need two extra steps. gitlab_rails['auto_migrate'] = false ``` - (For more details about these settings you can read [Configure the primary server](../setup/database.md#step-1-configure-the-primary-server)) + (For more details about these settings you can read [Configure the primary server](../setup/database.md#step-1-configure-the-primary-site)) 1. Save the file and reconfigure GitLab for the database listen changes and the replication slot changes to be applied: diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index 5a5d896c20a..6c3353e7d7e 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -45,7 +45,7 @@ be adapted for use with any other file-based data, such as [GitLab Pages](../../ ### Container registry By default, the container registry is not automatically replicated to secondary -sites and this needs to be manually configured, see [Docker Registry for a secondary site](../replication/docker_registry.md). +sites and this needs to be manually configured, see [Container Registry for a secondary site](../replication/container_registry.md). If you are using local storage on your current primary site for the container registry, you can `rsync` the container registry objects to the secondary diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index cf7d2649142..e3bf5ccdfe7 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -119,8 +119,8 @@ The following are required to run Geo: The following operating systems are known to ship with a current version of OpenSSH: - [CentOS](https://www.centos.org) 7.4 or later - [Ubuntu](https://ubuntu.com) 16.04 or later -- PostgreSQL 12 with [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication) - - PostgreSQL 13 is not supported for Geo, see [epic 3832](https://gitlab.com/groups/gitlab-org/-/epics/3832) +- PostgreSQL 12 or 13 with [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication) + - Note,[PostgreSQL 12 is deprecated](../../update/deprecations.md#postgresql-12-deprecated) and will be removed in GitLab 16.0. - Git 2.9 or later - Git-lfs 2.4.2 or later on the user side when using LFS - All sites must run [the same GitLab and PostgreSQL versions](setup/database.md#postgresql-replication). @@ -283,7 +283,7 @@ For information on using Geo in disaster recovery situations to mitigate data-lo ### Replicating the Container Registry -For more information on how to replicate the Container Registry, see [Docker Registry for a **secondary** site](replication/docker_registry.md). +For more information on how to replicate the Container Registry, see [Container Registry for a **secondary** site](replication/container_registry.md). ### Geo secondary proxy diff --git a/doc/administration/geo/replication/container_registry.md b/doc/administration/geo/replication/container_registry.md new file mode 100644 index 00000000000..b425e5dcc0d --- /dev/null +++ b/doc/administration/geo/replication/container_registry.md @@ -0,0 +1,167 @@ +--- +stage: Systems +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: howto +--- + +# Container Registry for a secondary site **(PREMIUM SELF)** + +You can set up a Container Registry on your **secondary** Geo site that mirrors the one on the **primary** Geo site. + +## Supported container registries + +Geo supports the following types of container registries: + +- [Docker](https://docs.docker.com/registry/) +- [OCI](https://github.com/opencontainers/distribution-spec/blob/main/spec.md) + +## Supported image formats + +The following container image formats are support by Geo: + +- [Docker V2, schema 1](https://docs.docker.com/registry/spec/manifest-v2-1/) +- [Docker V2, schema 2](https://docs.docker.com/registry/spec/manifest-v2-2/) +- [OCI (Open Container Initiative)](https://github.com/opencontainers/image-spec) + +In addition, Geo also supports [BuildKit cache images](https://github.com/moby/buildkit). + +## Supported storage + +### Docker + +For more information on supported registry storage drivers see +[Docker registry storage drivers](https://docs.docker.com/registry/storage-drivers/) + +Read the [Load balancing considerations](https://docs.docker.com/registry/deploying/#load-balancing-considerations) +when deploying the Registry, and how to set up the storage driver for the GitLab integrated +[Container Registry](../../packages/container_registry.md#use-object-storage). + +### Registries that support OCI artifacts + +The following registries support OCI artifacts: + +- CNCF Distribution - local/offline verification +- Azure Container Registry (ACR) +- Amazon Elastic Container Registry (ECR) +- Google Artifact Registry (GAR) +- GitHub Packages container registry (GHCR) +- Bundle Bar + +For more information, see the [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec). + +## Configure Container Registry replication + +You can enable a storage-agnostic replication so it +can be used for cloud or local storage. Whenever a new image is pushed to the +**primary** site, each **secondary** site pulls it to its own container +repository. + +To configure Container Registry replication: + +1. Configure the [**primary** site](#configure-primary-site). +1. Configure the [**secondary** site](#configure-secondary-site). +1. Verify Container Registry [replication](#verify-replication). + +### Configure **primary** site + +Make sure that you have Container Registry set up and working on +the **primary** site before following the next steps. + +We need to make Container Registry send notification events to the +**primary** site. + +1. SSH into your GitLab **primary** server and login as root: + + ```shell + sudo -i + ``` + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + registry['notifications'] = [ + { + 'name' => 'geo_event', + 'url' => 'https://<example.com>/api/v4/container_registry_event/events', + 'timeout' => '500ms', + 'threshold' => 5, + 'backoff' => '1s', + 'headers' => { + 'Authorization' => ['<replace_with_a_secret_token>'] + } + } + ] + ``` + + NOTE: + Replace `<example.com>` with the `external_url` defined in your primary site's `/etc/gitlab/gitlab.rb` file, and + replace `<replace_with_a_secret_token>` with a case sensitive alphanumeric string + that starts with a letter. You can generate one with `< /dev/urandom tr -dc _A-Z-a-z-0-9 | head -c 32 | sed "s/^[0-9]*//"; echo` + + NOTE: + If you use an external Registry (not the one integrated with GitLab), you must add + these settings to its configuration yourself. In this case, you also have to specify + notification secret in `registry.notification_secret` section of + `/etc/gitlab/gitlab.rb` file. + + NOTE: + If you use GitLab HA, you also have to specify + the notification secret in `registry.notification_secret` section of + `/etc/gitlab/gitlab.rb` file for every web node. + +1. Reconfigure the **primary** node for the change to take effect: + + ```shell + gitlab-ctl reconfigure + ``` + +### Configure **secondary** site + +Make sure you have Container Registry set up and working on +the **secondary** site before following the next steps. + +The following steps should be done on each **secondary** site you're +expecting to see the container images replicated. + +Because we need to allow the **secondary** site to communicate securely with +the **primary** site Container Registry, we need to have a single key +pair for all the sites. The **secondary** site uses this key to +generate a short-lived JWT that is pull-only-capable to access the +**primary** site Container Registry. + +For each application and Sidekiq node on the **secondary** site: + +1. SSH into the node and login as the `root` user: + + ```shell + sudo -i + ``` + +1. Copy `/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key` from the **primary** to the node. + +1. Edit `/etc/gitlab/gitlab.rb` and add: + + ```ruby + gitlab_rails['geo_registry_replication_enabled'] = true + + # Primary registry's hostname and port, it will be used by + # the secondary node to directly communicate to primary registry + gitlab_rails['geo_registry_replication_primary_api_url'] = 'https://primary.example.com:5050/' + ``` + +1. Reconfigure the node for the change to take effect: + + ```shell + gitlab-ctl reconfigure + ``` + +### Verify replication + +To verify Container Registry replication is working, on the **secondary** site: + +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Geo > Nodes**. + The initial replication, or "backfill", is probably still in progress. + +You can monitor the synchronization process on each Geo site from the **primary** site's **Geo Nodes** dashboard in your browser. diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index acd27d5feed..40b71684bac 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -57,6 +57,8 @@ verification methods: | Blobs | Pipeline artifacts _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | Pages _(file system)_ | Geo with API | SHA256 checksum | | Blobs | Pages _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | CI Secure Files _(file system)_ | Geo with API | SHA256 checksum | +| Blobs | CI Secure Files _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | - (*1*): Redis replication can be used as part of HA with Redis sentinel. It's not used between Geo sites. - (*2*): Object storage replication can be performed by Geo or by your object storage provider/appliance @@ -185,8 +187,8 @@ successfully, you must replicate their data using some other means. |Feature | Replicated (added in GitLab version) | Verified (added in GitLab version) | GitLab-managed object storage replication (added in GitLab version) | GitLab-managed object storage verification (added in GitLab version) | Notes | |:--------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------|:---------------------------------------------------------------------------|:--------------------------------------------------------------------|:----------------------------------------------------------------|:------| |[Application data in PostgreSQL](../../postgresql/index.md) | **Yes** (10.2) | **Yes** (10.2) | N/A | N/A | | -|[Project repository](../../../user/project/repository/) | **Yes** (10.2) | **Yes** (10.7) | N/A | N/A | | -|[Project wiki repository](../../../user/project/wiki/) | **Yes** (10.2) | **Yes** (10.7) | N/A | N/A | | +|[Project repository](../../../user/project/repository/index.md) | **Yes** (10.2) | **Yes** (10.7) | N/A | N/A | | +|[Project wiki repository](../../../user/project/wiki/index.md) | **Yes** (10.2) | **Yes** (10.7) | N/A | N/A | | |[Group wiki repository](../../../user/project/wiki/group.md) | [**Yes** (13.10)](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | No | N/A | N/A | Behind feature flag `geo_group_wiki_repository_replication`, enabled by default. | |[Uploads](../../uploads.md) | **Yes** (10.2) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Replication is behind the feature flag `geo_upload_replication`, enabled by default. Verification was behind the feature flag `geo_upload_verification`, removed in 14.8. | |[LFS objects](../../lfs/index.md) | **Yes** (10.2) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696).<br /><br />Replication is behind the feature flag `geo_lfs_object_replication`, enabled by default. Verification was behind the feature flag `geo_lfs_object_verification`, removed in 14.7. | @@ -194,7 +196,8 @@ successfully, you must replicate their data using some other means. |[Project snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | N/A | N/A | | |[CI job artifacts](../../../ci/pipelines/job_artifacts.md) | **Yes** (10.4) | **Yes** (14.10) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Verification is behind the feature flag `geo_job_artifact_replication`, enabled by default in 14.10. | |[CI Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/pipeline_artifact.rb) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Persists additional artifacts after a pipeline completes. | -|[Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | No | No | Disabled by default. See [instructions](docker_registry.md) to enable. | +|[CI Secure Files](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/secure_file.rb) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [No](object_storage.md#verification-of-files-in-object-storage) | Verification is behind the feature flag `geo_ci_secure_file_replication`, enabled by default in 15.3. | +|[Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | No | No | Disabled by default. See [instructions](container_registry.md) to enable. | |[Infrastructure Registry](../../../user/packages/infrastructure_registry/index.md) | **Yes** (14.0) | **Yes** (14.0) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Behind feature flag `geo_package_file_replication`, enabled by default. | |[Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | N/A | N/A | Designs also require replication of LFS objects and Uploads. | |[Package Registry](../../../user/packages/package_registry/index.md) | **Yes** (13.2) | **Yes** (13.10) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Behind feature flag `geo_package_file_replication`, enabled by default. | @@ -202,9 +205,10 @@ successfully, you must replicate their data using some other means. |[External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Replication is behind the feature flag `geo_merge_request_diff_replication`, enabled by default. Verification was behind the feature flag `geo_merge_request_diff_verification`, removed in 14.7.| |[Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [**Yes** (14.2)](https://gitlab.com/groups/gitlab-org/-/epics/2810) | N/A | N/A | Verification was implemented behind the feature flag `geo_snippet_repository_verification` in 13.11, and the feature flag was removed in 14.2. | |[GitLab Pages](../../pages/index.md) | [**Yes** (14.3)](https://gitlab.com/groups/gitlab-org/-/epics/589) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Behind feature flag `geo_pages_deployment_replication`, enabled by default. Verification was behind the feature flag `geo_pages_deployment_verification`, removed in 14.7. | +|[Project-level Secure files](../../../ci/secure_files/index.md) | **Yes** (15.3) | **Yes** (15.3) | **Yes** (15.3) | [No](object_storage.md#verification-of-files-in-object-storage) | | |[Incident Metric Images](../../../operations/incident_management/incidents.md#metrics) | [Planned](https://gitlab.com/gitlab-org/gitlab/-/issues/362561) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/362561) | No | No | | |[Alert Metric Images](../../../operations/incident_management/alerts.md#metrics-tab) | [Planned](https://gitlab.com/gitlab-org/gitlab/-/issues/362564) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/362564) | No | No | | |[Server-side Git hooks](../../server_hooks.md) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/1867) | No | N/A | N/A | Not planned because of current implementation complexity, low customer interest, and availability of alternatives to hooks. | |[Elasticsearch integration](../../../integration/elasticsearch.md) | [Not planned](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | No | Not planned because further product discovery is required and Elasticsearch (ES) clusters can be rebuilt. Secondaries use the same ES cluster as the primary. | |[Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [Not planned](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | No | Blocked by [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Replication of this cache is not needed for disaster recovery purposes because it can be recreated from external sources. | -|[Vulnerability Export](../../../user/application_security/vulnerability_report/#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | No | No | Not planned because they are ephemeral and sensitive information. They can be regenerated on demand. | +|[Vulnerability Export](../../../user/application_security/vulnerability_report/index.md#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | No | No | Not planned because they are ephemeral and sensitive information. They can be regenerated on demand. | diff --git a/doc/administration/geo/replication/disable_geo.md b/doc/administration/geo/replication/disable_geo.md index 5d4ae12c990..f0658ae45a2 100644 --- a/doc/administration/geo/replication/disable_geo.md +++ b/doc/administration/geo/replication/disable_geo.md @@ -28,7 +28,7 @@ To disable Geo, you need to first remove all your secondary Geo sites, which mea anymore on these sites. You can follow our docs to [remove your secondary Geo sites](remove_geo_site.md). If the current site that you want to keep using is a secondary site, you need to first promote it to primary. -You can use our steps on [how to promote a secondary site](../disaster_recovery/#step-3-promoting-a-secondary-site) +You can use our steps on [how to promote a secondary site](../disaster_recovery/index.md#step-3-promoting-a-secondary-site) to do that. ## Remove the primary site from the UI diff --git a/doc/administration/geo/replication/docker_registry.md b/doc/administration/geo/replication/docker_registry.md index 855e33d9a51..d0af6f2a66f 100644 --- a/doc/administration/geo/replication/docker_registry.md +++ b/doc/administration/geo/replication/docker_registry.md @@ -1,137 +1,11 @@ --- -stage: Systems -group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: howto +redirect_to: 'container_registry.md' +remove_date: '2022-10-29' --- -# Docker Registry for a secondary site **(PREMIUM SELF)** +This document was moved to [another location](container_registry.md). -You can set up a [Docker Registry](https://docs.docker.com/registry/) on your -**secondary** Geo site that mirrors the one on the **primary** Geo site. - -## Storage support - -Docker Registry currently supports a few types of storage. If you choose a -distributed storage (`azure`, `gcs`, `s3`, `swift`, or `oss`) for your Docker -Registry on the **primary** site, you can use the same storage for a **secondary** -Docker Registry as well. For more information, read the -[Load balancing considerations](https://docs.docker.com/registry/deploying/#load-balancing-considerations) -when deploying the Registry, and how to set up the storage driver for the GitLab -integrated [Container Registry](../../packages/container_registry.md#use-object-storage). - -## Replicating Docker Registry - -You can enable a storage-agnostic replication so it -can be used for cloud or local storage. Whenever a new image is pushed to the -**primary** site, each **secondary** site pulls it to its own container -repository. - -To configure Docker Registry replication: - -1. Configure the [**primary** site](#configure-primary-site). -1. Configure the [**secondary** site](#configure-secondary-site). -1. Verify Docker Registry [replication](#verify-replication). - -### Configure **primary** site - -Make sure that you have Container Registry set up and working on -the **primary** site before following the next steps. - -We need to make Docker Registry send notification events to the -**primary** site. - -1. SSH into your GitLab **primary** server and login as root: - - ```shell - sudo -i - ``` - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - registry['notifications'] = [ - { - 'name' => 'geo_event', - 'url' => 'https://<example.com>/api/v4/container_registry_event/events', - 'timeout' => '500ms', - 'threshold' => 5, - 'backoff' => '1s', - 'headers' => { - 'Authorization' => ['<replace_with_a_secret_token>'] - } - } - ] - ``` - - NOTE: - Replace `<example.com>` with the `external_url` defined in your primary site's `/etc/gitlab/gitlab.rb` file, and - replace `<replace_with_a_secret_token>` with a case sensitive alphanumeric string - that starts with a letter. You can generate one with `< /dev/urandom tr -dc _A-Z-a-z-0-9 | head -c 32 | sed "s/^[0-9]*//"; echo` - - NOTE: - If you use an external Registry (not the one integrated with GitLab), you must add - these settings to its configuration yourself. In this case, you also have to specify - notification secret in `registry.notification_secret` section of - `/etc/gitlab/gitlab.rb` file. - - NOTE: - If you use GitLab HA, you also have to specify - the notification secret in `registry.notification_secret` section of - `/etc/gitlab/gitlab.rb` file for every web node. - -1. Reconfigure the **primary** node for the change to take effect: - - ```shell - gitlab-ctl reconfigure - ``` - -### Configure **secondary** site - -Make sure you have Container Registry set up and working on -the **secondary** site before following the next steps. - -The following steps should be done on each **secondary** site you're -expecting to see the Docker images replicated. - -Because we need to allow the **secondary** site to communicate securely with -the **primary** site Container Registry, we need to have a single key -pair for all the sites. The **secondary** site uses this key to -generate a short-lived JWT that is pull-only-capable to access the -**primary** site Container Registry. - -For each application and Sidekiq node on the **secondary** site: - -1. SSH into the node and login as the `root` user: - - ```shell - sudo -i - ``` - -1. Copy `/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key` from the **primary** to the node. - -1. Edit `/etc/gitlab/gitlab.rb` and add: - - ```ruby - gitlab_rails['geo_registry_replication_enabled'] = true - - # Primary registry's hostname and port, it will be used by - # the secondary node to directly communicate to primary registry - gitlab_rails['geo_registry_replication_primary_api_url'] = 'https://primary.example.com:5050/' - ``` - -1. Reconfigure the node for the change to take effect: - - ```shell - gitlab-ctl reconfigure - ``` - -### Verify replication - -To verify Container Registry replication is working, on the **secondary** site: - -1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **Geo > Nodes**. - The initial replication, or "backfill", is probably still in progress. - -You can monitor the synchronization process on each Geo site from the **primary** site's **Geo Nodes** dashboard in your browser. +<!-- This redirect file can be deleted after <2022-10-29>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
\ No newline at end of file diff --git a/doc/administration/geo/replication/faq.md b/doc/administration/geo/replication/faq.md index bdf1771e8a8..7a67af1cfa2 100644 --- a/doc/administration/geo/replication/faq.md +++ b/doc/administration/geo/replication/faq.md @@ -65,10 +65,10 @@ connectivity between your sites, your hardware, and so on. That's totally fine. We use HTTP(s) to fetch repository changes from the **primary** site to all **secondary** sites. -## Is this possible to set up a Docker Registry for a **secondary** site that mirrors the one on the **primary** site? +## Is this possible to set up a Container Registry for a **secondary** site that mirrors the one on the **primary** site? -Yes. See [Docker Registry for a **secondary** site](docker_registry.md). +Yes. See [Container Registry for a **secondary** site](container_registry.md). -## Can you login to a secondary site? +## Can you log in to a secondary site? Yes, but secondary sites receive all authentication data (like user accounts and logins) from the primary instance. This means you are re-directed to the primary for authentication and then routed back. diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 082ecbbb208..26d192f62cd 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -129,7 +129,7 @@ http://secondary.example.com/ ``` To find more details about failed items, check -[the `gitlab-rails/geo.log` file](../../troubleshooting/log_parsing.md#find-most-common-geo-sync-errors) +[the `gitlab-rails/geo.log` file](../../logs/log_parsing.md#find-most-common-geo-sync-errors) ### Check if PostgreSQL replication is working @@ -191,7 +191,7 @@ If a replication slot is inactive, the `pg_wal` logs corresponding to the slot are reserved forever (or until the slot is active again). This causes continuous disk usage growth and the following messages appear repeatedly in the -[PostgreSQL logs](../../logs.md#postgresql-logs): +[PostgreSQL logs](../../logs/index.md#postgresql-logs): ```plaintext WARNING: oldest xmin is far in the past @@ -331,8 +331,7 @@ Be sure to restart PostgreSQL for this to take effect. See the This occurs when PostgreSQL does not have a replication slot for the **secondary** node by that name. -You may want to rerun the [replication -process](../setup/database.md) on the **secondary** node . +You may want to rerun the [replication process](../setup/database.md) on the **secondary** node . ### Message: "Command exceeded allowed execution time" when setting up replication? @@ -376,7 +375,7 @@ log data to build up in `pg_xlog`. Removing the unused slots can reduce the amou Slots where `active` is `f` are not active. - When this slot should be active, because you have a **secondary** node configured using that slot, - sign in to that **secondary** node and check the [PostgreSQL logs](../../logs.md#postgresql-logs) + sign in to that **secondary** node and check the [PostgreSQL logs](../../logs/index.md#postgresql-logs) to view why the replication is not running. - If you are no longer using the slot (for example, you no longer have Geo enabled), you can remove it with in the @@ -510,7 +509,7 @@ To solve this: 1. Back up [the `.git` folder](../../repository_storage_types.md#translate-hashed-storage-paths). -1. Optional. [Spot-check](../../troubleshooting/log_parsing.md#find-all-projects-affected-by-a-fatal-git-problem) +1. Optional. [Spot-check](../../logs/log_parsing.md#find-all-projects-affected-by-a-fatal-git-problem) a few of those IDs whether they indeed correspond to a project with known Geo replication failures. Use `fatal: 'geo'` as the `grep` term and the following API call: @@ -597,7 +596,7 @@ to start again from scratch, there are a few steps that can help you: gitlab-ctl stop geo-logcursor ``` - You can watch the [Sidekiq logs](../../logs.md#sidekiq-logs) to know when Sidekiq jobs processing has finished: + You can watch the [Sidekiq logs](../../logs/index.md#sidekiq-logs) to know when Sidekiq jobs processing has finished: ```shell gitlab-ctl tail sidekiq @@ -837,7 +836,7 @@ to transfer each affected repository from the primary to the secondary site. The following are possible error messages that might be encountered during failover or when promoting a secondary to a primary node with strategies to resolve them. -### Message: ActiveRecord::RecordInvalid: Validation failed: Name has already been taken +### Message: `ActiveRecord::RecordInvalid: Validation failed: Name has already been taken` When [promoting a **secondary** site](../disaster_recovery/index.md#step-3-promoting-a-secondary-site), you might encounter the following error message: @@ -869,11 +868,10 @@ or `gitlab-ctl promote-to-primary-node`, either: ``` - Upgrade to GitLab 12.6.3 or later if it is safe to do so. For example, - if the failover was just a test. A [caching-related - bug](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22021) was - fixed. + if the failover was just a test. A + [caching-related bug](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22021) was fixed. -### Message: ActiveRecord::RecordInvalid: Validation failed: Enabled Geo primary node cannot be disabled +### Message: `ActiveRecord::RecordInvalid: Validation failed: Enabled Geo primary node cannot be disabled` If you disabled a secondary node, either with the [replication pause task](../index.md#pausing-and-resuming-replication) (GitLab 13.2) or by using the user interface (GitLab 13.1 and earlier), you must first @@ -1127,12 +1125,6 @@ Geo secondary sites continue to replicate and verify data, and the secondary sit This bug was [fixed in GitLab 14.4](https://gitlab.com/gitlab-org/gitlab/-/issues/292983). -### GitLab Pages return 404 errors after promoting - -This is due to [Pages data not being managed by Geo](datatypes.md#limitations-on-replicationverification). -Find advice to resolve those error messages in the -[Pages administration documentation](../../../administration/pages/index.md#404-error-after-promoting-a-geo-secondary-to-a-primary-node). - ### Primary site returns 500 error when accessing `/admin/geo/replication/projects` Navigating to **Admin > Geo > Replication** (or `/admin/geo/replication/projects`) on a primary Geo site, shows a 500 error, while that same link on the secondary works fine. The primary's `production.log` has a similar entry to the following: @@ -1146,7 +1138,28 @@ Geo::TrackingBase::SecondaryNotConfigured: Geo secondary database is not configu On a Geo primary site this error can be ignored. -This happens because GitLab is attempting to display registries from the [Geo tracking database](../../../administration/geo/#geo-tracking-database) which doesn't exist on the primary site (only the original projects exist on the primary; no replicated projects are present, therefore no tracking database exists). +This happens because GitLab is attempting to display registries from the [Geo tracking database](../../../administration/geo/index.md#geo-tracking-database) which doesn't exist on the primary site (only the original projects exist on the primary; no replicated projects are present, therefore no tracking database exists). + +### Secondary site returns 400 error "Request header or cookie too large" + +This error can happen when the internal URL of the primary site is incorrect. + +For example, when you use a unified URL and the primary site's internal URL is also equal to the external URL. This causes a loop when a secondary site proxies requests to the primary site's internal URL. + +To fix this issue, set the primary site's internal URL to a URL that is: + +- Unique to the primary site. +- Accessible from all secondary sites. + +1. Enter the [Rails console](../../operations/rails_console.md) on the primary site. + +1. Run the following, replacing `https://unique.url.for.primary.site` with your specific internal URL. + For example, depending on your network configuration, you could use an IP address, like + `http://1.2.3.4`. + + ```ruby + GeoNode.where(primary: true).first.update!(internal_url: "https://unique.url.for.primary.site") + ``` ## Fixing client errors @@ -1158,6 +1171,23 @@ requests redirected from the secondary to the primary node do not properly send Authorization header. This may result in either an infinite `Authorization <-> Redirect` loop, or Authorization error messages. +### Error: Net::ReadTimeout when pushing through SSH on a Geo secondary + +When you push large repositories through SSH on a Geo secondary site, you may encounter a timeout. +This is because Rails proxies the push to the primary and has a 60 second default timeout, +[as described in this Geo issue](https://gitlab.com/gitlab-org/gitlab/-/issues/7405). + +Current workarounds are: + +- Push through HTTP instead, where Workhorse proxies the request to the primary (or redirects to the primary if Geo proxying is not enabled). +- Push directly to the primary. + +Example log (`gitlab-shell.log`): + +```plaintext +Failed to contact primary https://primary.domain.com/namespace/push_test.git\\nError: Net::ReadTimeout\",\"result\":null}" code=500 method=POST pid=5483 url="http://127.0.0.1:3000/api/v4/geo/proxy_git_push_ssh/push" +``` + ## Recovering from a partial failover The partial failover to a secondary Geo *site* may be the result of a temporary/transient issue. Therefore, first attempt to run the promote command again. diff --git a/doc/administration/geo/replication/tuning.md b/doc/administration/geo/replication/tuning.md index 670459624f3..755ab45a76c 100644 --- a/doc/administration/geo/replication/tuning.md +++ b/doc/administration/geo/replication/tuning.md @@ -30,7 +30,7 @@ However, this may not lead to more downloads in parallel unless the number of available Sidekiq threads is also increased. For example, if repository synchronization concurrency is increased from 25 to 50, you may also want to increase the number of Sidekiq threads from 25 to 50. See the -[Sidekiq concurrency documentation](../../operations/extra_sidekiq_processes.md#number-of-threads) +[Sidekiq concurrency documentation](../../sidekiq/extra_sidekiq_processes.md#number-of-threads) for more details. ## Repository re-verification diff --git a/doc/administration/geo/replication/upgrading_the_geo_sites.md b/doc/administration/geo/replication/upgrading_the_geo_sites.md index 30961de0381..ce1ff4fe6a5 100644 --- a/doc/administration/geo/replication/upgrading_the_geo_sites.md +++ b/doc/administration/geo/replication/upgrading_the_geo_sites.md @@ -28,12 +28,23 @@ and cause downtime. If you want to avoid downtime, consider using To upgrade the Geo sites when a new GitLab version is released, upgrade **primary** and all **secondary** sites: -1. **Optional:** [Pause replication on each **secondary** sites.](../index.md#pausing-and-resuming-replication) +1. Optional. [Pause replication on each **secondary** site](../index.md#pausing-and-resuming-replication) + to protect the disaster recovery (DR) capability of the **secondary** sites. 1. SSH into each node of the **primary** site. 1. [Upgrade GitLab on the **primary** site](../../../update/package/index.md#upgrade-using-the-official-repositories). +1. Perform testing on the **primary** site, particularly if you paused replication in step 1 to protect DR. [There are some suggestions for post-upgrade testing](../../../update/plan_your_upgrade.md#pre-upgrade-and-post-upgrade-checks) in the upgrade documentation. 1. SSH into each node of **secondary** sites. 1. [Upgrade GitLab on each **secondary** site](../../../update/package/index.md#upgrade-using-the-official-repositories). -1. If you paused replication in step 1, [resume replication on each **secondary**](../index.md#pausing-and-resuming-replication) +1. If you paused replication in step 1, [resume replication on each **secondary**](../index.md#pausing-and-resuming-replication). + Then, restart Puma and Sidekiq on each **secondary** site. This is to ensure they + are initialized against the newer database schema that is now replicated from + the previously upgraded **primary** site. + + ```shell + sudo gitlab-ctl restart sidekiq + sudo gitlab-ctl restart puma + ``` + 1. [Test](#check-status-after-upgrading) **primary** and **secondary** sites, and check version in each. ### Check status after upgrading diff --git a/doc/administration/geo/replication/version_specific_upgrades.md b/doc/administration/geo/replication/version_specific_upgrades.md index f0925bdf87e..350310c7076 100644 --- a/doc/administration/geo/replication/version_specific_upgrades.md +++ b/doc/administration/geo/replication/version_specific_upgrades.md @@ -178,11 +178,15 @@ GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab ## Upgrading to GitLab 13.7 -We've detected an issue with the `FetchRemove` call used by Geo secondaries. -This causes performance issues as we execute reference transaction hooks for -each upgraded reference. Delay any upgrade attempts until this is in the -[13.7.5 patch release.](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3002). -More details are available [in this issue](https://gitlab.com/gitlab-org/git/-/issues/79). +- We've detected an issue with the `FetchRemove` call used by Geo secondaries. + This causes performance issues as we execute reference transaction hooks for + each upgraded reference. Delay any upgrade attempts until this is in the + [13.7.5 patch release.](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3002). + More details are available [in this issue](https://gitlab.com/gitlab-org/git/-/issues/79). +- A new secret is generated in `/etc/gitlab/gitlab-secrets.json`. + In an HA GitLab or GitLab Geo environment, secrets need to be the same on all nodes. + Ensure this new secret is also accounted for if you are manually syncing the file across + nodes, or manually specifying secrets in `/etc/gitlab/gitlab.rb`. ## Upgrading to GitLab 13.5 @@ -243,8 +247,8 @@ the recommended procedure, see the ## Upgrading to GitLab 12.9 WARNING: -GitLab 12.9.0 through GitLab 12.9.3 are affected by [a bug that stops -repository verification](https://gitlab.com/gitlab-org/gitlab/-/issues/213523). +GitLab 12.9.0 through GitLab 12.9.3 are affected by +[a bug that stops repository verification](https://gitlab.com/gitlab-org/gitlab/-/issues/213523). The issue is fixed in GitLab 12.9.4. Upgrade to GitLab 12.9.4 or later. By default, GitLab 12.9 attempts to upgrade the embedded PostgreSQL server @@ -397,6 +401,6 @@ For the recommended procedure, see the ## Upgrading to GitLab 12.0 WARNING: -This version is affected by a [bug that results in new LFS objects not being -replicated to Geo secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). +This version is affected by a +[bug that results in new LFS objects not being replicated to Geo secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). The issue is fixed in GitLab 12.1. Be sure to upgrade to GitLab 12.1 or later. diff --git a/doc/administration/geo/secondary_proxy/index.md b/doc/administration/geo/secondary_proxy/index.md index e8c290e197b..6c1812b2754 100644 --- a/doc/administration/geo/secondary_proxy/index.md +++ b/doc/administration/geo/secondary_proxy/index.md @@ -112,8 +112,9 @@ gitlab: Since GitLab 15.1, Geo secondary proxying is enabled by default for separate URLs also. -There are minor known issues linked in the ["Geo secondary proxying with separate URLs" -epic](https://gitlab.com/groups/gitlab-org/-/epics/6865). You can also add feedback in the epic about any use-cases that +There are minor known issues linked in the +["Geo secondary proxying with separate URLs" epic](https://gitlab.com/groups/gitlab-org/-/epics/6865). +You can also add feedback in the epic about any use-cases that are not possible anymore with proxying enabled. If you run into issues, to disable this feature, disable the `geo_secondary_proxy_separate_urls` feature flag. diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index c0ed7829fce..8a919a0a269 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -8,7 +8,7 @@ type: howto # Geo database replication **(PREMIUM SELF)** This document describes the minimal required steps to replicate your primary -GitLab database to a secondary node's database. You may have to change some +GitLab database to a secondary site's database. You may have to change some values, based on attributes including your database's setup and size. NOTE: @@ -41,31 +41,31 @@ instructions on setting up replication with a Patroni cluster. ### PostgreSQL replication -The GitLab **primary** node where the write operations happen connects to -the **primary** database server, and **secondary** nodes +The GitLab **primary** site where the write operations happen connects to +the **primary** database server, and **secondary** sites connect to their own database servers (which are read-only). We recommend using [PostgreSQL replication slots](https://medium.com/@tk512/replication-slots-in-postgresql-b4b03d277c75) -to ensure that the **primary** node retains all the data necessary for the **secondary** nodes to +to ensure that the **primary** site retains all the data necessary for the **secondary** sites to recover. See below for more details. The following guide assumes that: - You are using Omnibus and therefore you are using PostgreSQL 12 or later which includes the [`pg_basebackup` tool](https://www.postgresql.org/docs/12/app-pgbasebackup.html). -- You have a **primary** node already set up (the GitLab server you are +- You have a **primary** site already set up (the GitLab server you are replicating from), running Omnibus' PostgreSQL (or equivalent version), and - you have a new **secondary** server set up with the same + you have a new **secondary** site set up with the same [versions of PostgreSQL](../index.md#requirements-for-running-geo), - OS, and GitLab on all nodes. + OS, and GitLab on all sites. WARNING: Geo works with streaming replication. Logical replication is not supported at this time. There is an [issue where support is being discussed](https://gitlab.com/gitlab-org/gitlab/-/issues/7420). -#### Step 1. Configure the **primary** server +#### Step 1. Configure the **primary** site -1. SSH into your GitLab **primary** server and login as root: +1. SSH into your GitLab **primary** site and login as root: ```shell sudo -i @@ -81,13 +81,13 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o gitlab_rails['geo_node_name'] = '<site_name_here>' ``` -1. Reconfigure the **primary** node for the change to take effect: +1. Reconfigure the **primary** site for the change to take effect: ```shell gitlab-ctl reconfigure ``` -1. Execute the command below to define the node as **primary** node: +1. Execute the command below to define the site as **primary** site: ```shell gitlab-ctl set-geo-primary-node @@ -154,17 +154,17 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o 1. Configure PostgreSQL to listen on network interfaces: For security reasons, PostgreSQL does not listen on any network interfaces - by default. However, Geo requires the **secondary** node to be able to - connect to the **primary** node's database. For this reason, we need the address of - each node. + by default. However, Geo requires the **secondary** site to be able to + connect to the **primary** site's database. For this reason, we need the IP address of + each site. NOTE: For external PostgreSQL instances, see [additional instructions](external_database.md). If you are using a cloud provider, you can lookup the addresses for each - Geo node through your cloud provider's management console. + Geo site through your cloud provider's management console. - To lookup the address of a Geo node, SSH in to the Geo node and execute: + To lookup the address of a Geo site, SSH in to the Geo site and execute: ```shell ## @@ -183,11 +183,11 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o | Configuration | Address | |:----------------------------------------|:----------------------------------------------------------------------| - | `postgresql['listen_address']` | **Primary** node's public or VPC private address. | - | `postgresql['md5_auth_cidr_addresses']` | **Primary** and **Secondary** nodes' public or VPC private addresses. | + | `postgresql['listen_address']` | **Primary** site's public or VPC private address. | + | `postgresql['md5_auth_cidr_addresses']` | **Primary** and **Secondary** sites' public or VPC private addresses. | If you are using Google Cloud Platform, SoftLayer, or any other vendor that - provides a virtual private cloud (VPC) you can use the **primary** and **secondary** nodes + provides a virtual private cloud (VPC) you can use the **primary** and **secondary** sites private addresses (corresponds to "internal address" for Google Cloud Platform) for `postgresql['md5_auth_cidr_addresses']` and `postgresql['listen_address']`. @@ -201,10 +201,10 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o `127.0.0.1`. For more information, see [omnibus-5258](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5258). Depending on your network configuration, the suggested addresses may not - be correct. If your **primary** node and **secondary** nodes connect over a local + be correct. If your **primary** site and **secondary** sites connect over a local area network, or a virtual network connecting availability zones like [Amazon's VPC](https://aws.amazon.com/vpc/) or [Google's VPC](https://cloud.google.com/vpc/) - you should use the **secondary** node's private address for `postgresql['md5_auth_cidr_addresses']`. + you should use the **secondary** site's private address for `postgresql['md5_auth_cidr_addresses']`. Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP addresses with addresses appropriate to your network configuration: @@ -225,19 +225,18 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ## Primary address ## - replace '<primary_node_ip>' with the public or VPC address of your Geo primary node ## - postgresql['listen_address'] = '<primary_node_ip>' + postgresql['listen_address'] = '<primary_site_ip>' ## # Allow PostgreSQL client authentication from the primary and secondary IPs. These IPs may be # public or VPC addresses in CIDR format, for example ['198.51.100.1/32', '198.51.100.2/32'] ## - postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32'] + postgresql['md5_auth_cidr_addresses'] = ['<primary_site_ip>/32', '<secondary_site_ip>/32'] ## ## Replication settings - ## - set this to be the number of Geo secondary nodes you have ## - postgresql['max_replication_slots'] = 1 + # postgresql['max_replication_slots'] = 1 # Set this to be the number of Geo secondary nodes if you have more than one # postgresql['max_wal_senders'] = 10 # postgresql['wal_keep_segments'] = 10 @@ -248,10 +247,10 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o gitlab_rails['auto_migrate'] = false ``` -1. Optional: If you want to add another **secondary** node, the relevant setting would look like: +1. Optional: If you want to add another **secondary** site, the relevant setting would look like: ```ruby - postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32', '<another_secondary_node_ip>/32'] + postgresql['md5_auth_cidr_addresses'] = ['<primary_site_ip>/32', '<secondary_site_ip>/32', '<another_secondary_site_ip>/32'] ``` You may also want to edit the `wal_keep_segments` and `max_wal_senders` to match your @@ -288,20 +287,20 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o 1. Now that the PostgreSQL server is set up to accept remote connections, run `netstat -plnt | grep 5432` to make sure that PostgreSQL is listening on port - `5432` to the **primary** server's private address. + `5432` to the **primary** site's private address. 1. A certificate was automatically generated when GitLab was reconfigured. This is used automatically to protect your PostgreSQL traffic from eavesdroppers, but to protect against active ("man-in-the-middle") attackers, - the **secondary** node needs a copy of the certificate. Make a copy of the PostgreSQL - `server.crt` file on the **primary** node by running this command: + the **secondary** site needs a copy of the certificate. Make a copy of the PostgreSQL + `server.crt` file on the **primary** site by running this command: ```shell cat ~gitlab-psql/data/server.crt ``` Copy the output into a clipboard or into a local file. You - need it when setting up the **secondary** node! The certificate is not sensitive + need it when setting up the **secondary** site! The certificate is not sensitive data. However, this certificate is created with a generic `PostgreSQL` Common Name. For this, @@ -324,7 +323,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o #### Step 2. Configure the **secondary** server -1. SSH into your GitLab **secondary** server and login as root: +1. SSH into your GitLab **secondary** site and login as root: ```shell sudo -i @@ -338,28 +337,28 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ``` NOTE: - This step is important so we don't try to execute anything before the node is fully configured. + This step is important so we don't try to execute anything before the site is fully configured. -1. [Check TCP connectivity](../../raketasks/maintenance.md) to the **primary** node's PostgreSQL server: +1. [Check TCP connectivity](../../raketasks/maintenance.md) to the **primary** site's PostgreSQL server: ```shell - gitlab-rake gitlab:tcp_check[<primary_node_ip>,5432] + gitlab-rake gitlab:tcp_check[<primary_site_ip>,5432] ``` NOTE: If this step fails, you may be using the wrong IP address, or a firewall may - be preventing access to the server. Check the IP address, paying close + be preventing access to the site. Check the IP address, paying close attention to the difference between public and private addresses and ensure - that, if a firewall is present, the **secondary** node is permitted to connect to the - **primary** node on port 5432. + that, if a firewall is present, the **secondary** site is permitted to connect to the + **primary** site on port 5432. -1. Create a file `server.crt` in the **secondary** server, with the content you got on the last step of the **primary** node's setup: +1. Create a file `server.crt` in the **secondary** site, with the content you got on the last step of the **primary** site's setup: ```shell editor server.crt ``` -1. Set up PostgreSQL TLS verification on the **secondary** node: +1. Set up PostgreSQL TLS verification on the **secondary** site: Install the `server.crt` file: @@ -374,9 +373,9 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o PostgreSQL now only recognizes that exact certificate when verifying TLS connections. The certificate can only be replicated by someone with access - to the private key, which is **only** present on the **primary** node. + to the private key, which is **only** present on the **primary** site. -1. Test that the `gitlab-psql` user can connect to the **primary** node's database +1. Test that the `gitlab-psql` user can connect to the **primary** site's database (the default Omnibus database name is `gitlabhq_production`): ```shell @@ -386,7 +385,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o -U gitlab_replicator \ -d "dbname=gitlabhq_production sslmode=verify-ca" \ -W \ - -h <primary_node_ip> + -h <primary_site_ip> ``` NOTE: @@ -397,11 +396,11 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o When prompted enter the _plaintext_ password you set in the first step for the `gitlab_replicator` user. If all worked correctly, you should see - the list of **primary** node's databases. + the list of **primary** site's databases. A failure to connect here indicates that the TLS configuration is incorrect. - Ensure that the contents of `~gitlab-psql/data/server.crt` on the **primary** node - match the contents of `~gitlab-psql/.postgresql/root.crt` on the **secondary** node. + Ensure that the contents of `~gitlab-psql/data/server.crt` on the **primary** site + match the contents of `~gitlab-psql/.postgresql/root.crt` on the **secondary** site. 1. Configure PostgreSQL: @@ -420,14 +419,14 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ## ## Secondary address - ## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node + ## - replace '<secondary_site_ip>' with the public or VPC address of your Geo secondary site ## - postgresql['listen_address'] = '<secondary_node_ip>' - postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32'] + postgresql['listen_address'] = '<secondary_site_ip>' + postgresql['md5_auth_cidr_addresses'] = ['<secondary_site_ip>/32'] ## - ## Database credentials password (defined previously in primary node) - ## - replicate same values here as defined in primary node + ## Database credentials password (defined previously in primary site) + ## - replicate same values here as defined in primary site ## postgresql['sql_replication_password'] = '<md5_hash_of_your_password>' postgresql['sql_user_password'] = '<md5_hash_of_your_password>' @@ -435,7 +434,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ``` For external PostgreSQL instances, see [additional instructions](external_database.md). - If you bring a former **primary** node back online to serve as a **secondary** node, then you also must remove `roles(['geo_primary_role'])` or `geo_primary_role['enable'] = true`. + If you bring a former **primary** site back online to serve as a **secondary** site, then you also must remove `roles(['geo_primary_role'])` or `geo_primary_role['enable'] = true`. 1. Reconfigure GitLab for the changes to take effect: @@ -451,24 +450,24 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o #### Step 3. Initiate the replication process -Below we provide a script that connects the database on the **secondary** node to -the database on the **primary** node, replicates the database, and creates the +Below we provide a script that connects the database on the **secondary** site to +the database on the **primary** site, replicates the database, and creates the needed files for streaming replication. The directories used are the defaults that are set up in Omnibus. If you have changed any defaults, configure it as you see fit replacing the directories and paths. WARNING: -Make sure to run this on the **secondary** server as it removes all PostgreSQL's +Make sure to run this on the **secondary** site as it removes all PostgreSQL's data before running `pg_basebackup`. -1. SSH into your GitLab **secondary** server and login as root: +1. SSH into your GitLab **secondary** site and login as root: ```shell sudo -i ``` -1. Choose a database-friendly name to use for your **secondary** node to +1. Choose a database-friendly name to use for your **secondary** site to use as the replication slot name. For example, if your domain is `secondary.geo.example.com`, you may use `secondary_example` as the slot name as shown in the commands below. @@ -476,13 +475,13 @@ data before running `pg_basebackup`. 1. Execute the command below to start a backup/restore and begin the replication WARNING: - Each Geo **secondary** node must have its own unique replication slot name. + Each Geo **secondary** site must have its own unique replication slot name. Using the same slot name between two secondaries breaks PostgreSQL replication. ```shell gitlab-ctl replicate-geo-database \ - --slot-name=<secondary_node_name> \ - --host=<primary_node_ip> \ + --slot-name=<secondary_site_name> \ + --host=<primary_site_ip> \ --sslmode=verify-ca ``` @@ -516,7 +515,7 @@ data before running `pg_basebackup`. - Change the `--slot-name` to the name of the replication slot to be used on the **primary** database. The script attempts to create the replication slot automatically if it does not exist. - - If you're repurposing an old server into a Geo **secondary** node, you must + - If you're repurposing an old site into a Geo **secondary** site, you must add `--force` to the command line. - When not in a production machine you can disable backup step if you really sure this is what you want by adding `--skip-backup` @@ -540,7 +539,7 @@ see [High Availability with Omnibus GitLab](../../postgresql/replication_and_fai To change the password for the [replication user](https://wiki.postgresql.org/wiki/Streaming_Replication) when using Omnibus-managed PostgreSQL instances: -On the GitLab Geo **primary** server: +On the GitLab Geo **primary** site: 1. The default value for the replication user is `gitlab_replicator`, but if you've set a custom replication user in your `/etc/gitlab/gitlab.rb` under the `postgresql['sql_replication_user']` setting, make sure to @@ -574,19 +573,18 @@ On the GitLab Geo **primary** server: sudo gitlab-ctl restart postgresql ``` -Until the password is updated on any **secondary** servers, the [PostgreSQL log](../../logs.md#postgresql-logs) on +Until the password is updated on any **secondary** sites, the [PostgreSQL log](../../logs/index.md#postgresql-logs) on the secondaries will report the following error message: ```console FATAL: could not connect to the primary server: FATAL: password authentication failed for user "gitlab_replicator" ``` -On all GitLab Geo **secondary** servers: +On all GitLab Geo **secondary** sites: 1. The first step isn't necessary from a configuration perspective, because the hashed `'sql_replication_password'` - is not used on the GitLab Geo **secondary**. However in the event that **secondary** needs to be promoted - to the GitLab Geo **primary**, make sure to match the `'sql_replication_password'` in the secondary - server configuration. + is not used on the GitLab Geo **secondary** sites. However in the event that **secondary** site needs to be promoted + to the GitLab Geo **primary**, make sure to match the `'sql_replication_password'` in the **secondary** site configuration. Edit `/etc/gitlab/gitlab.rb`: @@ -616,6 +614,28 @@ In GitLab 14.0, Patroni replaced `repmgr` as the supported NOTE: If you still haven't [migrated from repmgr to Patroni](#migrating-from-repmgr-to-patroni) you're highly advised to do so. +### Migrating from repmgr to Patroni + +1. Before migrating, we recommend that there is no replication lag between the **primary** and **secondary** sites and that replication is paused. In GitLab 13.2 and later, you can pause and resume replication with `gitlab-ctl geo-replication-pause` and `gitlab-ctl geo-replication-resume` on a Geo secondary database node. +1. Follow the [instructions to migrate repmgr to Patroni](../../postgresql/replication_and_failover.md#switching-from-repmgr-to-patroni). When configuring Patroni on each **primary** site database node, add `patroni['replication_slots'] = { '<slot_name>' => 'physical' }` +to `gitlab.rb` where `<slot_name>` is the name of the replication slot for your **secondary** site. This ensures that Patroni recognizes the replication slot as permanent and not drop it upon restarting. +1. If database replication to the **secondary** site was paused before migration, resume replication once Patroni is confirmed working on the **primary** site. + +### Migrating a single PostgreSQL node to Patroni + +Before the introduction of Patroni, Geo had no Omnibus support for HA setups on the **secondary** site. + +With Patroni it's now possible to support that. To migrate the existing PostgreSQL to Patroni: + +1. Make sure you have a Consul cluster setup on the secondary (similar to how you set it up on the **primary** site). +1. [Configure a permanent replication slot](#step-1-configure-patroni-permanent-replication-slot-on-the-primary-site). +1. [Configure the internal load balancer](#step-2-configure-the-internal-load-balancer-on-the-primary-site). +1. [Configure a PgBouncer node](#step-3-configure-pgbouncer-nodes-on-the-secondary-site) +1. [Configure a Standby Cluster](#step-4-configure-a-standby-cluster-on-the-secondary-site) + on that single node machine. + +You end up with a “Standby Cluster” with a single node. That allows you to later on add additional Patroni nodes by following the same instructions above. + ### Patroni support Patroni is the official replication management solution for Geo. It @@ -648,11 +668,11 @@ and other database best practices. ##### Step 1. Configure Patroni permanent replication slot on the primary site -To set up database replication with Patroni on a secondary node, we must -configure a _permanent replication slot_ on the primary node's Patroni cluster, +To set up database replication with Patroni on a secondary site, we must +configure a _permanent replication slot_ on the primary site's Patroni cluster, and ensure password authentication is used. -For each Patroni instance on the primary site **starting on the Patroni +On each node running a Patroni instance on the primary site **starting on the Patroni Leader instance**: 1. SSH into your Patroni instance and login as root: @@ -755,7 +775,7 @@ backend postgresql Refer to your preferred Load Balancer's documentation for further guidance. -##### Step 3. Configure a PgBouncer node on the secondary site +##### Step 3. Configure PgBouncer nodes on the secondary site A production-ready and highly available configuration requires at least three Consul nodes, a minimum of one PgBouncer node, but it's recommended to have @@ -764,7 +784,7 @@ more than one PgBouncer service nodes. The internal load balancer provides a sin endpoint for connecting to the PgBouncer cluster. For more information, see [High Availability with Omnibus GitLab](../../postgresql/replication_and_failover.md). -Follow the minimal configuration for the PgBouncer node: +On each node running a PgBouncer instance on the **secondary** site: 1. SSH into your PgBouncer node and login as root: @@ -820,11 +840,10 @@ Follow the minimal configuration for the PgBouncer node: ##### Step 4. Configure a Standby cluster on the secondary site NOTE: -If you are converting a secondary site to a Patroni Cluster, you must start -on the PostgreSQL instance. It becomes the Patroni Standby Leader instance, -and then you can switchover to another replica if you need. +If you are converting a secondary site with a single PostgreSQL instance to a Patroni Cluster, you must start on the PostgreSQL instance. It becomes the Patroni Standby Leader instance, +and then you can switch over to another replica if you need. -For each Patroni instance on the secondary site: +For each node running a Patroni instance on the secondary site: 1. SSH into your Patroni node and login as root: @@ -894,39 +913,29 @@ For each Patroni instance on the secondary site: gitlab-ctl start patroni ``` -### Migrating from repmgr to Patroni - -1. Before migrating, we recommend that there is no replication lag between the primary and secondary sites and that replication is paused. In GitLab 13.2 and later, you can pause and resume replication with `gitlab-ctl geo-replication-pause` and `gitlab-ctl geo-replication-resume` on a Geo secondary database node. -1. Follow the [instructions to migrate repmgr to Patroni](../../postgresql/replication_and_failover.md#switching-from-repmgr-to-patroni). When configuring Patroni on each primary site database node, add `patroni['replication_slots'] = { '<slot_name>' => 'physical' }` -to `gitlab.rb` where `<slot_name>` is the name of the replication slot for your Geo secondary. This ensures that Patroni recognizes the replication slot as permanent and not drop it upon restarting. -1. If database replication to the secondary was paused before migration, resume replication once Patroni is confirmed working on the primary. - -### Migrating a single PostgreSQL node to Patroni - -Before the introduction of Patroni, Geo had no Omnibus support for HA setups on the secondary node. +### Migrating a single tracking database node to Patroni -With Patroni it's now possible to support that. To migrate the existing PostgreSQL to Patroni: +Before the introduction of Patroni, Geo had no Omnibus support for HA setups on +the secondary site. -1. Make sure you have a Consul cluster setup on the secondary (similar to how you set it up on the primary). -1. [Configure a permanent replication slot](#step-1-configure-patroni-permanent-replication-slot-on-the-primary-site). -1. [Configure the internal load balancer](#step-2-configure-the-internal-load-balancer-on-the-primary-site). -1. [Configure a PgBouncer node](#step-3-configure-a-pgbouncer-node-on-the-secondary-site) -1. [Configure a Standby Cluster](#step-4-configure-a-standby-cluster-on-the-secondary-site) - on that single node machine. +With Patroni, it's now possible to support that. Due to some restrictions on the +Patroni implementation on Omnibus that do not allow us to manage two different +clusters on the same machine, we recommend setting up a new Patroni cluster for +the tracking database by following the same instructions above. -You end up with a "Standby Cluster" with a single node. That allows you to later on add additional Patroni nodes -by following the same instructions above. +The secondary nodes backfill the new tracking database, and no data +synchronization is required. ### Configuring Patroni cluster for the tracking PostgreSQL database -Secondary sites use a separate PostgreSQL installation as a tracking database to +**Secondary** sites use a separate PostgreSQL installation as a tracking database to keep track of replication status and automatically recover from potential replication issues. Omnibus automatically configures a tracking database when `roles(['geo_secondary_role'])` is set. If you want to run this database in a highly available configuration, don't use the `geo_secondary_role` above. Instead, follow the instructions below. -A production-ready and secure setup requires at least three Consul nodes, two +A production-ready and secure setup for the tracking PostgreSQL DB requires at least three Consul nodes, two Patroni nodes and one PgBouncer node on the secondary site. Because of [omnibus-6587](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6587), Consul can't track multiple @@ -935,9 +944,9 @@ services, so these must be different than the nodes used for the Standby Cluster Be sure to use [password credentials](../../postgresql/replication_and_failover.md#database-authorization-for-patroni) and other database best practices. -#### Step 1. Configure a PgBouncer node on the secondary site +#### Step 1. Configure PgBouncer nodes on the secondary site -Follow the minimal configuration for the PgBouncer node for the tracking database: +On each node running the PgBouncer service for the PostgreSQL tracking database: 1. SSH into your PgBouncer node and login as root: @@ -999,7 +1008,7 @@ Follow the minimal configuration for the PgBouncer node for the tracking databas #### Step 2. Configure a Patroni cluster -For each Patroni instance on the secondary site for the tracking database: +On each node running a Patroni instance on the secondary site for the PostgreSQL tracking database: 1. SSH into your Patroni node and login as root: @@ -1063,7 +1072,7 @@ For each Patroni instance on the secondary site for the tracking database: gitlab-ctl reconfigure ``` -#### Step 3. Configure the tracking database on the secondary nodes +#### Step 3. Configure the tracking database on the secondary sites For each node running the `gitlab-rails`, `sidekiq`, and `geo-logcursor` services: @@ -1100,19 +1109,6 @@ For each node running the `gitlab-rails`, `sidekiq`, and `geo-logcursor` service gitlab-rake db:migrate:geo ``` -### Migrating a single tracking database node to Patroni - -Before the introduction of Patroni, Geo had no Omnibus support for HA setups on -the secondary node. - -With Patroni, it's now possible to support that. Due to some restrictions on the -Patroni implementation on Omnibus that do not allow us to manage two different -clusters on the same machine, we recommend setting up a new Patroni cluster for -the tracking database by following the same instructions above. - -The secondary nodes backfill the new tracking database, and no data -synchronization is required. - ## Troubleshooting Read the [troubleshooting document](../replication/troubleshooting.md). diff --git a/doc/administration/geo/setup/index.md b/doc/administration/geo/setup/index.md index 5ddfee6774e..79b52ef71da 100644 --- a/doc/administration/geo/setup/index.md +++ b/doc/administration/geo/setup/index.md @@ -22,8 +22,10 @@ The steps below should be followed in the order they appear. **Make sure the Git If you installed GitLab using the Omnibus packages (highly recommended): +1. Confirm the [requirements for running Geo](../index.md#requirements-for-running-geo) are met. 1. [Install GitLab Enterprise Edition](https://about.gitlab.com/install/) on the nodes that serve as the **secondary** site. **Do not create an account or log in** to the new **secondary** site. The **GitLab version must match** across primary and secondary sites. 1. [Add the GitLab License](../../../user/admin_area/license.md) on the **primary** site to unlock Geo. The license must be for [GitLab Premium](https://about.gitlab.com/pricing/) or higher. +1. [Confirm network connectivity](../index.md#firewall-rules) between the **primary** and **secondary** site. 1. [Set up the database replication](database.md) (`primary (read-write) <-> secondary (read-only)` topology). 1. [Configure fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md). This step is required and needs to be done on **both** the **primary** and **secondary** sites. 1. [Configure GitLab](../replication/configuration.md) to set the **primary** and **secondary** sites. |