diff options
Diffstat (limited to 'doc/administration/geo')
18 files changed, 135 insertions, 109 deletions
diff --git a/doc/administration/geo/disaster_recovery/background_verification.md b/doc/administration/geo/disaster_recovery/background_verification.md index 99ac95cd536..ea4523c66e6 100644 --- a/doc/administration/geo/disaster_recovery/background_verification.md +++ b/doc/administration/geo/disaster_recovery/background_verification.md @@ -6,12 +6,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Automatic background verification **(PREMIUM SELF)** -NOTE: -Automatic background verification of repositories and wikis was added in -GitLab EE 10.6 but is enabled by default only on GitLab EE 11.1. You can -disable or enable this feature manually by following -[these instructions](#disabling-or-enabling-the-automatic-background-verification). - Automatic background verification ensures that the transferred data matches a calculated checksum. If the checksum of the data on the **primary** site matches checksum of the data on the **secondary** site, the data transferred successfully. Following a planned failover, @@ -179,19 +173,5 @@ If the **primary** and **secondary** sites have a checksum verification mismatch ## Current limitations -Automatic background verification doesn't cover attachments, LFS objects, -job artifacts, and user uploads in file storage. You can keep track of the -progress to include them in [Geo: Verify all replicated data](https://gitlab.com/groups/gitlab-org/-/epics/1430). - -For now, you can verify their integrity -manually by following [these instructions](../../raketasks/check.md) on both -sites, and comparing the output between them. - -In GitLab EE 12.1, Geo calculates checksums for attachments, LFS objects, and -archived traces on secondary sites after the transfer, compares it with the -stored checksums, and rejects transfers if mismatched. Geo -currently does not support an automatic way to verify these data if they have -been synced before GitLab EE 12.1. - -Data in object storage is **not verified**, as the object store is responsible -for ensuring the integrity of the data. +For more information on what replication and verification methods are supported, +see the [supported Geo data types](../replication/datatypes.md). diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index e9eb154d398..a52bdc759a2 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -38,9 +38,9 @@ performed ahead of the maintenance window; subsequent `rsync`s (including a final transfer inside the maintenance window) then transfers only the *changes* between the **primary** site and the **secondary** sites. -Repository-centric strategies for using `rsync` effectively can be found in the +Git repository-centric strategies for using `rsync` effectively can be found in the [moving repositories](../../operations/moving_repositories.md) documentation; these strategies can -be adapted for use with any other file-based data, such as [GitLab Pages](../../pages/index.md#change-storage-path). +be adapted for use with any other file-based data. ### Container registry diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md index cef0101dd40..feb5e030b80 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md @@ -6,7 +6,7 @@ type: howto --- WARNING: -This runbook is in [**Alpha**](../../../../policy/alpha-beta-support.md#alpha-features). For complete, production-ready documentation, see the +This runbook is an [Experiment](../../../../policy/alpha-beta-support.md#experiment). For complete, production-ready documentation, see the [disaster recovery documentation](../index.md). # Disaster Recovery (Geo) promotion runbooks **(PREMIUM SELF)** diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md index 085765ec51e..19150027cca 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md @@ -6,7 +6,7 @@ type: howto --- WARNING: -This runbook is in [**Alpha**](../../../../policy/alpha-beta-support.md#alpha-features). For complete, production-ready documentation, see the +This runbook is an [Experiment](../../../../policy/alpha-beta-support.md#experiment). For complete, production-ready documentation, see the [disaster recovery documentation](../index.md). # Disaster Recovery (Geo) promotion runbooks **(PREMIUM SELF)** diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index 3f98f1e12fe..31de7f5c62f 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -9,8 +9,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w Geo is the solution for widely distributed development teams and for providing a warm-standby as part of a disaster recovery strategy. -## Overview - WARNING: Geo undergoes significant changes from release to release. Upgrades are supported and [documented](#upgrading-geo), but you should ensure that you're @@ -123,7 +121,8 @@ The following are required to run Geo: - Note,[PostgreSQL 12 is deprecated](../../update/deprecations.md#postgresql-12-deprecated) and is removed in GitLab 16.0. - Git 2.9 or later - Git-lfs 2.4.2 or later on the user side when using LFS -- All sites must run [the same GitLab and PostgreSQL versions](setup/database.md#postgresql-replication). +- All sites must run the same GitLab version. +- All sites must run [the same PostgreSQL versions](setup/database.md#postgresql-replication). - If using different operating system versions between Geo sites, [check OS locale data compatibility](replication/troubleshooting.md#check-os-locale-data-compatibility) across Geo sites to avoid silent corruption of database indexes. diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index 912de360e43..5fa6df393b9 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -205,15 +205,16 @@ keys must be manually replicated to the **secondary** site. 1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Geo > Sites**. 1. Select **Add site**. -  - 1. Fill in **Name** with the `gitlab_rails['geo_node_name']` in - `/etc/gitlab/gitlab.rb`. These values must always match *exactly*, character +  + 1. In **Name**, enter the value for `gitlab_rails['geo_node_name']` in + `/etc/gitlab/gitlab.rb`. These values must always match **exactly**, character for character. - 1. Fill in **URL** with the `external_url` in `/etc/gitlab/gitlab.rb`. These + 1. In **External URL**, enter the value for `external_url` in `/etc/gitlab/gitlab.rb`. These values must always match, but it doesn't matter if one ends with a `/` and the other doesn't. - 1. (Optional) Choose which groups or storage shards should be replicated by the - **secondary** site. Leave blank to replicate all. Read more in + 1. Optional. In **Internal URL (optional)**, enter an internal URL for the primary site. + 1. Optional. Select which groups or storage shards should be replicated by the + **secondary** site. Leave blank to replicate all. For more information, see [selective synchronization](#selective-synchronization). 1. Select **Save changes** to add the **secondary** site. 1. SSH into **each Rails, and Sidekiq node on your secondary** site and restart the services: @@ -257,12 +258,12 @@ You can safely skip this step if: #### Custom or self-signed certificate for inbound connections -If your GitLab Geo **primary** site uses a custom or [self-signed certificate to secure inbound HTTPS connections](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates), this certificate can either be single-domain certificate or multi-domain. +If your GitLab Geo **primary** site uses a custom or [self-signed certificate to secure inbound HTTPS connections](https://docs.gitlab.com/omnibus/settings/ssl/index.html#install-custom-public-certificates), this can be either a single-domain or multi-domain certificate. Install the correct certificate based on your certificate type: - **Multi-domain certificate** that includes both primary and secondary site domains: Install the certificate at `/etc/gitlab/ssl` on all **Rails, Sidekiq, and Gitaly** nodes in the **secondary** site. -- **Single-domain certificate** where the certificates are specific to each Geo site domain: Generate a valid certificate for your **secondary** site's domain and install it at `/etc/gitlab/ssl` per [these instructions](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) on all **Rails, Sidekiq, and Gitaly** nodes in the **secondary** site. +- **Single-domain certificate** where the certificates are specific to each Geo site domain: Generate a valid certificate for your **secondary** site's domain and install it at `/etc/gitlab/ssl` following [these instructions](https://docs.gitlab.com/omnibus/settings/ssl/index.html#install-custom-public-certificates) on all **Rails, Sidekiq, and Gitaly** nodes in the **secondary** site. #### Connecting to external services that use custom certificates @@ -303,7 +304,7 @@ If your **primary** site is using a [custom or self-signed certificate for inbou sudo gitlab-ctl reconfigure ``` -### Step 5. Enable Git access over HTTP/HTTPS +### Step 5. Enable Git access over HTTP/HTTPS and SSH Geo synchronizes repositories over HTTP/HTTPS, and therefore requires this clone method to be enabled. This is enabled by default, but if converting an existing site to Geo it should be checked: @@ -313,7 +314,10 @@ On the **primary** site: 1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Settings > General**. 1. Expand **Visibility and access controls**. -1. Ensure "Enabled Git access protocols" is set to either "Both SSH and HTTP(S)" or "Only HTTP(S)". +1. If using Git over SSH, then: + 1. Ensure "Enabled Git access protocols" is set to "Both SSH and HTTP(S)". + 1. Follow [Fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md) on both primary and secondary sites. +1. If not using Git over SSH, then set "Enabled Git access protocols" to "Only HTTP(S)". ### Step 6. Verify proper functioning of the **secondary** site @@ -366,6 +370,9 @@ former is ideal for replicating data belonging to a subset of users, while the latter is more suited to progressively rolling out Geo to a large GitLab instance. +NOTE: +Geo's synchronization logic is outlined in the [documentation](../index.md). Both the solution and the documentation is subject to change from time to time. You must independently determine your legal obligations in regard to privacy and cybersecurity laws, and applicable trade control law on an ongoing basis. + Selective synchronization: 1. Does not restrict permissions from **secondary** sites. diff --git a/doc/administration/geo/replication/container_registry.md b/doc/administration/geo/replication/container_registry.md index 88ca8781dc3..66c67f29c1c 100644 --- a/doc/administration/geo/replication/container_registry.md +++ b/doc/administration/geo/replication/container_registry.md @@ -7,7 +7,7 @@ type: howto # Container Registry for a secondary site **(PREMIUM SELF)** -You can set up a Container Registry on your **secondary** Geo site that mirrors the one on the **primary** Geo site. +You can set up a Container Registry on your **secondary** Geo site that mirrors the one on the **primary** Geo site. NOTE: The Container Registry replication is used only for disaster recovery purposes. We do not recommend @@ -76,7 +76,9 @@ the **primary** site before following the next steps. We need to make Container Registry send notification events to the **primary** site. -1. SSH into your GitLab **primary** server and login as root: +For each application and Sidekiq node on the **primary** site: + +1. SSH into the node and login as the `root` user: ```shell sudo -i @@ -105,17 +107,15 @@ We need to make Container Registry send notification events to the that starts with a letter. You can generate one with `< /dev/urandom tr -dc _A-Z-a-z-0-9 | head -c 32 | sed "s/^[0-9]*//"; echo` NOTE: - If you use an external Registry (not the one integrated with GitLab), you must add - these settings to its configuration yourself. In this case, you also have to specify - notification secret in `registry.notification_secret` section of + If you use an external Registry (not the one integrated with GitLab), you also have to specify + the notification secret (`registry['notification_secret']`) in the `/etc/gitlab/gitlab.rb` file. NOTE: - If you use GitLab HA, you also have to specify - the notification secret in `registry.notification_secret` section of + If you use GitLab HA, you also have to specify the notification secret (`registry['notification_secret']`) in `/etc/gitlab/gitlab.rb` file for every web node. -1. Reconfigure the **primary** node for the change to take effect: +1. Reconfigure each node: ```shell gitlab-ctl reconfigure diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index 27e1b990b2b..d6ce5ef5067 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -43,12 +43,12 @@ verification methods: | Blobs | CI job artifacts _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | Archived CI build traces _(file system)_ | Geo with API | _Not implemented_ | | Blobs | Archived CI build traces _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | -| Blobs | Container registry _(file system)_ | Geo with API/Docker API | _Not implemented_ | -| Blobs | Container registry _(object storage)_ | Geo with API/Managed/Docker API (*2*) | _Not implemented_ | +| Blobs | Container registry _(file system)_ | Geo with API/Docker API | _SHA256 checksum_ | +| Blobs | Container registry _(object storage)_ | Geo with API/Managed/Docker API (*2*) | _SHA256 checksum_ | | Blobs | Package registry _(file system)_ | Geo with API | SHA256 checksum | | Blobs | Package registry _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | -| Blobs | Infrastructure registry _(file system)_ | Geo with API | SHA256 checksum | -| Blobs | Infrastructure registry _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | Terraform Module Registry _(file system)_ | Geo with API | SHA256 checksum | +| Blobs | Terraform Module Registry _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | Versioned Terraform State _(file system)_ | Geo with API | SHA256 checksum | | Blobs | Versioned Terraform State _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | External Merge Request Diffs _(file system)_ | Geo with API | SHA256 checksum | @@ -139,7 +139,7 @@ We use PostgreSQL's own replication functionality to replicate data from the **p We use Redis both as a cache store and to hold persistent data for our background jobs system. Because both use-cases have data that are exclusive to the same Geo site, we don't replicate it between sites. -Elasticsearch is an optional database that for advanced searching capabilities. It can improve search +Elasticsearch is an optional database for advanced search. It can improve search in both source-code level, and user generated content in issues, merge requests, and discussions. Elasticsearch is not supported in Geo. @@ -192,17 +192,17 @@ successfully, you must replicate their data using some other means. |:--------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------|:---------------------------------------------------------------------------|:--------------------------------------------------------------------|:----------------------------------------------------------------|:------| |[Application data in PostgreSQL](../../postgresql/index.md) | **Yes** (10.2) | **Yes** (10.2) | N/A | N/A | | |[Project repository](../../../user/project/repository/index.md) | **Yes** (10.2) | **Yes** (10.7) | N/A | N/A | | -|[Project wiki repository](../../../user/project/wiki/index.md) | **Yes** (10.2) | **Yes** (10.7) | N/A | N/A | | +|[Project wiki repository](../../../user/project/wiki/index.md) | **Yes** (10.2)<sup>2</sup> | **Yes** (10.7)<sup>2</sup> | N/A | N/A | Migrated to [self-service framework](../../../development/geo/framework.md) in 15.11. See GitLab issue [#367925](https://gitlab.com/gitlab-org/gitlab/-/issues/367925) for more details.<br /><br />Behind feature flag geo_project_wiki_repository_replication, enabled by default in (15.11). | |[Group wiki repository](../../../user/project/wiki/group.md) | [**Yes** (13.10)](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | No | N/A | N/A | Behind feature flag `geo_group_wiki_repository_replication`, enabled by default. | |[Uploads](../../uploads.md) | **Yes** (10.2) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Replication is behind the feature flag `geo_upload_replication`, enabled by default. Verification was behind the feature flag `geo_upload_verification`, removed in 14.8. | |[LFS objects](../../lfs/index.md) | **Yes** (10.2) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696).<br /><br />Replication is behind the feature flag `geo_lfs_object_replication`, enabled by default. Verification was behind the feature flag `geo_lfs_object_verification`, removed in 14.7. | |[Personal snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | N/A | N/A | | |[Project snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | N/A | N/A | | -|[CI job artifacts](../../../ci/pipelines/job_artifacts.md) | **Yes** (10.4) | **Yes** (14.10) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Verification is behind the feature flag `geo_job_artifact_replication`, enabled by default in 14.10. | +|[CI job artifacts](../../../ci/jobs/job_artifacts.md) | **Yes** (10.4) | **Yes** (14.10) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Verification is behind the feature flag `geo_job_artifact_replication`, enabled by default in 14.10. | |[CI Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/pipeline_artifact.rb) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Persists additional artifacts after a pipeline completes. | |[CI Secure Files](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/secure_file.rb) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [No](object_storage.md#verification-of-files-in-object-storage) | Verification is behind the feature flag `geo_ci_secure_file_replication`, enabled by default in 15.3. | -|[Container Registry](../../packages/container_registry.md) | **Yes** (12.3)* | No | No | No | See [instructions](container_registry.md) to set up the Container Registry replication. | -|[Infrastructure Registry](../../../user/packages/infrastructure_registry/index.md) | **Yes** (14.0) | **Yes** (14.0) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Container Registry](../../packages/container_registry.md) | **Yes** (12.3)<sup>1</sup> | **Yes** (15.10) | **Yes** (12.3)<sup>1</sup> | **Yes** (15.10) | See [instructions](container_registry.md) to set up the Container Registry replication. | +|[Terraform Module Registry](../../../user/packages/terraform_module_registry/index.md) | **Yes** (14.0) | **Yes** (14.0) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Behind feature flag `geo_package_file_replication`, enabled by default. | |[Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | N/A | N/A | Designs also require replication of LFS objects and Uploads. | |[Package Registry](../../../user/packages/package_registry/index.md) | **Yes** (13.2) | **Yes** (13.10) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Behind feature flag `geo_package_file_replication`, enabled by default. | |[Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | **Yes** (13.12) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Replication is behind the feature flag `geo_terraform_state_version_replication`, enabled by default. Verification was behind the feature flag `geo_terraform_state_version_verification`, which was removed in 14.0. | @@ -217,4 +217,6 @@ successfully, you must replicate their data using some other means. |[Dependency Proxy Images](../../../user/packages/dependency_proxy/index.md) | [**Yes** (15.7)](https://gitlab.com/groups/gitlab-org/-/epics/8833) | [**Yes** (15.7)](https://gitlab.com/groups/gitlab-org/-/epics/8833) | [**Yes** (15.7)](https://gitlab.com/groups/gitlab-org/-/epics/8833) | [No](object_storage.md#verification-of-files-in-object-storage) | | |[Vulnerability Export](../../../user/application_security/vulnerability_report/index.md#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | No | No | Not planned because they are ephemeral and sensitive information. They can be regenerated on demand. | -\* Migrated to [self-service framework](../../../development/geo/framework.md) in 15.5. See GitLab issue [#337436](https://gitlab.com/gitlab-org/gitlab/-/issues/337436) for more details. +<sup>1</sup> Migrated to [self-service framework](../../../development/geo/framework.md) in 15.5. See GitLab issue [#337436](https://gitlab.com/gitlab-org/gitlab/-/issues/337436) for more details. + +<sup>2</sup> Migrated to [self-service framework](../../../development/geo/framework.md) in 15.11. Behind feature flag `geo_project_wiki_repository_replication`, enabled by default. See GitLab issue [#367925](https://gitlab.com/gitlab-org/gitlab/-/issues/367925) for more details. diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md index a12dd8d9d68..cad3a396bfc 100644 --- a/doc/administration/geo/replication/geo_validation_tests.md +++ b/doc/administration/geo/replication/geo_validation_tests.md @@ -127,8 +127,7 @@ The following are PostgreSQL upgrade validation tests we performed. - Description: With PostgreSQL 12 available as an opt-in version in GitLab 13.3, we tested upgrading existing Geo installations from PostgreSQL 11 to 12. We also re-tested fresh installations of GitLab with Geo after fixes were made to support PostgreSQL 12. These tests were done using a - [nightly build](https://packages.gitlab.com/gitlab/nightly-builds/packages/ubuntu/bionic/gitlab-ee_13.3.6+rnightly.169516.d5209202-0_amd64.deb) - of GitLab 13.4. + nightly build of GitLab 13.4. - Outcome: Tests were successful for Geo deployments with a single database node on the primary and secondary. We encountered known issues with repmgr and Patroni managed PostgreSQL clusters on the Geo primary. Using PostgreSQL 12 with a database cluster on the primary is not recommended until the issues are resolved. diff --git a/doc/administration/geo/replication/img/adding_a_secondary_v13_3.png b/doc/administration/geo/replication/img/adding_a_secondary_v13_3.png Binary files differdeleted file mode 100644 index e43ace99666..00000000000 --- a/doc/administration/geo/replication/img/adding_a_secondary_v13_3.png +++ /dev/null diff --git a/doc/administration/geo/replication/img/adding_a_secondary_v15_8.png b/doc/administration/geo/replication/img/adding_a_secondary_v15_8.png Binary files differnew file mode 100644 index 00000000000..d7c99e6551e --- /dev/null +++ b/doc/administration/geo/replication/img/adding_a_secondary_v15_8.png diff --git a/doc/administration/geo/replication/single_sign_on.md b/doc/administration/geo/replication/single_sign_on.md index fc2f23552db..55e77d5657c 100644 --- a/doc/administration/geo/replication/single_sign_on.md +++ b/doc/administration/geo/replication/single_sign_on.md @@ -112,6 +112,15 @@ After configuring your SAML IdP to allow the secondary site's SAML callback URL, If you have configured SAML on the primary site correctly, then it should work on the secondary site without additional configuration. +## OpenID Connect + +If you use an [OpenID Connect (OIDC)](../../auth/oidc.md) OmniAuth provider, +in most cases, it should work without an issue: + +- **OIDC with Unified URL**: If you have configured OIDC on the primary site correctly, then it should work on the secondary site without additional configuration. +- **OIDC with separate URL with proxying disabled**: If you have configured OIDC on the primary site correctly, then it should work on the secondary site without additional configuration. +- **OIDC with separate URL with proxying enabled**: Geo with separate URL with proxying enabled does not support [OpenID Connect](../../auth/oidc.md). For more information, see [issue 396745](https://gitlab.com/gitlab-org/gitlab/-/issues/396745). + ## LDAP If you use LDAP on your **primary** site, you should also set up secondary LDAP servers on each **secondary** site. Otherwise, users cannot perform Git operations over HTTP(s) on the **secondary** site using HTTP basic authentication. However, users can still use Git with SSH and personal access tokens. diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 59a67fecfcd..a8736b3ed1d 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -42,7 +42,7 @@ to help identify if something is wrong:  -A site shows as "Unhealthy" if the site's status is more than 10 minutes old. In that case, try running the following in the [Rails console](../../operations/rails_console.md) on the affected site: +A site shows as "Unhealthy" if the site's status is more than 10 minutes old. In that case, try running the following in the [Rails console](../../operations/rails_console.md) on the affected secondary site: ```ruby Geo::MetricsUpdateWorker.new.perform @@ -52,7 +52,7 @@ If it raises an error, then the error is probably also preventing the jobs from If it successfully updates the status, then something may be wrong with Sidekiq. Is it running? Do the logs show errors? This job is supposed to be enqueued every minute. It takes an exclusive lease in Redis to ensure that only one of these jobs can run at a time. The primary site updates its status directly in the PostgreSQL database. Secondary sites send an HTTP Post request to the primary site with their status data. -A site also shows as "Unhealthy" if certain health checks fail. You can reveal the failure by running the following in the [Rails console](../../operations/rails_console.md) on the affected site: +A site also shows as "Unhealthy" if certain health checks fail. You can reveal the failure by running the following in the [Rails console](../../operations/rails_console.md) on the affected secondary site: ```ruby Gitlab::Geo::HealthCheck.new.perform_checks @@ -718,6 +718,8 @@ To solve this: ### Very large repositories never successfully synchronize on the **secondary** site +#### GitLab 10.1 and earlier + GitLab places a timeout on all repository clones, including project imports and Geo synchronization operations. If a fresh `git clone` of a repository on the **primary** takes more than the default three hours, you may be affected by this. @@ -740,6 +742,10 @@ add the following line to `/etc/gitlab/gitlab.rb`: This increases the timeout to four hours (14400 seconds). Choose a time long enough to accommodate a full clone of your largest repositories. +#### GitLab 10.2 and later + +Geo [replicates Git repositories over HTTPS](../index.md#how-it-works). GitLab does not place a timeout on these requests. If a Git repository is failing to replicate, with a consistent job execution time, then you should check for timeouts applied by external components such as load balancers. + ### New LFS objects are never replicated If new LFS objects are never replicated to secondary Geo sites, check the version of @@ -959,7 +965,7 @@ This behavior affects only the following data types through GitLab 14.6: | Package Registry | 13.10 | | CI Pipeline Artifacts | 13.11 | | Terraform State Versions | 13.12 | -| Infrastructure Registry | 14.0 | +| Infrastructure Registry (renamed to Terraform Module Registry in GitLab 15.11) | 14.0 | | External MR diffs | 14.6 | | LFS Objects | 14.6 | | Pages Deployments | 14.6 | @@ -1373,6 +1379,23 @@ The bug causes all wildcard domains (`.example.com`) to be ignored except for th You can have only one wildcard domain in the `no_proxy` list. +### Secondary site shows "Unhealthy" in UI after changing the value of `external_url` for the primary site + +If you have updated the value of `external_url` in `/etc/gitlab/gitlab.rb` for the primary site or changed the protocol from `http` to `https`, you may see that secondary sites are shown as `Unhealthy`. You may also find the following error in `geo.log`: + +```plaintext +"class": "Geo::NodeStatusRequestService", +... +"message": "Failed to Net::HTTP::Post to primary url: http://primary-site.gitlab.tld/api/v4/geo/status", + "error": "Failed to open TCP connection to <PRIMARY_IP_ADDRESS>:80 (Connection refused - connect(2) for \"<PRIMARY_ID_ADDRESS>\" port 80)" +``` + +In this case, make sure to update the changed URL on all your sites: + +1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, select **Admin > Geo > Sites**. +1. Change the URL and save the change. + ## Fixing non-PostgreSQL replication failures If you notice replication failures in `Admin > Geo > Sites` or the [Sync status Rake task](#sync-status-rake-task), you can try to resolve the failures with the following general steps: diff --git a/doc/administration/geo/replication/upgrading_the_geo_sites.md b/doc/administration/geo/replication/upgrading_the_geo_sites.md index 55395a55857..644232a2c9e 100644 --- a/doc/administration/geo/replication/upgrading_the_geo_sites.md +++ b/doc/administration/geo/replication/upgrading_the_geo_sites.md @@ -33,6 +33,7 @@ and all **secondary** sites: 1. SSH into each node of the **primary** site. 1. [Upgrade GitLab on the **primary** site](../../../update/package/index.md#upgrade-using-the-official-repositories). 1. Perform testing on the **primary** site, particularly if you paused replication in step 1 to protect DR. [There are some suggestions for post-upgrade testing](../../../update/plan_your_upgrade.md#pre-upgrade-and-post-upgrade-checks) in the upgrade documentation. +1. Ensure that the secrets in the `/etc/gitlab/gitlab-secrets.json` file of both the primary site and the secondary site are the same. The file must be the same on all of a site’s nodes. 1. SSH into each node of **secondary** sites. 1. [Upgrade GitLab on each **secondary** site](../../../update/package/index.md#upgrade-using-the-official-repositories). 1. If you paused replication in step 1, [resume replication on each **secondary**](../index.md#pausing-and-resuming-replication). diff --git a/doc/administration/geo/secondary_proxy/index.md b/doc/administration/geo/secondary_proxy/index.md index addf894f0a5..35ab1d8252c 100644 --- a/doc/administration/geo/secondary_proxy/index.md +++ b/doc/administration/geo/secondary_proxy/index.md @@ -165,6 +165,7 @@ It does not cover all data types. | LFS objects (using Git) | **{check-circle}** Yes | | Pages | **{dotted-circle}** No <sup>2</sup> | | Advanced search (using the web UI) | **{dotted-circle}** No | +| Container registry | **{dotted-circle}** No | 1. Git reads are served from the local secondary while pushes get proxied to the primary. Selective sync or cases where repositories don't exist locally on the Geo secondary throw a "not found" error. diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index 99f7b32be59..cf0c40dbbc5 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -21,7 +21,7 @@ NOTE: The stages of the setup process must be completed in the documented order. If not, [complete all prior stages](../setup/index.md#using-omnibus-gitlab) before proceeding. -Ensure the **secondary** site is running the same version of GitLab Enterprise Edition as the **primary** site. Confirm you have added the [premium or higher licenses](https://about.gitlab.com/pricing/) to your **primary** site. +Ensure the **secondary** site is running the same version of GitLab Enterprise Edition as the **primary** site. Confirm you have added a license for a [Premium or Ultimate subscription](https://about.gitlab.com/pricing/) to your **primary** site. Be sure to read and review all of these steps before you execute them in your testing or production environments. @@ -29,10 +29,10 @@ testing or production environments. ## Single instance database replication A single instance database replication is easier to set up and still provides the same Geo capabilities -as a clusterized alternative. It's useful for setups running on a single machine -or trying to evaluate Geo for a future clusterized installation. +as a clustered alternative. It's useful for setups running on a single machine +or trying to evaluate Geo for a future clustered installation. -A single instance can be expanded to a clusterized version using Patroni, which is recommended for a +A single instance can be expanded to a clustered version using Patroni, which is recommended for a highly available architecture. Follow the instructions below on how to set up PostgreSQL replication as a single instance database. @@ -151,6 +151,13 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ALTER USER gitlab_replicator WITH REPLICATION ENCRYPTED PASSWORD '<replication_password>'; ``` +1. Edit `/etc/gitlab/gitlab.rb` and set the role to `geo_primary_role` (for more information, see [Geo roles](https://docs.gitlab.com/omnibus/roles/#gitlab-geo-roles)): + + ```ruby + ## Geo Primary role + roles(['geo_primary_role']) + ``` + 1. Configure PostgreSQL to listen on network interfaces: For security reasons, PostgreSQL does not listen on any network interfaces @@ -211,17 +218,6 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ```ruby ## - ## Geo Primary role - ## - Configures Postgres settings for replication - ## - Prevents automatic upgrade of Postgres since it requires downtime of - ## streaming replication to Geo secondary sites - ## - Enables standard single-node GitLab services like NGINX, Puma, Redis, - ## or Sidekiq. If you are segregating services, then you will need to - ## explicitly disable unwanted services. - ## - roles(['geo_primary_role']) - - ## ## Primary address ## - replace '<primary_node_ip>' with the public or VPC address of your Geo primary node ## @@ -239,11 +235,13 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o # postgresql['max_replication_slots'] = 1 # Set this to be the number of Geo secondary nodes if you have more than one # postgresql['max_wal_senders'] = 10 # postgresql['wal_keep_segments'] = 10 + ``` - ## - ## Disable automatic database migrations temporarily - ## (until PostgreSQL is restarted and listening on the private address). - ## +1. Disable automatic database migrations temporarily until PostgreSQL is restarted and listening on the private address. + Edit `/etc/gitlab/gitlab.rb` and change the configuration to false: + + ```ruby + ## Disable automatic database migrations gitlab_rails['auto_migrate'] = false ``` @@ -402,6 +400,16 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o Ensure that the contents of `~gitlab-psql/data/server.crt` on the **primary** site match the contents of `~gitlab-psql/.postgresql/root.crt` on the **secondary** site. +1. Edit `/etc/gitlab/gitlab.rb` and set the role to `geo_secondary_role` (for more information, see [Geo roles](https://docs.gitlab.com/omnibus/roles/#gitlab-geo-roles)): + + ```ruby + ## + ## Geo Secondary role + ## - configure dependent flags automatically to enable Geo + ## + roles(['geo_secondary_role']) + ``` + 1. Configure PostgreSQL: This step is similar to how you configured the **primary** instance. @@ -412,12 +420,6 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ```ruby ## - ## Geo Secondary role - ## - configure dependent flags automatically to enable Geo - ## - roles(['geo_secondary_role']) - - ## ## Secondary address ## - replace '<secondary_site_ip>' with the public or VPC address of your Geo secondary site ## @@ -863,6 +865,7 @@ For each node running a Patroni instance on the secondary site: consul['configuration'] = { retry_join: %w[CONSUL_SECONDARY1_IP CONSUL_SECONDARY2_IP CONSUL_SECONDARY3_IP] } + consul['services'] = %w(postgresql) postgresql['md5_auth_cidr_addresses'] = [ 'PATRONI_SECONDARY1_IP/32', 'PATRONI_SECONDARY2_IP/32', 'PATRONI_SECONDARY3_IP/32', 'PATRONI_SECONDARY_PGBOUNCER/32', @@ -892,7 +895,7 @@ For each node running a Patroni instance on the secondary site: postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' postgresql['listen_address'] = '0.0.0.0' # You can use a public or VPC address here instead - gitlab_rails['dbpassword'] = 'POSTGRESQL_PASSWORD' + gitlab_rails['db_password'] = 'POSTGRESQL_PASSWORD' gitlab_rails['enable'] = true gitlab_rails['auto_migrate'] = false ``` @@ -937,6 +940,8 @@ Omnibus automatically configures a tracking database when `roles(['geo_secondary If you want to run this database in a highly available configuration, don't use the `geo_secondary_role` above. Instead, follow the instructions below. +If you want to run the Geo tracking database on a single node, see [Configure the Geo tracking database on the Geo secondary site](../replication/multiple_servers.md#step-3-configure-the-geo-tracking-database-on-the-geo-secondary-site). + A production-ready and secure setup for the tracking PostgreSQL DB requires at least three Consul nodes: two Patroni nodes, and one PgBouncer node on the secondary site. diff --git a/doc/administration/geo/setup/external_database.md b/doc/administration/geo/setup/external_database.md index 0fefc11f078..65ea2e6e5af 100644 --- a/doc/administration/geo/setup/external_database.md +++ b/doc/administration/geo/setup/external_database.md @@ -211,20 +211,20 @@ the tracking database on port 5432. 1. Set up PostgreSQL according to the [database requirements document](../../../install/requirements.md#database). -1. Set up a `gitlab_geo` user with a password of your choice, create the `gitlabhq_geo_production` database, and make the user an owner of the database. You can see an example of this setup in the [installation from source documentation](../../../install/installation.md#6-database). +1. Set up a `gitlab_geo` user with a password of your choice, create the `gitlabhq_geo_production` database, and make the user an owner of the database. You can see an example of this setup in the [installation from source documentation](../../../install/installation.md#7-database). 1. If you are **not** using a cloud-managed PostgreSQL database, ensure that your secondary site can communicate with your tracking database by manually changing the `pg_hba.conf` that is associated with your tracking database. Remember to restart PostgreSQL afterwards for the changes to take effect: - ```plaintext - ## - ## Geo Tracking Database Role - ## - pg_hba.conf - ## - host all all <trusted tracking IP>/32 md5 - host all all <trusted secondary IP>/32 md5 - ``` + ```plaintext + ## + ## Geo Tracking Database Role + ## - pg_hba.conf + ## + host all all <trusted tracking IP>/32 md5 + host all all <trusted secondary IP>/32 md5 + ``` 1. SSH into a GitLab **secondary** server and login as root: diff --git a/doc/administration/geo/setup/index.md b/doc/administration/geo/setup/index.md index c794b8ef219..da7e55509e7 100644 --- a/doc/administration/geo/setup/index.md +++ b/doc/administration/geo/setup/index.md @@ -7,33 +7,33 @@ type: howto # Setting up Geo **(PREMIUM SELF)** -These instructions assume you have a working instance of GitLab. They guide you through: +## Prerequisites -1. Making your existing instance the **primary** site. -1. Adding **secondary** sites. +- Two (or more) independently working GitLab sites: + - One GitLab site serves as the Geo **primary** site. Use the [GitLab reference architectures documentation](../../reference_architectures/index.md) to set this up. You can use different reference architecture sizes for each Geo site. If you already have a working GitLab instance that is in-use, it can be used as a **primary** site. + - The second GitLab site serves as the Geo **secondary** site. Use the [GitLab reference architectures documentation](../../reference_architectures/index.md) to set this up. It's a good idea to sign in and test it. However, be aware that **all of the data on the secondary are lost** as part of the process of replicating from the **primary** site. -You must use a [GitLab Premium](https://about.gitlab.com/pricing/) license or higher, -but you only need one license for all the sites. + NOTE: + Geo supports multiple secondaries. You can follow the same steps and make any changes accordingly. -WARNING: -The steps below should be followed in the order they appear. **Make sure the GitLab version is the same on all sites. Do not create an account or sign in to the new secondary.** +- Ensure the **primary** site has a [GitLab Premium or Ultimate](https://about.gitlab.com/pricing/) subscription to unlock Geo. You only need one license for all the sites. +- Confirm the [requirements for running Geo](../index.md#requirements-for-running-geo) are met by all sites. For example, sites must use the same GitLab version, and sites must be able to communicate with each other over certain ports. ## Using Omnibus GitLab If you installed GitLab using the Omnibus packages (highly recommended): -1. Confirm the [requirements for running Geo](../index.md#requirements-for-running-geo) are met. -1. [Install GitLab Enterprise Edition](https://about.gitlab.com/install/) on the nodes that serve as the **secondary** site. **Do not create an account or sign in** to the new **secondary** site. The **GitLab version must match** across primary and secondary sites. -1. [Add the GitLab License](../../../user/admin_area/license.md) on the **primary** site to unlock Geo. The license must be for [GitLab Premium](https://about.gitlab.com/pricing/) or higher. -1. [Confirm network connectivity](../index.md#firewall-rules) between the **primary** and **secondary** site. 1. [Set up the database replication](database.md) (`primary (read-write) <-> secondary (read-only)` topology). -1. [Configure fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md). This step is required and needs to be done on **both** the **primary** and **secondary** sites. 1. [Configure GitLab](../replication/configuration.md) to set the **primary** and **secondary** sites. 1. Optional: [Configure Object storage](../../object_storage.md) 1. Optional: [Configure a secondary LDAP server](../../auth/ldap/index.md) for the **secondary** sites. See [notes on LDAP](../index.md#ldap). 1. Optional: [Configure Geo secondary proxying](../secondary_proxy/index.md) to use a single, unified URL for all Geo sites. This step is recommended to accelerate most read requests while transparently proxying writes to the primary Geo site. 1. Follow the [Using a Geo Site](../replication/usage.md) guide. +## Using GitLab Charts + +[Configure the GitLab chart with GitLab Geo](https://docs.gitlab.com/charts/advanced/geo/). + ## Post-installation documentation After installing GitLab on the **secondary** sites and performing the initial configuration, see the [following documentation for post-installation information](../index.md#post-installation-documentation). |
