diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2020-07-20 15:26:25 +0300 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2020-07-20 15:26:25 +0300 |
commit | a09983ae35713f5a2bbb100981116d31ce99826e (patch) | |
tree | 2ee2af7bd104d57086db360a7e6d8c9d5d43667a /doc/administration/geo/replication | |
parent | 18c5ab32b738c0b6ecb4d0df3994000482f34bd8 (diff) |
Add latest changes from gitlab-org/gitlab@13-2-stable-ee
Diffstat (limited to 'doc/administration/geo/replication')
11 files changed, 242 insertions, 80 deletions
diff --git a/doc/administration/geo/replication/database.md b/doc/administration/geo/replication/database.md index 3f2d46ba457..02f51e79907 100644 --- a/doc/administration/geo/replication/database.md +++ b/doc/administration/geo/replication/database.md @@ -130,7 +130,8 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o connect to the **primary** node's database. For this reason, we need the address of each node. - NOTE: **Note:** For external PostgreSQL instances, see [additional instructions](external_database.md). + NOTE: **Note:** + For external PostgreSQL instances, see [additional instructions](external_database.md). If you are using a cloud provider, you can lookup the addresses for each Geo node through your cloud provider's management console. @@ -419,7 +420,8 @@ data before running `pg_basebackup`. 1. Execute the command below to start a backup/restore and begin the replication - CAUTION: **Warning:** Each Geo **secondary** node must have its own unique replication slot name. + CAUTION: **Warning:** + Each Geo **secondary** node must have its own unique replication slot name. Using the same slot name between two secondaries will break PostgreSQL replication. ```shell diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index f50da27d82f..5636ff79189 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -45,6 +45,8 @@ verification methods: | Blobs | Archived CI build traces _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | Container registry _(filesystem)_ | Geo with API/Docker API | _Not implemented_ | | Blobs | Container registry _(object storage)_ | Geo with API/Managed/Docker API (*2*) | _Not implemented_ | +| Blobs | Package registry _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | Package registry _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | - (*1*): Redis replication can be used as part of HA with Redis sentinel. It's not used between Geo nodes. - (*2*): Object storage replication can be performed by Geo or by your object storage provider/appliance @@ -124,41 +126,41 @@ these epics/issues: - [Unreplicated Data Types](https://gitlab.com/groups/gitlab-org/-/epics/893) - [Verify all replicated data](https://gitlab.com/groups/gitlab-org/-/epics/1430) -DANGER: **DANGER** +DANGER: **Danger:** Features not on this list, or with **No** in the **Replicated** column, are not replicated on the **secondary** node. Failing over without manually replicating data from those features will cause the data to be **lost**. If you wish to use those features on a **secondary** node, or to execute a failover successfully, you must replicate their data using some other means. -| Feature | Replicated | Verified | Notes | +| Feature | Replicated (added in GitLab version) | Verified (added in GitLab version) | Notes | |:---------------------------------------------------------------------|:---------------------------------------------------------|:--------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------| -| Application data in PostgreSQL | **Yes** | **Yes** | | -| Project repository | **Yes** | **Yes** | | -| Project wiki repository | **Yes** | **Yes** | | -| Project designs repository | **Yes** | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | | -| Uploads | **Yes** | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Verified only on transfer, or manually (*1*) | -| LFS objects | **Yes** | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8922) | Verified only on transfer, or manually (*1*). Unavailable for new LFS objects in 11.11.x and 12.0.x (*2*). | -| CI job artifacts (other than traces) | **Yes** | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Verified only manually (*1*) | -| Archived traces | **Yes** | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Verified only on transfer, or manually (*1*) | -| Personal snippets | **Yes** | **Yes** | | +| Application data in PostgreSQL | **Yes** (10.2) | **Yes** (10.2) | | +| Project repository | **Yes** (10.2) | **Yes** (10.7) | | +| Project wiki repository | **Yes** (10.2) | **Yes** (10.7) | | +| Project designs repository | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | | +| Uploads | **Yes** (10.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Verified only on transfer, or manually (*1*) | +| LFS objects | **Yes** (10.2) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8922) | Verified only on transfer, or manually (*1*). Unavailable for new LFS objects in 11.11.x and 12.0.x (*2*). | +| CI job artifacts (other than traces) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Verified only manually (*1*) | +| Archived traces | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Verified only on transfer, or manually (*1*) | +| Personal snippets | **Yes** (10.2) | **Yes** (10.2) | | | [Versioned snippets](../../../user/snippets.md#versioned-snippets) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2810) | | -| Project snippets | **Yes** | **Yes** | | -| Object pools for forked project deduplication | **Yes** | No | | -| [Server-side Git Hooks](../../custom_hooks.md) | No | No | | +| Project snippets | **Yes** (10.2) | **Yes** (10.2) | | +| Object pools for forked project deduplication | **Yes** | No | | +| [Server-side Git hooks](../../server_hooks.md) | No | No | | | [Elasticsearch integration](../../../integration/elasticsearch.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | | | [GitLab Pages](../../pages/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | | -| [Container Registry](../../packages/container_registry.md) | **Yes** | No | | -| [NPM Registry](../../../user/packages/npm_registry/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2346) | No | | -| [Maven Repository](../../../user/packages/maven_repository/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2346) | No | | -| [Conan Repository](../../../user/packages/conan_repository/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2346) | No | | -| [NuGet Repository](../../../user/packages/nuget_repository/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2346) | No | | -| [PyPi Repository](../../../user/packages/pypi_repository/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2554) | No | | -| [Composer Repository](../../../user/packages/composer_repository/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/3096) | No | | +| [Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | | +| [NPM Registry](../../../user/packages/npm_registry/index.md) | **Yes** (13.2) | No | | +| [Maven Repository](../../../user/packages/maven_repository/index.md) | **Yes** (13.2) | No | | +| [Conan Repository](../../../user/packages/conan_repository/index.md) | **Yes** (13.2) | No | | +| [NuGet Repository](../../../user/packages/nuget_repository/index.md) | **Yes** (13.2) | No | | +| [PyPi Repository](../../../user/packages/pypi_repository/index.md) | **Yes** (13.2) | No | | +| [Composer Repository](../../../user/packages/composer_repository/index.md) | **Yes** (13.2) | No | | | [External merge request diffs](../../merge_request_diffs.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/33817) | No | | | [Terraform State](../../terraform_state.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/3112)(*3*) | No | | -| [Vulnerability Export](../../../user/application_security/security_dashboard/#export-vulnerabilities-1) | [No](https://gitlab.com/groups/gitlab-org/-/epics/3111)(*3*) | No | | | -| Content in object storage | **Yes** | No | | +| [Vulnerability Export](../../../user/application_security/security_dashboard/#export-vulnerabilities) | [No](https://gitlab.com/groups/gitlab-org/-/epics/3111)(*3*) | No | | | +| Content in object storage | **Yes** (12.4) | No | | - (*1*): The integrity can be verified manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing diff --git a/doc/administration/geo/replication/disable_geo.md b/doc/administration/geo/replication/disable_geo.md new file mode 100644 index 00000000000..f53b4c1b796 --- /dev/null +++ b/doc/administration/geo/replication/disable_geo.md @@ -0,0 +1,93 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + +# Disabling Geo **(PREMIUM ONLY)** + +If you want to revert to a regular Omnibus setup after a test, or you have encountered a Disaster Recovery +situation and you want to disable Geo momentarily, you can use these instructions to disable your +Geo setup. + +There should be no functional difference between disabling Geo and having an active Geo setup with +no secondary Geo nodes if you remove them correctly. + +To disable Geo, follow these steps: + +1. [Remove all secondary Geo nodes](#remove-all-secondary-geo-nodes). +1. [Remove the primary node from the UI](#remove-the-primary-node-from-the-ui). +1. [Remove secondary replication slots](#remove-secondary-replication-slots). +1. [Remove Geo-related configuration](#remove-geo-related-configuration). +1. [(Optional) Revert PostgreSQL settings to use a password and listen on an IP](#optional-revert-postgresql-settings-to-use-a-password-and-listen-on-an-ip). + +## Remove all secondary Geo nodes + +To disable Geo, you need to first remove all your secondary Geo nodes, which means replication will not happen +anymore on these nodes. You can follow our docs to [remove your secondary Geo nodes](./remove_geo_node.md). + +If the current node that you want to keep using is a secondary node, you need to first promote it to primary. +You can use our steps on [how to promote a secondary node](../disaster_recovery/#step-3-promoting-a-secondary-node) +in order to do that. + +## Remove the primary node from the UI + +1. Go to **{admin}** **Admin Area >** **{location-dot}** **Geo** (`/admin/geo/nodes`). +1. Click the **Remove** button for the **primary** node. +1. Confirm by clicking **Remove** when the prompt appears. + +## Remove secondary replication slots + +To remove secondary replication slots, run one of the following queries on your primary +Geo node in a PostgreSQL console (`sudo gitlab-psql`): + +- If you already have a PostgreSQL cluster, drop individual replication slots by name to prevent + removing your secondary databases from the same cluster. You can use the following to get + all names and then drop each individual slot: + + ```sql + SELECT slot_name, slot_type, active FROM pg_replication_slots; -- view present replication slots + SELECT pg_drop_replication_slot('slot_name'); -- where slot_name is the one expected from above + ``` + +- To remove all secondary replication slots: + + ```sql + SELECT pg_drop_replication_slot(slot_name) FROM pg_replication_slots; + ``` + +## Remove Geo-related configuration + +1. SSH into your primary Geo node and log in as root: + + ```shell + sudo -i + ``` + +1. Edit `/etc/gitlab/gitlab.rb` and remove the Geo related configuration by + removing any lines that enabled `geo_primary_role`: + + ```ruby + ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. + geo_primary_role['enable'] = true + + ## In 11.5+ documentation, the role was enabled as follows. Remove this line. + roles ['geo_primary_role'] + ``` + +1. After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. + +## (Optional) Revert PostgreSQL settings to use a password and listen on an IP + +If you want to remove the PostgreSQL-specific settings and revert +to the defaults (using a socket instead), you can safely remove the following +lines from the `/etc/gitlab/gitlab.rb` file: + +```ruby +postgresql['sql_user_password'] = '...' +gitlab_rails['db_password'] = '...' +postgresql['listen_address'] = '...' +postgresql['md5_auth_cidr_addresses'] = ['...', '...'] +``` diff --git a/doc/administration/geo/replication/docker_registry.md b/doc/administration/geo/replication/docker_registry.md index bea6528dc9b..c34732cba67 100644 --- a/doc/administration/geo/replication/docker_registry.md +++ b/doc/administration/geo/replication/docker_registry.md @@ -18,7 +18,7 @@ Registry on the **primary** node, you can use the same storage for a **secondary Docker Registry as well. For more information, read the [Load balancing considerations](https://docs.docker.com/registry/deploying/#load-balancing-considerations) when deploying the Registry, and how to set up the storage driver for GitLab's -integrated [Container Registry](../../packages/container_registry.md#container-registry-storage-driver). +integrated [Container Registry](../../packages/container_registry.md#use-object-storage). ## Replicating Docker Registry diff --git a/doc/administration/geo/replication/external_database.md b/doc/administration/geo/replication/external_database.md index 491b3278ead..e85a82fa414 100644 --- a/doc/administration/geo/replication/external_database.md +++ b/doc/administration/geo/replication/external_database.md @@ -270,7 +270,8 @@ the tracking database on port 5432. query_exec "GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO ${GEO_DB_USER};" ``` - NOTE: **Note:** The script template above uses `gitlab-psql` as it's intended to be executed from the Geo machine, + NOTE: **Note:** + The script template above uses `gitlab-psql` as it's intended to be executed from the Geo machine, but you can change it to `psql` and run it from any machine that has access to the database. We also recommend using `psql` for AWS RDS. diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md index 7b186d15fae..0255e5c9883 100644 --- a/doc/administration/geo/replication/geo_validation_tests.md +++ b/doc/administration/geo/replication/geo_validation_tests.md @@ -16,20 +16,53 @@ This section contains a journal of recent validation tests and links to the rele The following are GitLab upgrade validation tests we performed. +### July 2020 + +[Upgrade Geo multi-node installation](https://gitlab.com/gitlab-org/gitlab/-/issues/225359): + +- Description: Tested upgrading from GitLab 12.10.12 to 13.0.10 package in a multi-node + configuration. As part of the issue to [Fix zero-downtime upgrade process/instructions for multi-node Geo deployments](https://gitlab.com/gitlab-org/gitlab/-/issues/22568), we monitored for downtime using the looping pipeline, HAProxy stats dashboards, and a script to log readiness status on both nodes. +- Outcome: Partial success because we observed downtime during the upgrade of the primary and secondary sites. +- Follow up issues/actions: + - [Investigate why `reconfigure` and `hup` cause downtime on multi-node Geo deployments](https://gitlab.com/gitlab-org/gitlab/-/issues/228898) + - [Geo multi-node deployment upgrade: investigate order when upgrading non-deploy nodes](https://gitlab.com/gitlab-org/gitlab/-/issues/228954) + +### June 2020 + +[Upgrade Geo multi-node installation](https://gitlab.com/gitlab-org/gitlab/-/issues/223284): + +- Description: Tested upgrading from GitLab 12.9.10 to 12.10.12 package in a multi-node + configuration. Monitored for downtime using the looping pipeline and HAProxy stats dashboards. +- Outcome: Partial success because we observed downtime during the upgrade of the primary and secondary sites. +- Follow up issues/actions: + - [Fix zero-downtime upgrade process/instructions for multi-node Geo deployments](https://gitlab.com/gitlab-org/gitlab/-/issues/225684) + - [Geo:check Rake task: Exclude AuthorizedKeysCommand check if node not running Puma/Unicorn](https://gitlab.com/gitlab-org/gitlab/-/issues/225454) + - [Update instructions in the next upgrade issue to include monitoring HAProxy dashboards](https://gitlab.com/gitlab-org/gitlab/-/issues/225359) + +[Upgrade Geo multi-node installation](https://gitlab.com/gitlab-org/gitlab/-/issues/208104): + +- Description: Tested upgrading from GitLab 12.8.1 to 12.9.10 package in a multi-node + configuration. +- Outcome: Partial success because we did not run the looping pipeline during the demo to validate + zero-downtime. +- Follow up issues: + - [Clarify hup Puma/Unicorn should include deploy node](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5460) + - [Investigate MR creation failure after upgrade to 12.9.10](https://gitlab.com/gitlab-org/gitlab/-/issues/223282) Closed as false positive. + ### February 2020 -[Upgrade Geo multi-server installation](https://gitlab.com/gitlab-org/gitlab/-/issues/201837): +[Upgrade Geo multi-node installation](https://gitlab.com/gitlab-org/gitlab/-/issues/201837): -- Description: Tested upgrading from GitLab 12.7.5 to the latest GitLab 12.8 package in a multi-server +- Description: Tested upgrading from GitLab 12.7.5 to the latest GitLab 12.8 package in a multi-node configuration. - Outcome: Partial success because we did not run the looping pipeline during the demo to monitor downtime. ### January 2020 -[Upgrade Geo multi-server installation](https://gitlab.com/gitlab-org/gitlab/-/issues/200085): +[Upgrade Geo multi-node installation](https://gitlab.com/gitlab-org/gitlab/-/issues/200085): -- Description: Tested upgrading from GitLab 12.6.x to the latest GitLab 12.7 package in a multi-server +- Description: Tested upgrading from GitLab 12.6.x to the latest GitLab 12.7 package in a multi-node configuration. - Outcome: Upgrade test was successful. - Follow up issues: @@ -37,16 +70,16 @@ The following are GitLab upgrade validation tests we performed. - [Add more logging to Geo end-to-end tests](https://gitlab.com/gitlab-org/gitlab/-/issues/201830). - [Excess service restarts during zero-downtime upgrade](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5047). -[Upgrade Geo multi-server installation](https://gitlab.com/gitlab-org/gitlab/-/issues/199836): +[Upgrade Geo multi-node installation](https://gitlab.com/gitlab-org/gitlab/-/issues/199836): -- Description: Tested upgrading from GitLab 12.5.7 to GitLab 12.6.6 in a multi-server configuration. +- Description: Tested upgrading from GitLab 12.5.7 to GitLab 12.6.6 in a multi-node configuration. - Outcome: Upgrade test was successful. - Follow up issue: [Update documentation for zero-downtime upgrades to ensure deploy node it not in use](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5046). -[Upgrade Geo multi-server installation](https://gitlab.com/gitlab-org/gitlab/-/issues/37044): +[Upgrade Geo multi-node installation](https://gitlab.com/gitlab-org/gitlab/-/issues/37044): -- Description: Tested upgrading from GitLab 12.4.x to the latest GitLab 12.5 package in a multi-server +- Description: Tested upgrading from GitLab 12.4.x to the latest GitLab 12.5 package in a multi-node configuration. - Outcome: Upgrade test was successful. - Follow up issues: @@ -55,17 +88,17 @@ The following are GitLab upgrade validation tests we performed. ### October 2019 -[Upgrade Geo multi-server installation](https://gitlab.com/gitlab-org/gitlab/-/issues/35262): +[Upgrade Geo multi-node installation](https://gitlab.com/gitlab-org/gitlab/-/issues/35262): -- Description: Tested upgrading from GitLab 12.3.5 to GitLab 12.4.1 in a multi-server configuration. +- Description: Tested upgrading from GitLab 12.3.5 to GitLab 12.4.1 in a multi-node configuration. - Outcome: Upgrade test was successful. -[Upgrade Geo multi-server installation](https://gitlab.com/gitlab-org/gitlab/-/issues/32437): +[Upgrade Geo multi-node installation](https://gitlab.com/gitlab-org/gitlab/-/issues/32437): - Description: Tested upgrading from GitLab 12.2.8 to GitLab 12.3.5. - Outcome: Upgrade test was successful. -[Upgrade Geo multi-server installation](https://gitlab.com/gitlab-org/gitlab/-/issues/32435): +[Upgrade Geo multi-node installation](https://gitlab.com/gitlab-org/gitlab/-/issues/32435): - Description: Tested upgrading from GitLab 12.1.9 to GitLab 12.2.8. - Outcome: Partial success due to possible misconfiguration issues. @@ -80,7 +113,7 @@ The following are PostgreSQL upgrade validation tests we performed. - Description: Prior to making PostgreSQL 11 the default version of PostgreSQL in GitLab 12.10, we tested upgrading to PostgreSQL 11 in Geo deployments on GitLab 12.9. -- Outcome: Partially successful. Issues were discovered in multi-server configurations with a separate +- Outcome: Partially successful. Issues were discovered in multi-node configurations with a separate tracking database and concerns were raised about allowing automatic upgrades when Geo enabled. - Follow up issues: - [`replicate-geo-database` incorrectly tries to back up repositories](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5241). @@ -102,6 +135,6 @@ The following are PostgreSQL upgrade validation tests we performed. various upgrade scenarios from GitLab 11.11.5 through to GitLab 12.1.8. - Outcome: Multiple issues were found when upgrading and addressed in follow-up issues. - Follow up issues: - - [`gitlab-ctl` reconfigure fails on Redis node in multi-server Geo setup](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4706). - - [Geo multi-server upgrade from 12.0.9 to 12.1.9 does not upgrade PostgreSQL](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4705). - - [Refresh foreign tables fails on app server in multi-server setup after upgrade to 12.1.9](https://gitlab.com/gitlab-org/gitlab/-/issues/32119). + - [`gitlab-ctl` reconfigure fails on Redis node in multi-node Geo setup](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4706). + - [Geo multi-node upgrade from 12.0.9 to 12.1.9 does not upgrade PostgreSQL](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4705). + - [Refresh foreign tables fails on app server in multi-node setup after upgrade to 12.1.9](https://gitlab.com/gitlab-org/gitlab/-/issues/32119). diff --git a/doc/administration/geo/replication/index.md b/doc/administration/geo/replication/index.md index 5b4b476bfa8..51c831abaf3 100644 --- a/doc/administration/geo/replication/index.md +++ b/doc/administration/geo/replication/index.md @@ -9,7 +9,7 @@ type: howto > - Introduced in GitLab Enterprise Edition 8.9. > - Using Geo in combination with -> [multi-server architectures](../../reference_architectures/index.md) +> [multi-node architectures](../../reference_architectures/index.md) > is considered **Generally Available** (GA) in > [GitLab Premium](https://about.gitlab.com/pricing/) 10.4. @@ -213,9 +213,29 @@ For information on configuring Geo, see [Geo configuration](configuration.md). For information on how to update your Geo nodes to the latest GitLab version, see [Updating the Geo nodes](updating_the_geo_nodes.md). -### Configuring Geo for multiple servers +### Pausing and resuming replication -For information on configuring Geo for multiple servers, see [Geo for multiple servers](multiple_servers.md). +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35913) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2. + +In some circumstances, like during [upgrades](updating_the_geo_nodes.md) or a [planned failover](../disaster_recovery/planned_failover.md), it is desirable to pause replication between the primary and secondary. + +Pausing and resuming replication is done via a command line tool from the secondary node. + +**To Pause: (from secondary)** + +```shell +gitlab-ctl geo-replication-pause +``` + +**To Resume: (from secondary)** + +```shell +gitlab-ctl geo-replication-resume +``` + +### Configuring Geo for multiple nodes + +For information on configuring Geo for multiple nodes, see [Geo for multiple servers](multiple_servers.md). ### Configuring Geo with Object Storage @@ -245,6 +265,10 @@ For an example of how to set up a location-aware Git remote URL with AWS Route53 For more information on removing a Geo node, see [Removing **secondary** Geo nodes](remove_geo_node.md). +## Disable Geo + +To find out how to disable Geo, see [Disabling Geo](disable_geo.md). + ## Current limitations CAUTION: **Caution:** diff --git a/doc/administration/geo/replication/location_aware_git_url.md b/doc/administration/geo/replication/location_aware_git_url.md index 49c83ee1718..8b086e3ff5f 100644 --- a/doc/administration/geo/replication/location_aware_git_url.md +++ b/doc/administration/geo/replication/location_aware_git_url.md @@ -18,7 +18,7 @@ Though these instructions use [AWS Route53](https://aws.amazon.com/route53/), other services such as [Cloudflare](https://www.cloudflare.com/) could be used as well. -NOTE: **Note** +NOTE: **Note:** You can also use a load balancer to distribute web UI or API traffic to [multiple Geo **secondary** nodes](../../../user/admin_area/geo_nodes.md#multiple-secondary-nodes-behind-a-load-balancer). Importantly, the **primary** node cannot yet be included. See the feature request diff --git a/doc/administration/geo/replication/multiple_servers.md b/doc/administration/geo/replication/multiple_servers.md index 2747aaeb105..d8f04e07fb0 100644 --- a/doc/administration/geo/replication/multiple_servers.md +++ b/doc/administration/geo/replication/multiple_servers.md @@ -5,15 +5,15 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Geo for multiple servers **(PREMIUM ONLY)** +# Geo for multiple nodes **(PREMIUM ONLY)** This document describes a minimal reference architecture for running Geo -in a multi-server configuration. If your multi-server setup differs from the one +in a multi-node configuration. If your multi-node setup differs from the one described, it is possible to adapt these instructions to your needs. ## Architecture overview -![Geo multi-server diagram](../../high_availability/img/geo-ha-diagram.png) +![Geo multi-node diagram](../../high_availability/img/geo-ha-diagram.png) _[diagram source - GitLab employees only](https://docs.google.com/drawings/d/1z0VlizKiLNXVVVaERFwgsIOuEgjcUqDTWPdQYsE7Z4c/edit)_ @@ -30,36 +30,36 @@ The only external way to access the two Geo deployments is by HTTPS at NOTE: **Note:** The **primary** and **secondary** Geo deployments must be able to communicate to each other over HTTPS. -## Redis and PostgreSQL for multiple servers +## Redis and PostgreSQL for multiple nodes Geo supports: -- Redis and PostgreSQL on the **primary** node configured for multiple servers. -- Redis on **secondary** nodes configured for multiple servers. +- Redis and PostgreSQL on the **primary** node configured for multiple nodes. +- Redis on **secondary** nodes configured for multiple nodes. NOTE: **Note:** -Support for PostgreSQL on **secondary** nodes in multi-server configuration +Support for PostgreSQL on **secondary** nodes in multi-node configuration [is planned](https://gitlab.com/groups/gitlab-org/-/epics/2536). Because of the additional complexity involved in setting up this configuration -for PostgreSQL and Redis, it is not covered by this Geo multi-server documentation. +for PostgreSQL and Redis, it is not covered by this Geo multi-node documentation. -For more information about setting up a multi-server PostgreSQL cluster and Redis cluster using the omnibus package see the multi-server documentation for +For more information about setting up a multi-node PostgreSQL cluster and Redis cluster using the omnibus package see the multi-node documentation for [PostgreSQL](../../postgresql/replication_and_failover.md) and -[Redis](../../high_availability/redis.md), respectively. +[Redis](../../redis/replication_and_failover.md), respectively. NOTE: **Note:** It is possible to use cloud hosted services for PostgreSQL and Redis, but this is beyond the scope of this document. -## Prerequisites: Two working GitLab multi-server clusters +## Prerequisites: Two working GitLab multi-node clusters One cluster will serve as the **primary** node. Use the -[GitLab multi-server documentation](../../reference_architectures/index.md) to set this up. If +[GitLab multi-node documentation](../../reference_architectures/index.md) to set this up. If you already have a working GitLab instance that is in-use, it can be used as a **primary**. The second cluster will serve as the **secondary** node. Again, use the -[GitLab multi-server documentation](../../reference_architectures/index.md) to set this up. +[GitLab multi-node documentation](../../reference_architectures/index.md) to set this up. It's a good idea to log in and test it, however, note that its data will be wiped out as part of the process of replicating from the **primary**. @@ -90,12 +90,13 @@ The following steps enable a GitLab cluster to serve as the **primary** node. After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. -NOTE: **Note:** PostgreSQL and Redis should have already been disabled on the +NOTE: **Note:** +PostgreSQL and Redis should have already been disabled on the application servers, and connections from the application servers to those -services on the backend servers configured, during normal GitLab multi-server set up. See -multi-server configuration documentation for +services on the backend servers configured, during normal GitLab multi-node set up. See +multi-node configuration documentation for [PostgreSQL](../../postgresql/replication_and_failover.md#configuring-the-application-nodes) -and [Redis](../../high_availability/redis.md#example-configuration-for-the-gitlab-application). +and [Redis](../../redis/replication_and_failover.md#example-configuration-for-the-gitlab-application). ### Step 2: Configure the **primary** database @@ -110,7 +111,7 @@ and [Redis](../../high_availability/redis.md#example-configuration-for-the-gitla ## Configure a **secondary** node -A **secondary** cluster is similar to any other GitLab multi-server cluster, with two +A **secondary** cluster is similar to any other GitLab multi-node cluster, with two major differences: - The main PostgreSQL database is a read-only replica of the **primary** node's @@ -119,8 +120,8 @@ major differences: called the "tracking database", which tracks the synchronization state of various resources. -Therefore, we will set up the multi-server components one-by-one, and include deviations -from the normal multi-server setup. However, we highly recommend first configuring a +Therefore, we will set up the multi-node components one-by-one, and include deviations +from the normal multi-node setup. However, we highly recommend first configuring a brand-new cluster as if it were not part of a Geo setup so that it can be tested and verified as a working cluster. And only then should it be modified for use as a Geo **secondary**. This helps to separate problems that are related @@ -128,10 +129,10 @@ and are not related to Geo setup. ### Step 1: Configure the Redis and Gitaly services on the **secondary** node -Configure the following services, again using the non-Geo multi-server +Configure the following services, again using the non-Geo multi-node documentation: -- [Configuring Redis for GitLab](../../high_availability/redis.md) for multiple servers. +- [Configuring Redis for GitLab](../../redis/replication_and_failover.md#example-configuration-for-the-gitlab-application) for multiple nodes. - [Gitaly](../../high_availability/gitaly.md), which will store data that is synchronized from the **primary** node. @@ -141,8 +142,9 @@ recommended. ### Step 2: Configure the main read-only replica PostgreSQL database on the **secondary** node -NOTE: **Note:** The following documentation assumes the database will be run on -a single node only. Multi-server PostgreSQL on **secondary** nodes is +NOTE: **Note:** +The following documentation assumes the database will be run on +a single node only. Multi-node PostgreSQL on **secondary** nodes is [not currently supported](https://gitlab.com/groups/gitlab-org/-/epics/2536). Configure the [**secondary** database](database.md) as a read-only replica of @@ -206,7 +208,8 @@ If using an external PostgreSQL instance, refer also to ### Step 3: Configure the tracking database on the **secondary** node -NOTE: **Note:** This documentation assumes the tracking database will be run on +NOTE: **Note:** +This documentation assumes the tracking database will be run on only a single machine, rather than as a PostgreSQL cluster. Configure the tracking database. @@ -282,7 +285,7 @@ application services. These services are enabled selectively in the configuration. Configure the application servers following -[Configuring GitLab for multiple servers](../../high_availability/gitlab.md), then make the +[Configuring GitLab for multiple nodes](../../high_availability/gitlab.md), then make the following modifications: 1. Edit `/etc/gitlab/gitlab.rb` on each application server in the **secondary** @@ -370,13 +373,13 @@ application servers. In this topology, a load balancer is required at each geographic location to route traffic to the application servers. -See [Load Balancer for GitLab with multiple servers](../../high_availability/load_balancer.md) for +See [Load Balancer for GitLab with multiple nodes](../../high_availability/load_balancer.md) for more information. ### Step 6: Configure the backend application servers on the **secondary** node The minimal reference architecture diagram above shows all application services -running together on the same machines. However, for multiple servers we +running together on the same machines. However, for multiple nodes we [strongly recommend running all services separately](../../reference_architectures/index.md). For example, a Sidekiq server could be configured similarly to the frontend diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index b03a2dae971..c2f8aa35d2d 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Geo Troubleshooting **(PREMIUM ONLY)** +# Troubleshooting Geo **(PREMIUM ONLY)** Setting up Geo requires careful attention to details and sometimes it's easy to miss a step. @@ -452,13 +452,13 @@ to start again from scratch, there are a few steps that can help you: chown git:git /var/opt/gitlab/git-data/repositories ``` - TIP: **Tip** + TIP: **Tip:** You may want to remove the `/var/opt/gitlab/git-data/repositories.old` in the future as soon as you confirmed that you don't need it anymore, to save disk space. 1. _(Optional)_ Rename other data folders and create new ones - CAUTION: **Caution**: + CAUTION: **Caution:** You may still have files on the **secondary** node that have been removed from **primary** node but removal have not been reflected. If you skip this step, they will never be removed from this Geo node. @@ -701,7 +701,8 @@ To check the configuration: Description | ``` - NOTE: **Note:** Pay particular attention to the host and port under + NOTE: **Note:** + Pay particular attention to the host and port under FDW options. That configuration should point to the Geo secondary database. diff --git a/doc/administration/geo/replication/updating_the_geo_nodes.md b/doc/administration/geo/replication/updating_the_geo_nodes.md index 6c2778ad0fe..c0b1bc6688c 100644 --- a/doc/administration/geo/replication/updating_the_geo_nodes.md +++ b/doc/administration/geo/replication/updating_the_geo_nodes.md @@ -36,15 +36,18 @@ different steps. ## General update steps -NOTE: **Note:** These general update steps are not intended for [high-availability deployments](https://docs.gitlab.com/omnibus/update/README.html#multi-node--ha-deployment), and will cause downtime. If you want to avoid downtime, consider using [zero downtime updates](https://docs.gitlab.com/omnibus/update/README.html#zero-downtime-updates). +NOTE: **Note:** +These general update steps are not intended for [high-availability deployments](https://docs.gitlab.com/omnibus/update/README.html#multi-node--ha-deployment), and will cause downtime. If you want to avoid downtime, consider using [zero downtime updates](https://docs.gitlab.com/omnibus/update/README.html#zero-downtime-updates). To update the Geo nodes when a new GitLab version is released, update **primary** and all **secondary** nodes: +1. **Optional:** [Pause replication on each **secondary** node.](./index.md#pausing-and-resuming-replication) 1. Log into the **primary** node. 1. [Update GitLab on the **primary** node using Omnibus](https://docs.gitlab.com/omnibus/update/README.html). 1. Log into each **secondary** node. 1. [Update GitLab on each **secondary** node using Omnibus](https://docs.gitlab.com/omnibus/update/README.html). +1. If you paused replication in step 1, [resume replication on each **secondary**](./index.md#pausing-and-resuming-replication) 1. [Test](#check-status-after-updating) **primary** and **secondary** nodes, and check version in each. ### Check status after updating |