Welcome to mirror list, hosted at ThFree Co, Russian Federation.

gitlab.com/gitlab-org/gitlab-foss.git - Unnamed repository; edit this file 'description' to name the repository.
summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorGitLab Bot <gitlab-bot@gitlab.com>2020-07-20 15:26:25 +0300
committerGitLab Bot <gitlab-bot@gitlab.com>2020-07-20 15:26:25 +0300
commita09983ae35713f5a2bbb100981116d31ce99826e (patch)
tree2ee2af7bd104d57086db360a7e6d8c9d5d43667a /doc/administration/geo/replication
parent18c5ab32b738c0b6ecb4d0df3994000482f34bd8 (diff)
Add latest changes from gitlab-org/gitlab@13-2-stable-ee
Diffstat (limited to 'doc/administration/geo/replication')
-rw-r--r--doc/administration/geo/replication/database.md6
-rw-r--r--doc/administration/geo/replication/datatypes.md48
-rw-r--r--doc/administration/geo/replication/disable_geo.md93
-rw-r--r--doc/administration/geo/replication/docker_registry.md2
-rw-r--r--doc/administration/geo/replication/external_database.md3
-rw-r--r--doc/administration/geo/replication/geo_validation_tests.md65
-rw-r--r--doc/administration/geo/replication/index.md30
-rw-r--r--doc/administration/geo/replication/location_aware_git_url.md2
-rw-r--r--doc/administration/geo/replication/multiple_servers.md59
-rw-r--r--doc/administration/geo/replication/troubleshooting.md9
-rw-r--r--doc/administration/geo/replication/updating_the_geo_nodes.md5
11 files changed, 242 insertions, 80 deletions
diff --git a/doc/administration/geo/replication/database.md b/doc/administration/geo/replication/database.md
index 3f2d46ba457..02f51e79907 100644
--- a/doc/administration/geo/replication/database.md
+++ b/doc/administration/geo/replication/database.md
@@ -130,7 +130,8 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
connect to the **primary** node's database. For this reason, we need the address of
each node.
- NOTE: **Note:** For external PostgreSQL instances, see [additional instructions](external_database.md).
+ NOTE: **Note:**
+ For external PostgreSQL instances, see [additional instructions](external_database.md).
If you are using a cloud provider, you can lookup the addresses for each
Geo node through your cloud provider's management console.
@@ -419,7 +420,8 @@ data before running `pg_basebackup`.
1. Execute the command below to start a backup/restore and begin the replication
- CAUTION: **Warning:** Each Geo **secondary** node must have its own unique replication slot name.
+ CAUTION: **Warning:**
+ Each Geo **secondary** node must have its own unique replication slot name.
Using the same slot name between two secondaries will break PostgreSQL replication.
```shell
diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md
index f50da27d82f..5636ff79189 100644
--- a/doc/administration/geo/replication/datatypes.md
+++ b/doc/administration/geo/replication/datatypes.md
@@ -45,6 +45,8 @@ verification methods:
| Blobs | Archived CI build traces _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ |
| Blobs | Container registry _(filesystem)_ | Geo with API/Docker API | _Not implemented_ |
| Blobs | Container registry _(object storage)_ | Geo with API/Managed/Docker API (*2*) | _Not implemented_ |
+| Blobs | Package registry _(filesystem)_ | Geo with API | _Not implemented_ |
+| Blobs | Package registry _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ |
- (*1*): Redis replication can be used as part of HA with Redis sentinel. It's not used between Geo nodes.
- (*2*): Object storage replication can be performed by Geo or by your object storage provider/appliance
@@ -124,41 +126,41 @@ these epics/issues:
- [Unreplicated Data Types](https://gitlab.com/groups/gitlab-org/-/epics/893)
- [Verify all replicated data](https://gitlab.com/groups/gitlab-org/-/epics/1430)
-DANGER: **DANGER**
+DANGER: **Danger:**
Features not on this list, or with **No** in the **Replicated** column,
are not replicated on the **secondary** node. Failing over without manually
replicating data from those features will cause the data to be **lost**.
If you wish to use those features on a **secondary** node, or to execute a failover
successfully, you must replicate their data using some other means.
-| Feature | Replicated | Verified | Notes |
+| Feature | Replicated (added in GitLab version) | Verified (added in GitLab version) | Notes |
|:---------------------------------------------------------------------|:---------------------------------------------------------|:--------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------|
-| Application data in PostgreSQL | **Yes** | **Yes** | |
-| Project repository | **Yes** | **Yes** | |
-| Project wiki repository | **Yes** | **Yes** | |
-| Project designs repository | **Yes** | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | |
-| Uploads | **Yes** | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Verified only on transfer, or manually (*1*) |
-| LFS objects | **Yes** | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8922) | Verified only on transfer, or manually (*1*). Unavailable for new LFS objects in 11.11.x and 12.0.x (*2*). |
-| CI job artifacts (other than traces) | **Yes** | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Verified only manually (*1*) |
-| Archived traces | **Yes** | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Verified only on transfer, or manually (*1*) |
-| Personal snippets | **Yes** | **Yes** | |
+| Application data in PostgreSQL | **Yes** (10.2) | **Yes** (10.2) | |
+| Project repository | **Yes** (10.2) | **Yes** (10.7) | |
+| Project wiki repository | **Yes** (10.2) | **Yes** (10.7) | |
+| Project designs repository | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | |
+| Uploads | **Yes** (10.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Verified only on transfer, or manually (*1*) |
+| LFS objects | **Yes** (10.2) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8922) | Verified only on transfer, or manually (*1*). Unavailable for new LFS objects in 11.11.x and 12.0.x (*2*). |
+| CI job artifacts (other than traces) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Verified only manually (*1*) |
+| Archived traces | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Verified only on transfer, or manually (*1*) |
+| Personal snippets | **Yes** (10.2) | **Yes** (10.2) | |
| [Versioned snippets](../../../user/snippets.md#versioned-snippets) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2810) | |
-| Project snippets | **Yes** | **Yes** | |
-| Object pools for forked project deduplication | **Yes** | No | |
-| [Server-side Git Hooks](../../custom_hooks.md) | No | No | |
+| Project snippets | **Yes** (10.2) | **Yes** (10.2) | |
+| Object pools for forked project deduplication | **Yes** | No | |
+| [Server-side Git hooks](../../server_hooks.md) | No | No | |
| [Elasticsearch integration](../../../integration/elasticsearch.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | |
| [GitLab Pages](../../pages/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | |
-| [Container Registry](../../packages/container_registry.md) | **Yes** | No | |
-| [NPM Registry](../../../user/packages/npm_registry/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2346) | No | |
-| [Maven Repository](../../../user/packages/maven_repository/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2346) | No | |
-| [Conan Repository](../../../user/packages/conan_repository/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2346) | No | |
-| [NuGet Repository](../../../user/packages/nuget_repository/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2346) | No | |
-| [PyPi Repository](../../../user/packages/pypi_repository/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2554) | No | |
-| [Composer Repository](../../../user/packages/composer_repository/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/3096) | No | |
+| [Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | |
+| [NPM Registry](../../../user/packages/npm_registry/index.md) | **Yes** (13.2) | No | |
+| [Maven Repository](../../../user/packages/maven_repository/index.md) | **Yes** (13.2) | No | |
+| [Conan Repository](../../../user/packages/conan_repository/index.md) | **Yes** (13.2) | No | |
+| [NuGet Repository](../../../user/packages/nuget_repository/index.md) | **Yes** (13.2) | No | |
+| [PyPi Repository](../../../user/packages/pypi_repository/index.md) | **Yes** (13.2) | No | |
+| [Composer Repository](../../../user/packages/composer_repository/index.md) | **Yes** (13.2) | No | |
| [External merge request diffs](../../merge_request_diffs.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/33817) | No | |
| [Terraform State](../../terraform_state.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/3112)(*3*) | No | |
-| [Vulnerability Export](../../../user/application_security/security_dashboard/#export-vulnerabilities-1) | [No](https://gitlab.com/groups/gitlab-org/-/epics/3111)(*3*) | No | | |
-| Content in object storage | **Yes** | No | |
+| [Vulnerability Export](../../../user/application_security/security_dashboard/#export-vulnerabilities) | [No](https://gitlab.com/groups/gitlab-org/-/epics/3111)(*3*) | No | | |
+| Content in object storage | **Yes** (12.4) | No | |
- (*1*): The integrity can be verified manually using
[Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing
diff --git a/doc/administration/geo/replication/disable_geo.md b/doc/administration/geo/replication/disable_geo.md
new file mode 100644
index 00000000000..f53b4c1b796
--- /dev/null
+++ b/doc/administration/geo/replication/disable_geo.md
@@ -0,0 +1,93 @@
+---
+stage: Enablement
+group: Geo
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+type: howto
+---
+
+# Disabling Geo **(PREMIUM ONLY)**
+
+If you want to revert to a regular Omnibus setup after a test, or you have encountered a Disaster Recovery
+situation and you want to disable Geo momentarily, you can use these instructions to disable your
+Geo setup.
+
+There should be no functional difference between disabling Geo and having an active Geo setup with
+no secondary Geo nodes if you remove them correctly.
+
+To disable Geo, follow these steps:
+
+1. [Remove all secondary Geo nodes](#remove-all-secondary-geo-nodes).
+1. [Remove the primary node from the UI](#remove-the-primary-node-from-the-ui).
+1. [Remove secondary replication slots](#remove-secondary-replication-slots).
+1. [Remove Geo-related configuration](#remove-geo-related-configuration).
+1. [(Optional) Revert PostgreSQL settings to use a password and listen on an IP](#optional-revert-postgresql-settings-to-use-a-password-and-listen-on-an-ip).
+
+## Remove all secondary Geo nodes
+
+To disable Geo, you need to first remove all your secondary Geo nodes, which means replication will not happen
+anymore on these nodes. You can follow our docs to [remove your secondary Geo nodes](./remove_geo_node.md).
+
+If the current node that you want to keep using is a secondary node, you need to first promote it to primary.
+You can use our steps on [how to promote a secondary node](../disaster_recovery/#step-3-promoting-a-secondary-node)
+in order to do that.
+
+## Remove the primary node from the UI
+
+1. Go to **{admin}** **Admin Area >** **{location-dot}** **Geo** (`/admin/geo/nodes`).
+1. Click the **Remove** button for the **primary** node.
+1. Confirm by clicking **Remove** when the prompt appears.
+
+## Remove secondary replication slots
+
+To remove secondary replication slots, run one of the following queries on your primary
+Geo node in a PostgreSQL console (`sudo gitlab-psql`):
+
+- If you already have a PostgreSQL cluster, drop individual replication slots by name to prevent
+ removing your secondary databases from the same cluster. You can use the following to get
+ all names and then drop each individual slot:
+
+ ```sql
+ SELECT slot_name, slot_type, active FROM pg_replication_slots; -- view present replication slots
+ SELECT pg_drop_replication_slot('slot_name'); -- where slot_name is the one expected from above
+ ```
+
+- To remove all secondary replication slots:
+
+ ```sql
+ SELECT pg_drop_replication_slot(slot_name) FROM pg_replication_slots;
+ ```
+
+## Remove Geo-related configuration
+
+1. SSH into your primary Geo node and log in as root:
+
+ ```shell
+ sudo -i
+ ```
+
+1. Edit `/etc/gitlab/gitlab.rb` and remove the Geo related configuration by
+ removing any lines that enabled `geo_primary_role`:
+
+ ```ruby
+ ## In pre-11.5 documentation, the role was enabled as follows. Remove this line.
+ geo_primary_role['enable'] = true
+
+ ## In 11.5+ documentation, the role was enabled as follows. Remove this line.
+ roles ['geo_primary_role']
+ ```
+
+1. After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure)
+ for the changes to take effect.
+
+## (Optional) Revert PostgreSQL settings to use a password and listen on an IP
+
+If you want to remove the PostgreSQL-specific settings and revert
+to the defaults (using a socket instead), you can safely remove the following
+lines from the `/etc/gitlab/gitlab.rb` file:
+
+```ruby
+postgresql['sql_user_password'] = '...'
+gitlab_rails['db_password'] = '...'
+postgresql['listen_address'] = '...'
+postgresql['md5_auth_cidr_addresses'] = ['...', '...']
+```
diff --git a/doc/administration/geo/replication/docker_registry.md b/doc/administration/geo/replication/docker_registry.md
index bea6528dc9b..c34732cba67 100644
--- a/doc/administration/geo/replication/docker_registry.md
+++ b/doc/administration/geo/replication/docker_registry.md
@@ -18,7 +18,7 @@ Registry on the **primary** node, you can use the same storage for a **secondary
Docker Registry as well. For more information, read the
[Load balancing considerations](https://docs.docker.com/registry/deploying/#load-balancing-considerations)
when deploying the Registry, and how to set up the storage driver for GitLab's
-integrated [Container Registry](../../packages/container_registry.md#container-registry-storage-driver).
+integrated [Container Registry](../../packages/container_registry.md#use-object-storage).
## Replicating Docker Registry
diff --git a/doc/administration/geo/replication/external_database.md b/doc/administration/geo/replication/external_database.md
index 491b3278ead..e85a82fa414 100644
--- a/doc/administration/geo/replication/external_database.md
+++ b/doc/administration/geo/replication/external_database.md
@@ -270,7 +270,8 @@ the tracking database on port 5432.
query_exec "GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO ${GEO_DB_USER};"
```
- NOTE: **Note:** The script template above uses `gitlab-psql` as it's intended to be executed from the Geo machine,
+ NOTE: **Note:**
+ The script template above uses `gitlab-psql` as it's intended to be executed from the Geo machine,
but you can change it to `psql` and run it from any machine that has access to the database. We also recommend using
`psql` for AWS RDS.
diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md
index 7b186d15fae..0255e5c9883 100644
--- a/doc/administration/geo/replication/geo_validation_tests.md
+++ b/doc/administration/geo/replication/geo_validation_tests.md
@@ -16,20 +16,53 @@ This section contains a journal of recent validation tests and links to the rele
The following are GitLab upgrade validation tests we performed.
+### July 2020
+
+[Upgrade Geo multi-node installation](https://gitlab.com/gitlab-org/gitlab/-/issues/225359):
+
+- Description: Tested upgrading from GitLab 12.10.12 to 13.0.10 package in a multi-node
+ configuration. As part of the issue to [Fix zero-downtime upgrade process/instructions for multi-node Geo deployments](https://gitlab.com/gitlab-org/gitlab/-/issues/22568), we monitored for downtime using the looping pipeline, HAProxy stats dashboards, and a script to log readiness status on both nodes.
+- Outcome: Partial success because we observed downtime during the upgrade of the primary and secondary sites.
+- Follow up issues/actions:
+ - [Investigate why `reconfigure` and `hup` cause downtime on multi-node Geo deployments](https://gitlab.com/gitlab-org/gitlab/-/issues/228898)
+ - [Geo multi-node deployment upgrade: investigate order when upgrading non-deploy nodes](https://gitlab.com/gitlab-org/gitlab/-/issues/228954)
+
+### June 2020
+
+[Upgrade Geo multi-node installation](https://gitlab.com/gitlab-org/gitlab/-/issues/223284):
+
+- Description: Tested upgrading from GitLab 12.9.10 to 12.10.12 package in a multi-node
+ configuration. Monitored for downtime using the looping pipeline and HAProxy stats dashboards.
+- Outcome: Partial success because we observed downtime during the upgrade of the primary and secondary sites.
+- Follow up issues/actions:
+ - [Fix zero-downtime upgrade process/instructions for multi-node Geo deployments](https://gitlab.com/gitlab-org/gitlab/-/issues/225684)
+ - [Geo:check Rake task: Exclude AuthorizedKeysCommand check if node not running Puma/Unicorn](https://gitlab.com/gitlab-org/gitlab/-/issues/225454)
+ - [Update instructions in the next upgrade issue to include monitoring HAProxy dashboards](https://gitlab.com/gitlab-org/gitlab/-/issues/225359)
+
+[Upgrade Geo multi-node installation](https://gitlab.com/gitlab-org/gitlab/-/issues/208104):
+
+- Description: Tested upgrading from GitLab 12.8.1 to 12.9.10 package in a multi-node
+ configuration.
+- Outcome: Partial success because we did not run the looping pipeline during the demo to validate
+ zero-downtime.
+- Follow up issues:
+ - [Clarify hup Puma/Unicorn should include deploy node](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5460)
+ - [Investigate MR creation failure after upgrade to 12.9.10](https://gitlab.com/gitlab-org/gitlab/-/issues/223282) Closed as false positive.
+
### February 2020
-[Upgrade Geo multi-server installation](https://gitlab.com/gitlab-org/gitlab/-/issues/201837):
+[Upgrade Geo multi-node installation](https://gitlab.com/gitlab-org/gitlab/-/issues/201837):
-- Description: Tested upgrading from GitLab 12.7.5 to the latest GitLab 12.8 package in a multi-server
+- Description: Tested upgrading from GitLab 12.7.5 to the latest GitLab 12.8 package in a multi-node
configuration.
- Outcome: Partial success because we did not run the looping pipeline during the demo to monitor
downtime.
### January 2020
-[Upgrade Geo multi-server installation](https://gitlab.com/gitlab-org/gitlab/-/issues/200085):
+[Upgrade Geo multi-node installation](https://gitlab.com/gitlab-org/gitlab/-/issues/200085):
-- Description: Tested upgrading from GitLab 12.6.x to the latest GitLab 12.7 package in a multi-server
+- Description: Tested upgrading from GitLab 12.6.x to the latest GitLab 12.7 package in a multi-node
configuration.
- Outcome: Upgrade test was successful.
- Follow up issues:
@@ -37,16 +70,16 @@ The following are GitLab upgrade validation tests we performed.
- [Add more logging to Geo end-to-end tests](https://gitlab.com/gitlab-org/gitlab/-/issues/201830).
- [Excess service restarts during zero-downtime upgrade](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5047).
-[Upgrade Geo multi-server installation](https://gitlab.com/gitlab-org/gitlab/-/issues/199836):
+[Upgrade Geo multi-node installation](https://gitlab.com/gitlab-org/gitlab/-/issues/199836):
-- Description: Tested upgrading from GitLab 12.5.7 to GitLab 12.6.6 in a multi-server configuration.
+- Description: Tested upgrading from GitLab 12.5.7 to GitLab 12.6.6 in a multi-node configuration.
- Outcome: Upgrade test was successful.
- Follow up issue:
[Update documentation for zero-downtime upgrades to ensure deploy node it not in use](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5046).
-[Upgrade Geo multi-server installation](https://gitlab.com/gitlab-org/gitlab/-/issues/37044):
+[Upgrade Geo multi-node installation](https://gitlab.com/gitlab-org/gitlab/-/issues/37044):
-- Description: Tested upgrading from GitLab 12.4.x to the latest GitLab 12.5 package in a multi-server
+- Description: Tested upgrading from GitLab 12.4.x to the latest GitLab 12.5 package in a multi-node
configuration.
- Outcome: Upgrade test was successful.
- Follow up issues:
@@ -55,17 +88,17 @@ The following are GitLab upgrade validation tests we performed.
### October 2019
-[Upgrade Geo multi-server installation](https://gitlab.com/gitlab-org/gitlab/-/issues/35262):
+[Upgrade Geo multi-node installation](https://gitlab.com/gitlab-org/gitlab/-/issues/35262):
-- Description: Tested upgrading from GitLab 12.3.5 to GitLab 12.4.1 in a multi-server configuration.
+- Description: Tested upgrading from GitLab 12.3.5 to GitLab 12.4.1 in a multi-node configuration.
- Outcome: Upgrade test was successful.
-[Upgrade Geo multi-server installation](https://gitlab.com/gitlab-org/gitlab/-/issues/32437):
+[Upgrade Geo multi-node installation](https://gitlab.com/gitlab-org/gitlab/-/issues/32437):
- Description: Tested upgrading from GitLab 12.2.8 to GitLab 12.3.5.
- Outcome: Upgrade test was successful.
-[Upgrade Geo multi-server installation](https://gitlab.com/gitlab-org/gitlab/-/issues/32435):
+[Upgrade Geo multi-node installation](https://gitlab.com/gitlab-org/gitlab/-/issues/32435):
- Description: Tested upgrading from GitLab 12.1.9 to GitLab 12.2.8.
- Outcome: Partial success due to possible misconfiguration issues.
@@ -80,7 +113,7 @@ The following are PostgreSQL upgrade validation tests we performed.
- Description: Prior to making PostgreSQL 11 the default version of PostgreSQL in GitLab 12.10, we
tested upgrading to PostgreSQL 11 in Geo deployments on GitLab 12.9.
-- Outcome: Partially successful. Issues were discovered in multi-server configurations with a separate
+- Outcome: Partially successful. Issues were discovered in multi-node configurations with a separate
tracking database and concerns were raised about allowing automatic upgrades when Geo enabled.
- Follow up issues:
- [`replicate-geo-database` incorrectly tries to back up repositories](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5241).
@@ -102,6 +135,6 @@ The following are PostgreSQL upgrade validation tests we performed.
various upgrade scenarios from GitLab 11.11.5 through to GitLab 12.1.8.
- Outcome: Multiple issues were found when upgrading and addressed in follow-up issues.
- Follow up issues:
- - [`gitlab-ctl` reconfigure fails on Redis node in multi-server Geo setup](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4706).
- - [Geo multi-server upgrade from 12.0.9 to 12.1.9 does not upgrade PostgreSQL](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4705).
- - [Refresh foreign tables fails on app server in multi-server setup after upgrade to 12.1.9](https://gitlab.com/gitlab-org/gitlab/-/issues/32119).
+ - [`gitlab-ctl` reconfigure fails on Redis node in multi-node Geo setup](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4706).
+ - [Geo multi-node upgrade from 12.0.9 to 12.1.9 does not upgrade PostgreSQL](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4705).
+ - [Refresh foreign tables fails on app server in multi-node setup after upgrade to 12.1.9](https://gitlab.com/gitlab-org/gitlab/-/issues/32119).
diff --git a/doc/administration/geo/replication/index.md b/doc/administration/geo/replication/index.md
index 5b4b476bfa8..51c831abaf3 100644
--- a/doc/administration/geo/replication/index.md
+++ b/doc/administration/geo/replication/index.md
@@ -9,7 +9,7 @@ type: howto
> - Introduced in GitLab Enterprise Edition 8.9.
> - Using Geo in combination with
-> [multi-server architectures](../../reference_architectures/index.md)
+> [multi-node architectures](../../reference_architectures/index.md)
> is considered **Generally Available** (GA) in
> [GitLab Premium](https://about.gitlab.com/pricing/) 10.4.
@@ -213,9 +213,29 @@ For information on configuring Geo, see [Geo configuration](configuration.md).
For information on how to update your Geo nodes to the latest GitLab version, see [Updating the Geo nodes](updating_the_geo_nodes.md).
-### Configuring Geo for multiple servers
+### Pausing and resuming replication
-For information on configuring Geo for multiple servers, see [Geo for multiple servers](multiple_servers.md).
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35913) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2.
+
+In some circumstances, like during [upgrades](updating_the_geo_nodes.md) or a [planned failover](../disaster_recovery/planned_failover.md), it is desirable to pause replication between the primary and secondary.
+
+Pausing and resuming replication is done via a command line tool from the secondary node.
+
+**To Pause: (from secondary)**
+
+```shell
+gitlab-ctl geo-replication-pause
+```
+
+**To Resume: (from secondary)**
+
+```shell
+gitlab-ctl geo-replication-resume
+```
+
+### Configuring Geo for multiple nodes
+
+For information on configuring Geo for multiple nodes, see [Geo for multiple servers](multiple_servers.md).
### Configuring Geo with Object Storage
@@ -245,6 +265,10 @@ For an example of how to set up a location-aware Git remote URL with AWS Route53
For more information on removing a Geo node, see [Removing **secondary** Geo nodes](remove_geo_node.md).
+## Disable Geo
+
+To find out how to disable Geo, see [Disabling Geo](disable_geo.md).
+
## Current limitations
CAUTION: **Caution:**
diff --git a/doc/administration/geo/replication/location_aware_git_url.md b/doc/administration/geo/replication/location_aware_git_url.md
index 49c83ee1718..8b086e3ff5f 100644
--- a/doc/administration/geo/replication/location_aware_git_url.md
+++ b/doc/administration/geo/replication/location_aware_git_url.md
@@ -18,7 +18,7 @@ Though these instructions use [AWS Route53](https://aws.amazon.com/route53/),
other services such as [Cloudflare](https://www.cloudflare.com/) could be used
as well.
-NOTE: **Note**
+NOTE: **Note:**
You can also use a load balancer to distribute web UI or API traffic to
[multiple Geo **secondary** nodes](../../../user/admin_area/geo_nodes.md#multiple-secondary-nodes-behind-a-load-balancer).
Importantly, the **primary** node cannot yet be included. See the feature request
diff --git a/doc/administration/geo/replication/multiple_servers.md b/doc/administration/geo/replication/multiple_servers.md
index 2747aaeb105..d8f04e07fb0 100644
--- a/doc/administration/geo/replication/multiple_servers.md
+++ b/doc/administration/geo/replication/multiple_servers.md
@@ -5,15 +5,15 @@ info: To determine the technical writer assigned to the Stage/Group associated w
type: howto
---
-# Geo for multiple servers **(PREMIUM ONLY)**
+# Geo for multiple nodes **(PREMIUM ONLY)**
This document describes a minimal reference architecture for running Geo
-in a multi-server configuration. If your multi-server setup differs from the one
+in a multi-node configuration. If your multi-node setup differs from the one
described, it is possible to adapt these instructions to your needs.
## Architecture overview
-![Geo multi-server diagram](../../high_availability/img/geo-ha-diagram.png)
+![Geo multi-node diagram](../../high_availability/img/geo-ha-diagram.png)
_[diagram source - GitLab employees only](https://docs.google.com/drawings/d/1z0VlizKiLNXVVVaERFwgsIOuEgjcUqDTWPdQYsE7Z4c/edit)_
@@ -30,36 +30,36 @@ The only external way to access the two Geo deployments is by HTTPS at
NOTE: **Note:**
The **primary** and **secondary** Geo deployments must be able to communicate to each other over HTTPS.
-## Redis and PostgreSQL for multiple servers
+## Redis and PostgreSQL for multiple nodes
Geo supports:
-- Redis and PostgreSQL on the **primary** node configured for multiple servers.
-- Redis on **secondary** nodes configured for multiple servers.
+- Redis and PostgreSQL on the **primary** node configured for multiple nodes.
+- Redis on **secondary** nodes configured for multiple nodes.
NOTE: **Note:**
-Support for PostgreSQL on **secondary** nodes in multi-server configuration
+Support for PostgreSQL on **secondary** nodes in multi-node configuration
[is planned](https://gitlab.com/groups/gitlab-org/-/epics/2536).
Because of the additional complexity involved in setting up this configuration
-for PostgreSQL and Redis, it is not covered by this Geo multi-server documentation.
+for PostgreSQL and Redis, it is not covered by this Geo multi-node documentation.
-For more information about setting up a multi-server PostgreSQL cluster and Redis cluster using the omnibus package see the multi-server documentation for
+For more information about setting up a multi-node PostgreSQL cluster and Redis cluster using the omnibus package see the multi-node documentation for
[PostgreSQL](../../postgresql/replication_and_failover.md) and
-[Redis](../../high_availability/redis.md), respectively.
+[Redis](../../redis/replication_and_failover.md), respectively.
NOTE: **Note:**
It is possible to use cloud hosted services for PostgreSQL and Redis, but this is beyond the scope of this document.
-## Prerequisites: Two working GitLab multi-server clusters
+## Prerequisites: Two working GitLab multi-node clusters
One cluster will serve as the **primary** node. Use the
-[GitLab multi-server documentation](../../reference_architectures/index.md) to set this up. If
+[GitLab multi-node documentation](../../reference_architectures/index.md) to set this up. If
you already have a working GitLab instance that is in-use, it can be used as a
**primary**.
The second cluster will serve as the **secondary** node. Again, use the
-[GitLab multi-server documentation](../../reference_architectures/index.md) to set this up.
+[GitLab multi-node documentation](../../reference_architectures/index.md) to set this up.
It's a good idea to log in and test it, however, note that its data will be
wiped out as part of the process of replicating from the **primary**.
@@ -90,12 +90,13 @@ The following steps enable a GitLab cluster to serve as the **primary** node.
After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect.
-NOTE: **Note:** PostgreSQL and Redis should have already been disabled on the
+NOTE: **Note:**
+PostgreSQL and Redis should have already been disabled on the
application servers, and connections from the application servers to those
-services on the backend servers configured, during normal GitLab multi-server set up. See
-multi-server configuration documentation for
+services on the backend servers configured, during normal GitLab multi-node set up. See
+multi-node configuration documentation for
[PostgreSQL](../../postgresql/replication_and_failover.md#configuring-the-application-nodes)
-and [Redis](../../high_availability/redis.md#example-configuration-for-the-gitlab-application).
+and [Redis](../../redis/replication_and_failover.md#example-configuration-for-the-gitlab-application).
### Step 2: Configure the **primary** database
@@ -110,7 +111,7 @@ and [Redis](../../high_availability/redis.md#example-configuration-for-the-gitla
## Configure a **secondary** node
-A **secondary** cluster is similar to any other GitLab multi-server cluster, with two
+A **secondary** cluster is similar to any other GitLab multi-node cluster, with two
major differences:
- The main PostgreSQL database is a read-only replica of the **primary** node's
@@ -119,8 +120,8 @@ major differences:
called the "tracking database", which tracks the synchronization state of
various resources.
-Therefore, we will set up the multi-server components one-by-one, and include deviations
-from the normal multi-server setup. However, we highly recommend first configuring a
+Therefore, we will set up the multi-node components one-by-one, and include deviations
+from the normal multi-node setup. However, we highly recommend first configuring a
brand-new cluster as if it were not part of a Geo setup so that it can be
tested and verified as a working cluster. And only then should it be modified
for use as a Geo **secondary**. This helps to separate problems that are related
@@ -128,10 +129,10 @@ and are not related to Geo setup.
### Step 1: Configure the Redis and Gitaly services on the **secondary** node
-Configure the following services, again using the non-Geo multi-server
+Configure the following services, again using the non-Geo multi-node
documentation:
-- [Configuring Redis for GitLab](../../high_availability/redis.md) for multiple servers.
+- [Configuring Redis for GitLab](../../redis/replication_and_failover.md#example-configuration-for-the-gitlab-application) for multiple nodes.
- [Gitaly](../../high_availability/gitaly.md), which will store data that is
synchronized from the **primary** node.
@@ -141,8 +142,9 @@ recommended.
### Step 2: Configure the main read-only replica PostgreSQL database on the **secondary** node
-NOTE: **Note:** The following documentation assumes the database will be run on
-a single node only. Multi-server PostgreSQL on **secondary** nodes is
+NOTE: **Note:**
+The following documentation assumes the database will be run on
+a single node only. Multi-node PostgreSQL on **secondary** nodes is
[not currently supported](https://gitlab.com/groups/gitlab-org/-/epics/2536).
Configure the [**secondary** database](database.md) as a read-only replica of
@@ -206,7 +208,8 @@ If using an external PostgreSQL instance, refer also to
### Step 3: Configure the tracking database on the **secondary** node
-NOTE: **Note:** This documentation assumes the tracking database will be run on
+NOTE: **Note:**
+This documentation assumes the tracking database will be run on
only a single machine, rather than as a PostgreSQL cluster.
Configure the tracking database.
@@ -282,7 +285,7 @@ application services. These services are enabled selectively in the
configuration.
Configure the application servers following
-[Configuring GitLab for multiple servers](../../high_availability/gitlab.md), then make the
+[Configuring GitLab for multiple nodes](../../high_availability/gitlab.md), then make the
following modifications:
1. Edit `/etc/gitlab/gitlab.rb` on each application server in the **secondary**
@@ -370,13 +373,13 @@ application servers.
In this topology, a load balancer is required at each geographic location to
route traffic to the application servers.
-See [Load Balancer for GitLab with multiple servers](../../high_availability/load_balancer.md) for
+See [Load Balancer for GitLab with multiple nodes](../../high_availability/load_balancer.md) for
more information.
### Step 6: Configure the backend application servers on the **secondary** node
The minimal reference architecture diagram above shows all application services
-running together on the same machines. However, for multiple servers we
+running together on the same machines. However, for multiple nodes we
[strongly recommend running all services separately](../../reference_architectures/index.md).
For example, a Sidekiq server could be configured similarly to the frontend
diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md
index b03a2dae971..c2f8aa35d2d 100644
--- a/doc/administration/geo/replication/troubleshooting.md
+++ b/doc/administration/geo/replication/troubleshooting.md
@@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w
type: howto
---
-# Geo Troubleshooting **(PREMIUM ONLY)**
+# Troubleshooting Geo **(PREMIUM ONLY)**
Setting up Geo requires careful attention to details and sometimes it's easy to
miss a step.
@@ -452,13 +452,13 @@ to start again from scratch, there are a few steps that can help you:
chown git:git /var/opt/gitlab/git-data/repositories
```
- TIP: **Tip**
+ TIP: **Tip:**
You may want to remove the `/var/opt/gitlab/git-data/repositories.old` in the future
as soon as you confirmed that you don't need it anymore, to save disk space.
1. _(Optional)_ Rename other data folders and create new ones
- CAUTION: **Caution**:
+ CAUTION: **Caution:**
You may still have files on the **secondary** node that have been removed from **primary** node but
removal have not been reflected. If you skip this step, they will never be removed
from this Geo node.
@@ -701,7 +701,8 @@ To check the configuration:
Description |
```
- NOTE: **Note:** Pay particular attention to the host and port under
+ NOTE: **Note:**
+ Pay particular attention to the host and port under
FDW options. That configuration should point to the Geo secondary
database.
diff --git a/doc/administration/geo/replication/updating_the_geo_nodes.md b/doc/administration/geo/replication/updating_the_geo_nodes.md
index 6c2778ad0fe..c0b1bc6688c 100644
--- a/doc/administration/geo/replication/updating_the_geo_nodes.md
+++ b/doc/administration/geo/replication/updating_the_geo_nodes.md
@@ -36,15 +36,18 @@ different steps.
## General update steps
-NOTE: **Note:** These general update steps are not intended for [high-availability deployments](https://docs.gitlab.com/omnibus/update/README.html#multi-node--ha-deployment), and will cause downtime. If you want to avoid downtime, consider using [zero downtime updates](https://docs.gitlab.com/omnibus/update/README.html#zero-downtime-updates).
+NOTE: **Note:**
+These general update steps are not intended for [high-availability deployments](https://docs.gitlab.com/omnibus/update/README.html#multi-node--ha-deployment), and will cause downtime. If you want to avoid downtime, consider using [zero downtime updates](https://docs.gitlab.com/omnibus/update/README.html#zero-downtime-updates).
To update the Geo nodes when a new GitLab version is released, update **primary**
and all **secondary** nodes:
+1. **Optional:** [Pause replication on each **secondary** node.](./index.md#pausing-and-resuming-replication)
1. Log into the **primary** node.
1. [Update GitLab on the **primary** node using Omnibus](https://docs.gitlab.com/omnibus/update/README.html).
1. Log into each **secondary** node.
1. [Update GitLab on each **secondary** node using Omnibus](https://docs.gitlab.com/omnibus/update/README.html).
+1. If you paused replication in step 1, [resume replication on each **secondary**](./index.md#pausing-and-resuming-replication)
1. [Test](#check-status-after-updating) **primary** and **secondary** nodes, and check version in each.
### Check status after updating