diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2021-06-16 21:25:58 +0300 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2021-06-16 21:25:58 +0300 |
commit | a5f4bba440d7f9ea47046a0a561d49adf0a1e6d4 (patch) | |
tree | fb69158581673816a8cd895f9d352dcb3c678b1e /doc/administration/geo | |
parent | d16b2e8639e99961de6ddc93909f3bb5c1445ba1 (diff) |
Add latest changes from gitlab-org/gitlab@14-0-stable-eev14.0.0-rc42
Diffstat (limited to 'doc/administration/geo')
-rw-r--r-- | doc/administration/geo/disaster_recovery/index.md | 27 | ||||
-rw-r--r-- | doc/administration/geo/disaster_recovery/planned_failover.md | 24 | ||||
-rw-r--r-- | doc/administration/geo/index.md | 25 | ||||
-rw-r--r-- | doc/administration/geo/replication/datatypes.md | 60 | ||||
-rw-r--r-- | doc/administration/geo/replication/faq.md | 10 | ||||
-rw-r--r-- | doc/administration/geo/replication/geo_validation_tests.md | 4 | ||||
-rw-r--r-- | doc/administration/geo/replication/multiple_servers.md | 48 | ||||
-rw-r--r-- | doc/administration/geo/replication/remove_geo_node.md | 1 | ||||
-rw-r--r-- | doc/administration/geo/replication/security_review.md | 2 | ||||
-rw-r--r-- | doc/administration/geo/replication/troubleshooting.md | 87 | ||||
-rw-r--r-- | doc/administration/geo/replication/using_a_geo_server.md | 1 | ||||
-rw-r--r-- | doc/administration/geo/replication/version_specific_updates.md | 10 | ||||
-rw-r--r-- | doc/administration/geo/setup/database.md | 232 | ||||
-rw-r--r-- | doc/administration/geo/setup/external_database.md | 41 |
14 files changed, 282 insertions, 290 deletions
diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index 7c6f4a32b57..f6f88e9b193 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -7,17 +7,14 @@ type: howto # Disaster Recovery (Geo) **(PREMIUM SELF)** -Geo replicates your database, your Git repositories, and few other assets. -We will support and replicate more data in the future, that will enable you to -failover with minimal effort, in a disaster situation. - -See [Geo limitations](../index.md#limitations) for more information. +Geo replicates your database, your Git repositories, and few other assets, +but there are some [limitations](../index.md#limitations). WARNING: Disaster recovery for multi-secondary configurations is in **Alpha**. For the latest updates, check the [Disaster Recovery epic for complete maturity](https://gitlab.com/groups/gitlab-org/-/epics/3574). Multi-secondary configurations require the complete re-synchronization and re-configuration of all non-promoted secondaries and -will cause downtime. +causes downtime. ## Promoting a **secondary** Geo node in single-secondary configurations @@ -91,13 +88,16 @@ Note the following when promoting a secondary: before proceeding. If the secondary node [has been paused](../../geo/index.md#pausing-and-resuming-replication), the promotion performs a point-in-time recovery to the last known state. - Data that was created on the primary while the secondary was paused will be lost. + Data that was created on the primary while the secondary was paused is lost. - A new **secondary** should not be added at this time. If you want to add a new **secondary**, do this after you have completed the entire process of promoting the **secondary** to the **primary**. - If you encounter an `ActiveRecord::RecordInvalid: Validation failed: Name has already been taken` error message during this process, for more information, see this [troubleshooting advice](../replication/troubleshooting.md#fixing-errors-during-a-failover-or-when-promoting-a-secondary-to-a-primary-node). +- If you run into errors when using `--force` or `--skip-preflight-checks` before 13.5 during this process, + for more information, see this + [troubleshooting advice](../replication/troubleshooting.md#errors-when-using---skip-preflight-checks-or---force). #### Promoting a **secondary** node running on a single machine @@ -243,6 +243,7 @@ required: sets the database to read-write. The instructions vary depending on where your database is hosted: - [Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Promote) - [Azure PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/howto-read-replicas-portal#stop-replication) + - [Google Cloud SQL](https://cloud.google.com/sql/docs/mysql/replication/manage-replicas#promote-replica) - For other external PostgreSQL databases, save the following script in your secondary node, for example `/tmp/geo_promote.sh`, and modify the connection parameters to match your environment. Then, execute it to promote the replica: @@ -493,7 +494,7 @@ must disable the **primary** site: WARNING: If the secondary site [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs a point-in-time recovery to the last known state. -Data that was created on the primary while the secondary was paused will be lost. +Data that was created on the primary while the secondary was paused is lost. 1. SSH in to the database node in the **secondary** and trigger PostgreSQL to promote to read-write: @@ -509,7 +510,7 @@ Data that was created on the primary while the secondary was paused will be lost `geo_secondary_role`: NOTE: - Depending on your architecture these steps will need to be run on any GitLab node that is external to the **secondary** Kubernetes cluster. + Depending on your architecture, these steps need to run on any GitLab node that is external to the **secondary** Kubernetes cluster. ```ruby ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. @@ -537,13 +538,13 @@ Data that was created on the primary while the secondary was paused will be lost 1. Update the existing cluster configuration. - You can retrieve the existing config with Helm: + You can retrieve the existing configuration with Helm: ```shell helm --namespace gitlab get values gitlab-geo > gitlab.yaml ``` - The existing config will contain a section for Geo that should resemble: + The existing configuration contains a section for Geo that should resemble: ```yaml geo: @@ -560,9 +561,9 @@ Data that was created on the primary while the secondary was paused will be lost To promote the **secondary** cluster to a **primary** cluster, update `role: secondary` to `role: primary`. - You can remove the entire `psql` section if the cluster will remain as a primary site, this refers to the tracking database and will be ignored whilst the cluster is acting as a primary site. + If the cluster remains as a primary site, you can remove the entire `psql` section; it refers to the tracking database and is ignored whilst the cluster is acting as a primary site. - Update the cluster with the new config: + Update the cluster with the new configuration: ```shell helm upgrade --install --version <current Chart version> gitlab-geo gitlab/gitlab --namespace gitlab -f gitlab.yaml diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index bd8467f5437..d50078da172 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -35,7 +35,7 @@ required scheduled maintenance period significantly. A common strategy for keeping this period as short as possible for data stored in files is to use `rsync` to transfer the data. An initial `rsync` can be performed ahead of the maintenance window; subsequent `rsync`s (including a -final transfer inside the maintenance window) will then transfer only the +final transfer inside the maintenance window) then transfers only the *changes* between the **primary** node and the **secondary** nodes. Repository-centric strategies for using `rsync` effectively can be found in the @@ -50,7 +50,7 @@ this command reports `ERROR - Replication is not up-to-date` even if replication is actually up-to-date. This bug was fixed in GitLab 13.8 and later. -Run this command to list out all preflight checks and automatically check if replication and verification are complete before scheduling a planned failover to ensure the process will go smoothly: +Run this command to list out all preflight checks and automatically check if replication and verification are complete before scheduling a planned failover to ensure the process goes smoothly: ```shell gitlab-ctl promotion-preflight-checks @@ -73,7 +73,7 @@ In GitLab 12.4, you can optionally allow GitLab to manage replication of Object Database settings are automatically replicated to the **secondary** node, but the `/etc/gitlab/gitlab.rb` file must be set up manually, and differs between nodes. If features such as Mattermost, OAuth or LDAP integration are enabled -on the **primary** node but not the **secondary** node, they will be lost during failover. +on the **primary** node but not the **secondary** node, they are lost during failover. Review the `/etc/gitlab/gitlab.rb` file for both nodes and ensure the **secondary** node supports everything the **primary** node does **before** scheduling a planned failover. @@ -119,7 +119,7 @@ time to complete If any objects are failing to replicate, this should be investigated before scheduling the maintenance window. Following a planned failover, anything that -failed to replicate will be **lost**. +failed to replicate is **lost**. You can use the [Geo status API](../../../api/geo_nodes.md#retrieve-project-sync-or-verification-failures-that-occurred-on-the-current-node) to review failed objects and the reasons for failure. @@ -136,9 +136,9 @@ This [content was moved to another location](background_verification.md). On the **primary** node, navigate to **Admin Area > Messages**, add a broadcast message. You can check under **Admin Area > Geo** to estimate how long it -will take to finish syncing. An example message would be: +takes to finish syncing. An example message would be: -> A scheduled maintenance will take place at XX:XX UTC. We expect it to take +> A scheduled maintenance takes place at XX:XX UTC. We expect it to take > less than 1 hour. ## Prevent updates to the **primary** node @@ -151,7 +151,7 @@ be disabled on the primary site: 1. Disable non-Geo periodic background jobs on the **primary** node by navigating to **Admin Area > Monitoring > Background Jobs > Cron**, pressing `Disable All`, and then pressing `Enable` for the `geo_sidekiq_cron_config_worker` cron job. - This job will re-enable several other cron jobs that are essential for planned + This job re-enables several other cron jobs that are essential for planned failover to complete successfully. ## Finish replicating and verifying all data @@ -161,7 +161,7 @@ be disabled on the primary site: 1. On the **primary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues** and wait for all queues except those with `geo` in the name to drop to 0. These queues contain work that has been submitted by your users; failing over - before it is completed will cause the work to be lost. + before it is completed, causes the work to be lost. 1. On the **primary** node, navigate to **Admin Area > Geo** and wait for the following conditions to be true of the **secondary** node you are failing over to: @@ -176,15 +176,15 @@ be disabled on the primary site: to verify the integrity of CI artifacts, LFS objects, and uploads in file storage. -At this point, your **secondary** node will contain an up-to-date copy of everything the -**primary** node has, meaning nothing will be lost when you fail over. +At this point, your **secondary** node contains an up-to-date copy of everything the +**primary** node has, meaning nothing was lost when you fail over. ## Promote the **secondary** node Finally, follow the [Disaster Recovery docs](index.md) to promote the -**secondary** node to a **primary** node. This process will cause a brief outage on the **secondary** node, and users may need to log in again. +**secondary** node to a **primary** node. This process causes a brief outage on the **secondary** node, and users may need to log in again. -Once it is completed, the maintenance window is over! Your new **primary** node will now +Once it is completed, the maintenance window is over! Your new **primary** node, now begin to diverge from the old one. If problems do arise at this point, failing back to the old **primary** node [is possible](bring_primary_back.md), but likely to result in the loss of any data uploaded to the new **primary** in the meantime. diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index 780e391973c..295a448c432 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -27,7 +27,7 @@ to clone and fetch large repositories, speeding up development. For a video introduction to Geo, see [Introduction to GitLab Geo - GitLab Features](https://www.youtube.com/watch?v=-HDLxSjEh6w). -To make sure you're using the right version of the documentation, navigate to [the Geo page on GitLab.com](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/geo/index.md) and choose the appropriate release from the **Switch branch/tag** dropdown. For example, [`v13.7.6-ee`](https://gitlab.com/gitlab-org/gitlab/-/blob/v13.7.6-ee/doc/administration/geo/index.md). +To make sure you're using the right version of the documentation, navigate to [the Geo page on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/administration/geo/index.md) and choose the appropriate release from the **Switch branch/tag** dropdown. For example, [`v13.7.6-ee`](https://gitlab.com/gitlab-org/gitlab/-/blob/v13.7.6-ee/doc/administration/geo/index.md). Geo uses a set of defined terms that is described in the [Geo Glossary](glossary.md), please familiarize yourself with those terms. @@ -56,11 +56,12 @@ Geo provides: ### Gitaly Cluster Geo should not be confused with [Gitaly Cluster](../gitaly/praefect.md). For more information about -the difference between Geo and Gitaly Cluster, see [Gitaly Cluster compared to Geo](../gitaly/index.md#gitaly-cluster-compared-to-geo). +the difference between Geo and Gitaly Cluster, see +[How does Gitaly Cluster compare to Geo?](../gitaly/faq.md#how-does-gitaly-cluster-compare-to-geo). ## How it works -Your Geo instance can be used for cloning and fetching projects, in addition to reading any data. This will make working with large repositories over large distances much faster. +Your Geo instance can be used for cloning and fetching projects, in addition to reading any data. This makes working with large repositories over large distances much faster. ![Geo overview](replication/img/geo_overview.png) @@ -121,7 +122,7 @@ The following are required to run Geo: The following operating systems are known to ship with a current version of OpenSSH: - [CentOS](https://www.centos.org) 7.4+ - [Ubuntu](https://ubuntu.com) 16.04+ -- PostgreSQL 11+ with [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication) +- PostgreSQL 12+ with [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication) - Git 2.9+ - Git-lfs 2.4.2+ on the user side when using LFS - All sites must run the same GitLab version. @@ -150,17 +151,17 @@ NOTE: When using HTTP or HTTPS proxying, your load balancer must be configured to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the [web terminal](../integration/terminal.md) integration guide for more details. NOTE: -When using HTTPS protocol for port 443, you will need to add an SSL certificate to the load balancers. +When using HTTPS protocol for port 443, you need to add an SSL certificate to the load balancers. If you wish to terminate SSL at the GitLab application server instead, use TCP protocol. ### LDAP -We recommend that if you use LDAP on your **primary** site, you also set up secondary LDAP servers on each **secondary** site. Otherwise, users will not be able to perform Git operations over HTTP(s) on the **secondary** site using HTTP Basic Authentication. However, Git via SSH and personal access tokens will still work. +We recommend that if you use LDAP on your **primary** site, you also set up secondary LDAP servers on each **secondary** site. Otherwise, users are unable to perform Git operations over HTTP(s) on the **secondary** site using HTTP Basic Authentication. However, Git via SSH and personal access tokens still works. NOTE: -It is possible for all **secondary** sites to share an LDAP server, but additional latency can be an issue. Also, consider what LDAP server will be available in a [disaster recovery](disaster_recovery/index.md) scenario if a **secondary** site is promoted to be a **primary** site. +It is possible for all **secondary** sites to share an LDAP server, but additional latency can be an issue. Also, consider what LDAP server is available in a [disaster recovery](disaster_recovery/index.md) scenario if a **secondary** site is promoted to be a **primary** site. -Check for instructions on how to set up replication in your LDAP service. Instructions will be different depending on the software or service used. For example, OpenLDAP provides [these instructions](https://www.openldap.org/doc/admin24/replication.html). +Check for instructions on how to set up replication in your LDAP service. Instructions are different depending on the software or service used. For example, OpenLDAP provides [these instructions](https://www.openldap.org/doc/admin24/replication.html). ### Geo Tracking Database @@ -179,9 +180,9 @@ This daemon: - Reads a log of events replicated by the **primary** site to the **secondary** database instance. - Updates the Geo Tracking Database instance with changes that need to be executed. -When something is marked to be updated in the tracking database instance, asynchronous jobs running on the **secondary** site will execute the required operations and update the state. +When something is marked to be updated in the tracking database instance, asynchronous jobs running on the **secondary** site execute the required operations and update the state. -This new architecture allows GitLab to be resilient to connectivity issues between the sites. It doesn't matter how long the **secondary** site is disconnected from the **primary** site as it will be able to replay all the events in the correct order and become synchronized with the **primary** site again. +This new architecture allows GitLab to be resilient to connectivity issues between the sites. It doesn't matter how long the **secondary** site is disconnected from the **primary** site as it is able to replay all the events in the correct order and become synchronized with the **primary** site again. ## Limitations @@ -196,7 +197,7 @@ This list of limitations only reflects the latest version of GitLab. If you are - Object pools for forked project deduplication work only on the **primary** site, and are duplicated on the **secondary** site. - GitLab Runners cannot register with a **secondary** site. Support for this is [planned for the future](https://gitlab.com/gitlab-org/gitlab/-/issues/3294). - Configuring Geo **secondary** sites to [use high-availability configurations of PostgreSQL](https://gitlab.com/groups/gitlab-org/-/epics/2536) is currently in **alpha** support. -- [Selective synchronization](replication/configuration.md#selective-synchronization) only limits what repositories are replicated. The entire PostgreSQL data is still replicated. Selective synchronization is not built to accomodate compliance / export control use cases. +- [Selective synchronization](replication/configuration.md#selective-synchronization) only limits what repositories are replicated. The entire PostgreSQL data is still replicated. Selective synchronization is not built to accommodate compliance / export control use cases. ### Limitations on replication/verification @@ -280,7 +281,7 @@ For an example of how to set up a location-aware Git remote URL with AWS Route53 ### Backfill -Once a **secondary** site is set up, it will start replicating missing data from +Once a **secondary** site is set up, it starts replicating missing data from the **primary** site in a process known as **backfill**. You can monitor the synchronization process on each Geo site from the **primary** site's **Geo Nodes** dashboard in your browser. diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index e2f12cbd8dc..a1461a64518 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -47,8 +47,8 @@ verification methods: | Blobs | Container registry _(file system)_ | Geo with API/Docker API | _Not implemented_ | | Blobs | Container registry _(object storage)_ | Geo with API/Managed/Docker API (*2*) | _Not implemented_ | | Blobs | Package registry _(file system)_ | Geo with API | SHA256 checksum | -| Blobs | Package registry _(object storage)_ | Geo with API/Managed (*2*) | SHA256 checksum | -| Blobs | Versioned Terraform State _(file system)_ | Geo with API | _Not implemented_ | +| Blobs | Package registry _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | Versioned Terraform State _(file system)_ | Geo with API | SHA256 checksum | | Blobs | Versioned Terraform State _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | External Merge Request Diffs _(file system)_ | Geo with API | _Not implemented_ | | Blobs | External Merge Request Diffs _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | @@ -180,7 +180,7 @@ successfully, you must replicate their data using some other means. |[Project wiki repository](../../../user/project/wiki/) | **Yes** (10.2) | **Yes** (10.7) | No | | |[Group wiki repository](../../../user/project/wiki/index.md#group-wikis) | [**Yes** (13.10)](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | No | No | Behind feature flag `geo_group_wiki_repository_replication`, enabled by default. | |[Uploads](../../uploads.md) | **Yes** (10.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | No | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. | -|[LFS objects](../../lfs/index.md) | **Yes** (10.2) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8922) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). | +|[LFS objects](../../lfs/index.md) | **Yes** (10.2) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8922) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696).<br /><br />Behind feature flag `geo_lfs_object_replication`, enabled by default. | |[Personal snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | No | | |[Project snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | No | | |[CI job artifacts (other than Job Logs)](../../../ci/pipelines/job_artifacts.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. | @@ -189,49 +189,25 @@ successfully, you must replicate their data using some other means. |[Object pools for forked project deduplication](../../../development/git_object_deduplication.md) | **Yes** | No | No | | |[Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | No | Disabled by default. See [instructions](docker_registry.md) to enable. | |[Content in object storage (beta)](object_storage.md) | **Yes** (12.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) | No | | -|[Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | Via Object Storage provider if supported. Native Geo support (Beta). | | -|[Package Registry for npm](../../../user/packages/npm_registry/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | -|[Package Registry for Maven](../../../user/packages/maven_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | -|[Package Registry for Conan](../../../user/packages/conan_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | -|[Package Registry for NuGet](../../../user/packages/nuget_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | -|[Package Registry for PyPI](../../../user/packages/pypi_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | -|[Package Registry for Composer](../../../user/packages/composer_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | -|[Package Registry for generic packages](../../../user/packages/generic_packages/index.md) | **Yes** (13.5) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | -|[Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_terraform_state_version_replication`, enabled by default. | -|[External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_merge_request_diff_replication`, enabled by default. | +|[Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | No | Designs also require replication of LFS objects and Uploads. | +|[Package Registry for npm](../../../user/packages/npm_registry/index.md) | **Yes** (13.2) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for Maven](../../../user/packages/maven_repository/index.md) | **Yes** (13.2) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for Conan](../../../user/packages/conan_repository/index.md) | **Yes** (13.2) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for NuGet](../../../user/packages/nuget_repository/index.md) | **Yes** (13.2) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for PyPI](../../../user/packages/pypi_repository/index.md) | **Yes** (13.2) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for Composer](../../../user/packages/composer_repository/index.md) | **Yes** (13.2) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for generic packages](../../../user/packages/generic_packages/index.md) | **Yes** (13.5) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.12) | Via Object Storage provider if supported. Native Geo support (Beta). | Replication is behind the feature flag `geo_terraform_state_version_replication`, enabled by default. Verification was behind the feature flag `geo_terraform_state_version_verification`, which was removed in 14.0| +|[External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Replication is behind the feature flag `geo_merge_request_diff_replication`, enabled by default. Verification is under development, behind the feature flag `geo_merge_request_diff_verification`, introduced in 14.0.| |[Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2810) | No | | |[Server-side Git hooks](../../server_hooks.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1867) | No | No | | |[Elasticsearch integration](../../../integration/elasticsearch.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | | -|[GitLab Pages](../../pages/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | No | | +|[GitLab Pages](../../pages/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | Via Object Storage provider if supported. **No** native Geo support (Beta). | | |[Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | Blocked on [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Note that replication of this cache is not needed for Disaster Recovery purposes because it can be recreated from external sources. | -|[Vulnerability Export](../../../user/application_security/vulnerability_report/#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Not planned because they are ephemeral and sensitive. They can be regenerated on demand. | +|[Vulnerability Export](../../../user/application_security/vulnerability_report/#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | | Not planned because they are ephemeral and sensitive. They can be regenerated on demand. | -#### LFS object replication using the self service framework +#### Limitation of verification for files in Object Storage -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276696) in GitLab 13.12. -> - [Deployed behind a feature flag](../../../user/feature_flags.md), enabled by default. -> - Not enabled on GitLab.com as Geo is not enabled. -> - Recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-lfs-object-replication-using-the-self-service-framework). +GitLab managed Object Storage replication support [is in beta](object_storage.md#enabling-gitlab-managed-object-storage-replication). -There can be [risks when disabling released features](../../../user/feature_flags.md#risks-when-disabling-released-features). -Refer to this feature's version history for more details. - -##### Enable or disable LFS object replication using the self service framework - -LFS object replication using the self service framework is under development but ready for production use. It is -deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) -can opt to disable it. - -To enable it: - -```ruby -Feature.enable(:geo_lfs_object_replication) -``` - -To disable it: - -```ruby -Feature.disable(:geo_lfs_object_replication) -``` +Locally stored files are verified but remote stored files are not. diff --git a/doc/administration/geo/replication/faq.md b/doc/administration/geo/replication/faq.md index a83a1c22db6..ef41b2ff172 100644 --- a/doc/administration/geo/replication/faq.md +++ b/doc/administration/geo/replication/faq.md @@ -17,13 +17,13 @@ On each **secondary** site, there is a read-only replicated copy of the GitLab d A **secondary** site also has a tracking database where it stores which projects have been synced. Geo compares the two databases to find projects that are not yet tracked. -At the start, this tracking database is empty, so Geo will start trying to update from every project that it can see in the GitLab database. +At the start, this tracking database is empty, so Geo tries to update from every project that it can see in the GitLab database. For each project to sync: -1. Geo will issue a `git fetch geo --mirror` to get the latest information from the **primary** site. - If there are no changes, the sync will be fast and end quickly. Otherwise, it will pull the latest commits. -1. The **secondary** site will update the tracking database to store the fact that it has synced projects A, B, C, etc. +1. Geo issues a `git fetch geo --mirror` to get the latest information from the **primary** site. + If there are no changes, the sync is fast. Otherwise, it has to pull the latest commits. +1. The **secondary** site updates the tracking database to store the fact that it has synced projects A, B, C, etc. 1. Repeat until all projects are synced. When someone pushes a commit to the **primary** site, it generates an event in the GitLab database that the repository has changed. @@ -70,4 +70,4 @@ Yes. See [Docker Registry for a **secondary** site](docker_registry.md). ## Can I login to a secondary site? -Yes, but secondary sites receive all authentication data (like user accounts and logins) from the primary instance. This means you will be re-directed to the primary for authentication and routed back afterwards. +Yes, but secondary sites receive all authentication data (like user accounts and logins) from the primary instance. This means you are re-directed to the primary for authentication and then routed back. diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md index 8f67e70c9e2..c6b1078ddf0 100644 --- a/doc/administration/geo/replication/geo_validation_tests.md +++ b/doc/administration/geo/replication/geo_validation_tests.md @@ -43,7 +43,7 @@ The following are GitLab upgrade validation tests we performed. - Outcome: Partial success because we observed downtime during the upgrade of the primary and secondary sites. - Follow up issues/actions: - [Fix zero-downtime upgrade process/instructions for multi-node Geo deployments](https://gitlab.com/gitlab-org/gitlab/-/issues/225684) - - [Geo:check Rake task: Exclude AuthorizedKeysCommand check if node not running Puma/Unicorn](https://gitlab.com/gitlab-org/gitlab/-/issues/225454) + - [Geo:check Rake task: Exclude AuthorizedKeysCommand check if node not running Puma](https://gitlab.com/gitlab-org/gitlab/-/issues/225454) - [Update instructions in the next upgrade issue to include monitoring HAProxy dashboards](https://gitlab.com/gitlab-org/gitlab/-/issues/225359) [Upgrade Geo multi-node installation](https://gitlab.com/gitlab-org/gitlab/-/issues/208104): @@ -53,7 +53,7 @@ The following are GitLab upgrade validation tests we performed. - Outcome: Partial success because we did not run the looping pipeline during the demo to validate zero-downtime. - Follow up issues: - - [Clarify how Puma/Unicorn should include deploy node](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5460) + - [Clarify how Puma should include deploy node](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5460) - [Investigate MR creation failure after upgrade to 12.9.10](https://gitlab.com/gitlab-org/gitlab/-/issues/223282) Closed as false positive. ### February 2020 diff --git a/doc/administration/geo/replication/multiple_servers.md b/doc/administration/geo/replication/multiple_servers.md index 59bb3884a02..ea2488b65fb 100644 --- a/doc/administration/geo/replication/multiple_servers.md +++ b/doc/administration/geo/replication/multiple_servers.md @@ -17,9 +17,9 @@ described, it is possible to adapt these instructions to your needs. _[diagram source - GitLab employees only](https://docs.google.com/drawings/d/1z0VlizKiLNXVVVaERFwgsIOuEgjcUqDTWPdQYsE7Z4c/edit)_ -The topology above assumes that the **primary** and **secondary** Geo clusters +The topology above assumes the **primary** and **secondary** Geo clusters are located in two separate locations, on their own virtual network -with private IP addresses. The network is configured such that all machines within +with private IP addresses. The network is configured such that all machines in one geographic location can communicate with each other using their private IP addresses. The IP addresses given are examples and may be different depending on the network topology of your deployment. @@ -44,9 +44,10 @@ Support for PostgreSQL on **secondary** nodes in multi-node configuration Because of the additional complexity involved in setting up this configuration for PostgreSQL and Redis, it is not covered by this Geo multi-node documentation. -For more information about setting up a multi-node PostgreSQL cluster and Redis cluster using the omnibus package see the multi-node documentation for -[PostgreSQL](../../postgresql/replication_and_failover.md) and -[Redis](../../redis/replication_and_failover.md), respectively. +For more information on setting up a multi-node PostgreSQL cluster and Redis cluster using the Omnibus GitLab package, see: + +- [PostgreSQL multi-node documentation](../../postgresql/replication_and_failover.md) +- [Redis multi-node documentation](../../redis/replication_and_failover.md) NOTE: It is possible to use cloud hosted services for PostgreSQL and Redis, but this is beyond the scope of this document. @@ -60,8 +61,8 @@ you already have a working GitLab instance that is in-use, it can be used as a The second cluster serves as the **secondary** node. Again, use the [GitLab multi-node documentation](../../reference_architectures/index.md) to set this up. -It's a good idea to log in and test it, however, note that its data is -wiped out as part of the process of replicating from the **primary**. +It's a good idea to log in and test it. However, be aware that its data is +wiped out as part of the process of replicating from the **primary** node. ## Configure the GitLab cluster to be the **primary** node @@ -92,9 +93,9 @@ After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus NOTE: PostgreSQL and Redis should have already been disabled on the -application servers, and connections from the application servers to those -services on the backend servers configured, during normal GitLab multi-node set up. See -multi-node configuration documentation for +application servers during normal GitLab multi-node setup. Connections +from the application servers to services on the backend servers should +have also been configured. See multi-node configuration documentation for [PostgreSQL](../../postgresql/replication_and_failover.md#configuring-the-application-nodes) and [Redis](../../redis/replication_and_failover.md#example-configuration-for-the-gitlab-application). @@ -120,12 +121,12 @@ major differences: called the "tracking database", which tracks the synchronization state of various resources. -Therefore, we set up the multi-node components one-by-one, and include deviations -from the normal multi-node setup. However, we highly recommend first configuring a -brand-new cluster as if it were not part of a Geo setup so that it can be -tested and verified as a working cluster. And only then should it be modified -for use as a Geo **secondary**. This helps to separate problems that are related -and are not related to Geo setup. +Therefore, we set up the multi-node components one by one and include deviations +from the normal multi-node setup. However, we highly recommend configuring a +brand-new cluster first, as if it were not part of a Geo setup. This allows +verifying that it is a working cluster. And only then should it be modified +for use as a Geo **secondary**. This helps to separate Geo setup problems from +unrelated problems. ### Step 1: Configure the Redis and Gitaly services on the **secondary** node @@ -218,11 +219,10 @@ the **primary** database. Use the following as a guide. prometheus['enable'] = false redis['enable'] = false redis_exporter['enable'] = false - repmgr['enable'] = false + patroni['enable'] = false sidekiq['enable'] = false sidekiq_cluster['enable'] = false puma['enable'] = false - unicorn['enable'] = false ``` After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. @@ -290,11 +290,10 @@ Configure the tracking database. prometheus['enable'] = false redis['enable'] = false redis_exporter['enable'] = false - repmgr['enable'] = false + patroni['enable'] = false sidekiq['enable'] = false sidekiq_cluster['enable'] = false puma['enable'] = false - unicorn['enable'] = false ``` After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. @@ -366,10 +365,10 @@ then make the following modifications: ``` NOTE: -If you had set up PostgreSQL cluster using the omnibus package and you had set -up `postgresql['sql_user_password'] = 'md5 digest of secret'` setting, keep in +If you had set up PostgreSQL cluster using the omnibus package and had set +`postgresql['sql_user_password'] = 'md5 digest of secret'`, keep in mind that `gitlab_rails['db_password']` and `geo_secondary['db_password']` -mentioned above contains the plaintext passwords. This is used to let the Rails +contains the plaintext passwords. This is used to let the Rails servers connect to the databases. NOTE: @@ -438,9 +437,8 @@ application servers above, with some changes to run only the `sidekiq` service: prometheus['enable'] = false redis['enable'] = false redis_exporter['enable'] = false - repmgr['enable'] = false + patroni['enable'] = false puma['enable'] = false - unicorn['enable'] = false ## ## The unique identifier for the Geo node. diff --git a/doc/administration/geo/replication/remove_geo_node.md b/doc/administration/geo/replication/remove_geo_node.md index 697d8c6ae38..b72cd3cbb95 100644 --- a/doc/administration/geo/replication/remove_geo_node.md +++ b/doc/administration/geo/replication/remove_geo_node.md @@ -1,5 +1,6 @@ --- redirect_to: '../../geo/replication/remove_geo_site.md' +remove_date: '2021-06-01' --- This document was moved to [another location](../../geo/replication/remove_geo_site.md). diff --git a/doc/administration/geo/replication/security_review.md b/doc/administration/geo/replication/security_review.md index f84d7a2171d..ae41599311b 100644 --- a/doc/administration/geo/replication/security_review.md +++ b/doc/administration/geo/replication/security_review.md @@ -184,7 +184,7 @@ from [owasp.org](https://owasp.org/). ### What databases and application servers support the application? -- PostgreSQL >= 11, Redis, Sidekiq, Puma. +- PostgreSQL >= 12, Redis, Sidekiq, Puma. ### How will database connection strings, encryption keys, and other sensitive components be stored, accessed, and protected from unauthorized detection? diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 6d990fd12ba..1fd923dbaf1 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -583,64 +583,6 @@ to start again from scratch, there are a few steps that can help you: gitlab-ctl start ``` -## Fixing errors during a PostgreSQL upgrade or downgrade - -### Message: `ERROR: psql: FATAL: role "gitlab-consul" does not exist` - -When -[upgrading PostgreSQL on a Geo instance](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance), you might encounter the -following error: - -```plaintext -$ sudo gitlab-ctl pg-upgrade --target-version=11 -Checking for an omnibus managed postgresql: OK -Checking if postgresql['version'] is set: OK -Checking if we already upgraded: NOT OK -Checking for a newer version of PostgreSQL to install -Upgrading PostgreSQL to 11.7 -Checking if PostgreSQL bin files are symlinked to the expected location: OK -Waiting 30 seconds to ensure tasks complete before PostgreSQL upgrade. -See https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server for details -If you do not want to upgrade the PostgreSQL server at this time, enter Ctrl-C and see the documentation for details - -Please hit Ctrl-C now if you want to cancel the operation. -..............................Detected an HA cluster. -Error running command: /opt/gitlab/embedded/bin/psql -qt -d gitlab_repmgr -h /var/opt/gitlab/postgresql -p 5432 -c "SELECT name FROM repmgr_gitlab_cluster.repl_nodes WHERE type='master' AND active != 'f'" -U gitlab-consul -ERROR: psql: FATAL: role "gitlab-consul" does not exist -Traceback (most recent call last): - 10: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>' - 9: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load' - 8: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>' - 7: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run' - 6: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category' - 5: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:171:in `block in load_file' - 4: from /opt/gitlab/embedded/service/omnibus-ctl-ee/lib/repmgr.rb:248:in `is_master?' - 3: from /opt/gitlab/embedded/service/omnibus-ctl-ee/lib/repmgr.rb:100:in `execute_psql' - 2: from /opt/gitlab/embedded/service/omnibus-ctl-ee/lib/repmgr.rb:113:in `cmd' - 1: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/mixlib-shellout-3.0.9/lib/mixlib/shellout.rb:287:in `error!' -/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/mixlib-shellout-3.0.9/lib/mixlib/shellout.rb:300:in `invalid!': Expected process to exit with [0], but received '2' (Mixlib::ShellOut::ShellCommandFailed) ----- Begin output of /opt/gitlab/embedded/bin/psql -qt -d gitlab_repmgr -h /var/opt/gitlab/postgresql -p 5432 -c "SELECT name FROM repmgr_gitlab_cluster.repl_nodes WHERE type='master' AND active != 'f'" -U gitlab-consul ---- -STDOUT: -STDERR: psql: FATAL: role "gitlab-consul" does not exist ----- End output of /opt/gitlab/embedded/bin/psql -qt -d gitlab_repmgr -h /var/opt/gitlab/postgresql -p 5432 -c "SELECT name FROM repmgr_gitlab_cluster.repl_nodes WHERE type='master' AND active != 'f'" -U gitlab-consul ---- -Ran /opt/gitlab/embedded/bin/psql -qt -d gitlab_repmgr -h /var/opt/gitlab/postgresql -p 5432 -c "SELECT name FROM repmgr_gitlab_cluster.repl_nodes WHERE type='master' AND active != 'f'" -U gitlab-consul returned 2 -``` - -If you are upgrading the PostgreSQL read-replica of a Geo secondary node, and -you are not using `consul` or `repmgr`, you may need to disable `consul` and/or -`repmgr` services in `gitlab.rb`: - -```ruby -consul['enable'] = false -repmgr['enable'] = false -``` - -Then reconfigure GitLab: - -```shell -sudo gitlab-ctl reconfigure -``` - ## Fixing errors during a failover or when promoting a secondary to a primary node The following are possible errors that might be encountered during failover or @@ -756,6 +698,30 @@ this command reports `ERROR - Replication is not up-to-date` even if replication is actually up-to-date. If replication and verification output shows that it is complete, you can add `--skip-preflight-checks` to make the command complete promotion. This bug was fixed in GitLab 13.8 and later. +### Errors when using `--skip-preflight-checks` or `--force` + +Before GitLab 13.5, you could bump into one of the following errors when using +`--skip-preflight-checks` or `--force`: + +```plaintext +get_ctl_options': invalid option: --skip-preflight-checks (OptionParser::InvalidOption) + +get_ctl_options': invalid option: --force (OptionParser::InvalidOption) +``` + +This can happen with XFS or filesystems that list files in lexical order, because the +load order of the Omnibus command files can be different than expected, and a global function would get redefined. +More details can be found in [the related issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6076). + +The workaround is to manually run the preflight checks and promote the database, by running +the following commands on the Geo secondary site: + +```shell +sudo gitlab-ctl promotion-preflight-checks +sudo /opt/gitlab/embedded/bin/gitlab-pg-ctl promote +sudo gitlab-ctl reconfigure +sudo gitlab-rake geo:set_secondary_as_primary + ## Expired artifacts If you notice for some reason there are more artifacts on the Geo @@ -854,6 +820,11 @@ To resolve this issue: the **primary** node using IPv4 in the `/etc/hosts` file. Alternatively, you should [enable IPv6 on the **primary** node](https://docs.gitlab.com/omnibus/settings/nginx.html#setting-the-nginx-listen-address-or-addresses). +### Geo Admin Area shows 'Unknown' for health status and 'Request failed with status code 401' + +If using a load balancer, ensure that the load balancer's URL is set as the `external_url` in the +`/etc/gitlab/gitlab.rb` of the nodes behind the load balancer. + ### GitLab Pages return 404 errors after promoting This is due to [Pages data not being managed by Geo](datatypes.md#limitations-on-replicationverification). diff --git a/doc/administration/geo/replication/using_a_geo_server.md b/doc/administration/geo/replication/using_a_geo_server.md index f8ce72ac3f8..04c30514a89 100644 --- a/doc/administration/geo/replication/using_a_geo_server.md +++ b/doc/administration/geo/replication/using_a_geo_server.md @@ -1,5 +1,6 @@ --- redirect_to: '../../geo/replication/usage.md' +remove_date: '2022-06-01' --- This document was moved to [another location](../../geo/replication/usage.md). diff --git a/doc/administration/geo/replication/version_specific_updates.md b/doc/administration/geo/replication/version_specific_updates.md index 4a101c52325..301be931b29 100644 --- a/doc/administration/geo/replication/version_specific_updates.md +++ b/doc/administration/geo/replication/version_specific_updates.md @@ -11,6 +11,10 @@ Review this page for update instructions for your version. These steps accompany the [general steps](updating_the_geo_nodes.md#general-update-steps) for updating Geo nodes. +## Updating to GitLab 13.11 + +We found an [issue with Git clone/pull through HTTP(s)](https://gitlab.com/gitlab-org/gitlab/-/issues/330787) on Geo secondaries and on any GitLab instance if maintenance mode is enabled. This was caused by a regression in GitLab Workhorse. This is fixed in the [GitLab 13.11.4 patch release](https://about.gitlab.com/releases/2021/05/14/gitlab-13-11-4-released/). To avoid this issue, upgrade to GitLab 13.11.4 or later. + ## Updating to GitLab 13.9 We've detected an issue [with a column rename](https://gitlab.com/gitlab-org/gitlab/-/issues/324160) @@ -78,6 +82,12 @@ paused fails. Do not pause replication before promoting a secondary. If the node is paused, be sure to resume before promoting. To avoid this issue, upgrade to GitLab 13.4 or later. +WARNING: +Promoting the database during a failover can fail on XFS and filesystems ordering files lexically, +when using `--force` or `--skip-preflight-checks`, due to [an issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6076) fixed in 13.5. +The [troubleshooting steps](troubleshooting.md#errors-when-using---skip-preflight-checks-or---force) +contain a workaround if you run into errors during the failover. + ## Updating to GitLab 13.2 In GitLab 13.2, promoting a secondary node to a primary while the secondary is diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index b87a606e349..f6e72092a5f 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -9,7 +9,7 @@ type: howto NOTE: If your GitLab installation uses external (not managed by Omnibus) PostgreSQL -instances, the Omnibus roles will not be able to perform all necessary +instances, the Omnibus roles are unable to perform all necessary configuration steps. In this case, [follow the Geo with external PostgreSQL instances document instead](external_database.md). @@ -25,10 +25,23 @@ size. You are encouraged to first read through all the steps before executing them in your testing/production environment. -## PostgreSQL replication +## Single instance database replication -The GitLab **primary** node where the write operations happen will connect to -the **primary** database server, and **secondary** nodes will +A single instance database replication is easier to set up and still provides the same Geo capabilities +as a clusterized alternative. It's useful for setups running on a single machine +or trying to evaluate Geo for a future clusterized installation. + +A single instance can be expanded to a clusterized version using Patroni, which is recommended for a +highly available architecture. + +Follow below the instructions on how to set up PostgreSQL replication as a single instance database. +Alternatively, you can look at the [Multi-node database replication](#multi-node-database-replication) +instructions on setting up replication with a Patroni cluster. + +### PostgreSQL replication + +The GitLab **primary** node where the write operations happen connects to +the **primary** database server, and **secondary** nodes connect to their own database servers (which are also read-only). We recommend using [PostgreSQL replication slots](https://medium.com/@tk512/replication-slots-in-postgresql-b4b03d277c75) @@ -37,8 +50,8 @@ recover. See below for more details. The following guide assumes that: -- You are using Omnibus and therefore you are using PostgreSQL 11 or later - which includes the [`pg_basebackup` tool](https://www.postgresql.org/docs/11/app-pgbasebackup.html). +- You are using Omnibus and therefore you are using PostgreSQL 12 or later + which includes the [`pg_basebackup` tool](https://www.postgresql.org/docs/12/app-pgbasebackup.html). - You have a **primary** node already set up (the GitLab server you are replicating from), running Omnibus' PostgreSQL (or equivalent version), and you have a new **secondary** server set up with the same versions of the OS, @@ -48,7 +61,7 @@ WARNING: Geo works with streaming replication. Logical replication is not supported at this time. There is an [issue where support is being discussed](https://gitlab.com/gitlab-org/gitlab/-/issues/7420). -### Step 1. Configure the **primary** server +#### Step 1. Configure the **primary** server 1. SSH into your GitLab **primary** server and login as root: @@ -75,13 +88,9 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o gitlab-ctl set-geo-primary-node ``` - This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`. - -1. GitLab 10.4 and up only: Do the following to make sure the `gitlab` database user has a password defined: + This command uses your defined `external_url` in `/etc/gitlab/gitlab.rb`. - NOTE: - Until FDW settings are removed in GitLab version 14.0, avoid using single or double quotes in the - password for PostgreSQL as that will lead to errors when reconfiguring. +1. Define a password for the `gitlab` database user: Generate a MD5 hash of the desired password: @@ -103,18 +112,28 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o # must be present in all application nodes. gitlab_rails['db_password'] = '<your_password_here>' ``` + +1. Define a password for the database [replication user](https://wiki.postgresql.org/wiki/Streaming_Replication). -1. Omnibus GitLab already has a [replication user](https://wiki.postgresql.org/wiki/Streaming_Replication) - called `gitlab_replicator`. You must set the password for this user manually. - You will be prompted to enter a password: + We will use the username defined in `/etc/gitlab/gitlab.rb` under the `postgresql['sql_replication_user']` + setting. The default value is `gitlab_replicator`, but if you changed it to something else, adapt + the instructions below. + + Generate a MD5 hash of the desired password: ```shell - gitlab-ctl set-replication-password + gitlab-ctl pg-password-md5 gitlab_replicator + # Enter password: <your_password_here> + # Confirm password: <your_password_here> + # 950233c0dfc2f39c64cf30457c3b7f1e ``` - This command will also read the `postgresql['sql_replication_user']` Omnibus - setting in case you have changed `gitlab_replicator` username to something - else. + Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab_replicator` + postgresql['sql_replication_password'] = '<md5_hash_of_your_password>' + ``` If you are using an external database not managed by Omnibus GitLab, you need to create the replicator user and define a password to it manually: @@ -154,7 +173,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o echo "External address: $(curl --silent "ipinfo.io/ip")" ``` - In most cases, the following addresses will be used to configure GitLab + In most cases, the following addresses are used to configure GitLab Geo: | Configuration | Address | @@ -168,11 +187,11 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o `postgresql['md5_auth_cidr_addresses']` and `postgresql['listen_address']`. The `listen_address` option opens PostgreSQL up to network connections with the interface - corresponding to the given address. See [the PostgreSQL documentation](https://www.postgresql.org/docs/11/runtime-config-connection.html) + corresponding to the given address. See [the PostgreSQL documentation](https://www.postgresql.org/docs/12/runtime-config-connection.html) for more details. NOTE: - If you need to use `0.0.0.0` or `*` as the listen_address, you will also need to add + If you need to use `0.0.0.0` or `*` as the listen_address, you also need to add `127.0.0.1/32` to the `postgresql['md5_auth_cidr_addresses']` setting, to allow Rails to connect through `127.0.0.1`. For more information, see [omnibus-5258](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5258). @@ -190,7 +209,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ## Geo Primary role ## - configure dependent flags automatically to enable Geo ## - roles ['geo_primary_role'] + roles(['geo_primary_role']) ## ## Primary address @@ -226,7 +245,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ``` You may also want to edit the `wal_keep_segments` and `max_wal_senders` to match your - database replication requirements. Consult the [PostgreSQL - Replication documentation](https://www.postgresql.org/docs/11/runtime-config-replication.html) + database replication requirements. Consult the [PostgreSQL - Replication documentation](https://www.postgresql.org/docs/12/runtime-config-replication.html) for more information. 1. Save the file and reconfigure GitLab for the database listen changes and @@ -262,7 +281,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o `5432` to the **primary** server's private address. 1. A certificate was automatically generated when GitLab was reconfigured. This - will be used automatically to protect your PostgreSQL traffic from + is used automatically to protect your PostgreSQL traffic from eavesdroppers, but to protect against active ("man-in-the-middle") attackers, the **secondary** node needs a copy of the certificate. Make a copy of the PostgreSQL `server.crt` file on the **primary** node by running this command: @@ -272,10 +291,10 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ``` Copy the output into a clipboard or into a local file. You - will need it when setting up the **secondary** node! The certificate is not sensitive + need it when setting up the **secondary** node! The certificate is not sensitive data. -### Step 2. Configure the **secondary** server +#### Step 2. Configure the **secondary** server 1. SSH into your GitLab **secondary** server and login as root: @@ -325,7 +344,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o -T server.crt ~gitlab-psql/.postgresql/root.crt ``` - PostgreSQL will now only recognize that exact certificate when verifying TLS + PostgreSQL now only recognizes that exact certificate when verifying TLS connections. The certificate can only be replicated by someone with access to the private key, which is **only** present on the **primary** node. @@ -363,7 +382,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ## Geo Secondary role ## - configure dependent flags automatically to enable Geo ## - roles ['geo_secondary_role'] + roles(['geo_secondary_role']) ## ## Secondary address @@ -376,12 +395,13 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ## Database credentials password (defined previously in primary node) ## - replicate same values here as defined in primary node ## + postgresql['sql_replication_password'] = '<md5_hash_of_your_password>' postgresql['sql_user_password'] = '<md5_hash_of_your_password>' gitlab_rails['db_password'] = '<your_password_here>' ``` For external PostgreSQL instances, see [additional instructions](external_database.md). - If you bring a former **primary** node back online to serve as a **secondary** node, then you also need to remove `roles ['geo_primary_role']` or `geo_primary_role['enable'] = true`. + If you bring a former **primary** node back online to serve as a **secondary** node, then you also need to remove `roles(['geo_primary_role'])` or `geo_primary_role['enable'] = true`. 1. Reconfigure GitLab for the changes to take effect: @@ -395,7 +415,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o gitlab-ctl restart postgresql ``` -### Step 3. Initiate the replication process +#### Step 3. Initiate the replication process Below we provide a script that connects the database on the **secondary** node to the database on the **primary** node, replicates the database, and creates the @@ -423,7 +443,7 @@ data before running `pg_basebackup`. WARNING: Each Geo **secondary** node must have its own unique replication slot name. - Using the same slot name between two secondaries will break PostgreSQL replication. + Using the same slot name between two secondaries breaks PostgreSQL replication. ```shell gitlab-ctl replicate-geo-database \ @@ -441,57 +461,57 @@ data before running `pg_basebackup`. to list them all, but here are a couple of tips: - If PostgreSQL is listening on a non-standard port, add `--port=` as well. - - If your database is too large to be transferred in 30 minutes, you will need + - If your database is too large to be transferred in 30 minutes, you need to increase the timeout, e.g., `--backup-timeout=3600` if you expect the initial replication to take under an hour. - Pass `--sslmode=disable` to skip PostgreSQL TLS authentication altogether (e.g., you know the network path is secure, or you are using a site-to-site VPN). This is **not** safe over the public Internet! - You can read more details about each `sslmode` in the - [PostgreSQL documentation](https://www.postgresql.org/docs/11/libpq-ssl.html#LIBPQ-SSL-PROTECTION); + [PostgreSQL documentation](https://www.postgresql.org/docs/12/libpq-ssl.html#LIBPQ-SSL-PROTECTION); the instructions above are carefully written to ensure protection against both passive eavesdroppers and active "man-in-the-middle" attackers. - Change the `--slot-name` to the name of the replication slot - to be used on the **primary** database. The script will attempt to create the + to be used on the **primary** database. The script attempts to create the replication slot automatically if it does not exist. - - If you're repurposing an old server into a Geo **secondary** node, you'll need to + - If you're repurposing an old server into a Geo **secondary** node, you need to add `--force` to the command line. - When not in a production machine you can disable backup step if you really sure this is what you want by adding `--skip-backup` The replication process is now complete. -## PgBouncer support (optional) +### PgBouncer support (optional) [PgBouncer](https://www.pgbouncer.org/) may be used with GitLab Geo to pool -PostgreSQL connections. We recommend using PgBouncer if you use GitLab in a -high-availability configuration with a cluster of nodes supporting a Geo -**primary** site and two other clusters of nodes supporting a Geo **secondary** site. -One for the main database and the other for the tracking database. For more information, +PostgreSQL connections, which can improve performance even when using in a +single instance installation. + +We recommend using PgBouncer if you use GitLab in a highly available +configuration with a cluster of nodes supporting a Geo **primary** site and +two other clusters of nodes supporting a Geo **secondary** site. One for the +main database and the other for the tracking database. For more information, see [High Availability with Omnibus GitLab](../../postgresql/replication_and_failover.md). -## Patroni support +## Multi-node database replication -Support for Patroni is intended to replace `repmgr` as a -[highly available PostgreSQL solution](../../postgresql/replication_and_failover.md) -on the primary node, but it can also be used for PostgreSQL HA on a secondary -site. Similar to `repmgr`, using Patroni on a secondary node is optional. +In GitLab 14.0, Patroni replaced `repmgr` as the supported +[highly available PostgreSQL solution](../../postgresql/replication_and_failover.md). -Starting with GitLab 13.5, Patroni is available for _experimental_ use with Geo -primary and secondary sites. Due to its experimental nature, Patroni support is -subject to change without notice. +NOTE: +If you still haven't [migrated from repmgr to Patroni](#migrating-from-repmgr-to-patroni) you're highly advised to do so. -This experimental implementation has the following limitations: +### Patroni support -- Whenever `gitlab-ctl reconfigure` runs on a Patroni Leader instance, there's a - chance the node will be demoted due to the required short-time restart. To - avoid this, you can pause auto-failover by running `gitlab-ctl patroni pause`. - After a reconfigure, it resumes on its own. +Patroni is the official replication management solution for Geo. It +can be used to build a highly available cluster on the **primary** and a **secondary** Geo site. +Using Patroni on a **secondary** site is optional and you don't have to use the same amount of +nodes on each Geo site. For instructions about how to set up Patroni on the primary site, see the [PostgreSQL replication and failover with Omnibus GitLab](../../postgresql/replication_and_failover.md#patroni) page. -### Configuring Patroni cluster for a Geo secondary site +#### Configuring Patroni cluster for a Geo secondary site In a Geo secondary site, the main PostgreSQL database is a read-only replica of the primary site’s PostgreSQL database. @@ -503,7 +523,7 @@ configuration for the secondary site. The internal load balancer provides a sing endpoint for connecting to the Patroni cluster's leader whenever a new leader is elected. Be sure to use [password credentials](../../postgresql/replication_and_failover.md#database-authorization-for-patroni) and other database best practices. -#### Step 1. Configure Patroni permanent replication slot on the primary site +##### Step 1. Configure Patroni permanent replication slot on the primary site To set up database replication with Patroni on a secondary node, we need to configure a _permanent replication slot_ on the primary node's Patroni cluster, @@ -521,16 +541,16 @@ Leader instance**: 1. Edit `/etc/gitlab/gitlab.rb` and add the following: ```ruby - consul['enable'] = true + roles(['patroni_role']) + + consul['services'] = %w(postgresql) consul['configuration'] = { retry_join: %w[CONSUL_PRIMARY1_IP CONSUL_PRIMARY2_IP CONSUL_PRIMARY3_IP] } - - repmgr['enable'] = false - + # You need one entry for each secondary, with a unique name following PostgreSQL slot_name constraints: # - # Configuration syntax will be: 'unique_slotname' => { 'type' => 'physical' }, + # Configuration syntax is: 'unique_slotname' => { 'type' => 'physical' }, # We don't support setting a permanent replication slot for logical replication type patroni['replication_slots'] = { 'geo_secondary' => { 'type' => 'physical' } @@ -539,15 +559,18 @@ Leader instance**: patroni['use_pg_rewind'] = true patroni['postgresql']['max_wal_senders'] = 8 # Use double of the amount of patroni/reserved slots (3 patronis + 1 reserved slot for a Geo secondary). patroni['postgresql']['max_replication_slots'] = 8 # Use double of the amount of patroni/reserved slots (3 patronis + 1 reserved slot for a Geo secondary). + patroni['replication_password'] = 'PLAIN_TEXT_POSTGRESQL_REPLICATION_PASSWORD' - postgresql['md5_auth_cidr_addresses'] = [ - 'PATRONI_PRIMARY1_IP/32', 'PATRONI_PRIMARY2_IP/32', 'PATRONI_PRIMARY3_IP/32', 'PATRONI_PRIMARY_PGBOUNCER/32', - 'PATRONI_SECONDARY1_IP/32', 'PATRONI_SECONDARY2_IP/32', 'PATRONI_SECONDARY3_IP/32', 'PATRONI_SECONDARY_PGBOUNCER/32' # We list all secondary instances as they can all become a Standby Leader + # We list all secondary instances as they can all become a Standby Leader + postgresql['md5_auth_cidr_addresses'] = %w[ + PATRONI_PRIMARY1_IP/32 PATRONI_PRIMARY2_IP/32 PATRONI_PRIMARY3_IP/32 PATRONI_PRIMARY_PGBOUNCER/32 + PATRONI_SECONDARY1_IP/32 PATRONI_SECONDARY2_IP/32 PATRONI_SECONDARY3_IP/32 PATRONI_SECONDARY_PGBOUNCER/32 ] postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH' postgresql['sql_replication_password'] = 'POSTGRESQL_REPLICATION_PASSWORD_HASH' postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' + postgresql['listen_address'] = '0.0.0.0' # You can use a public or VPC address here instead ``` 1. Reconfigure GitLab for the changes to take effect: @@ -556,17 +579,17 @@ Leader instance**: gitlab-ctl reconfigure ``` -#### Step 2. Configure the internal load balancer on the primary site +##### Step 2. Configure the internal load balancer on the primary site To avoid reconfiguring the Standby Leader on the secondary site whenever a new -Leader is elected on the primary site, we'll need to set up a TCP internal load -balancer which will give a single endpoint for connecting to the Patroni +Leader is elected on the primary site, we need to set up a TCP internal load +balancer which gives a single endpoint for connecting to the Patroni cluster's Leader. The Omnibus GitLab packages do not include a Load Balancer. Here's how you could do it with [HAProxy](https://www.haproxy.org/). -The following IPs and names will be used as an example: +The following IPs and names are used as an example: - `10.6.0.21`: Patroni 1 (`patroni1.internal`) - `10.6.0.21`: Patroni 2 (`patroni2.internal`) @@ -600,7 +623,7 @@ backend postgresql Refer to your preferred Load Balancer's documentation for further guidance. -#### Step 3. Configure a PgBouncer node on the secondary site +##### Step 3. Configure a PgBouncer node on the secondary site A production-ready and highly available configuration requires at least three Consul nodes, a minimum of one PgBouncer node, but it’s recommended to have @@ -621,22 +644,26 @@ Follow the minimal configuration for the PgBouncer node: ```ruby # Disable all components except Pgbouncer and Consul agent - roles ['pgbouncer_role'] + roles(['pgbouncer_role']) # PgBouncer configuration + pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) pgbouncer['users'] = { + 'gitlab-consul': { + # Generate it with: `gitlab-ctl pg-password-md5 gitlab-consul` + password: 'GITLAB_CONSUL_PASSWORD_HASH' + }, 'pgbouncer': { + # Generate it with: `gitlab-ctl pg-password-md5 pgbouncer` password: 'PGBOUNCER_PASSWORD_HASH' } } # Consul configuration consul['watchers'] = %w(postgresql) - consul['configuration'] = { retry_join: %w[CONSUL_SECONDARY1_IP CONSUL_SECONDARY2_IP CONSUL_SECONDARY3_IP] } - consul['monitoring_service_discovery'] = true ``` @@ -652,17 +679,17 @@ Follow the minimal configuration for the PgBouncer node: gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul ``` -1. Restart the PgBouncer service: +1. Reload the PgBouncer service: ```shell - gitlab-ctl restart pgbouncer + gitlab-ctl hup pgbouncer ``` -#### Step 4. Configure a Standby cluster on the secondary site +##### Step 4. Configure a Standby cluster on the secondary site NOTE: If you are converting a secondary site to a Patroni Cluster, you must start -on the PostgreSQL instance. It will become the Patroni Standby Leader instance, +on the PostgreSQL instance. It becomes the Patroni Standby Leader instance, and then you can switchover to another replica if you need. For each Patroni instance on the secondary site: @@ -676,21 +703,18 @@ For each Patroni instance on the secondary site: 1. Edit `/etc/gitlab/gitlab.rb` and add the following: ```ruby - roles ['consul_role', 'postgres_role'] + roles(['consul_role', 'patroni_role']) consul['enable'] = true consul['configuration'] = { retry_join: %w[CONSUL_SECONDARY1_IP CONSUL_SECONDARY2_IP CONSUL_SECONDARY3_IP] } - repmgr['enable'] = false - postgresql['md5_auth_cidr_addresses'] = [ 'PATRONI_SECONDARY1_IP/32', 'PATRONI_SECONDARY2_IP/32', 'PATRONI_SECONDARY3_IP/32', 'PATRONI_SECONDARY_PGBOUNCER/32', # Any other instance that needs access to the database as per documentation ] - patroni['enable'] = false patroni['standby_cluster']['enable'] = true patroni['standby_cluster']['host'] = 'INTERNAL_LOAD_BALANCER_PRIMARY_IP' patroni['standby_cluster']['port'] = INTERNAL_LOAD_BALANCER_PRIMARY_PORT @@ -699,6 +723,15 @@ For each Patroni instance on the secondary site: patroni['use_pg_rewind'] = true patroni['postgresql']['max_wal_senders'] = 5 # A minimum of three for one replica, plus two for each additional replica patroni['postgresql']['max_replication_slots'] = 5 # A minimum of three for one replica, plus two for each additional replica + + postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH' + postgresql['sql_replication_password'] = 'POSTGRESQL_REPLICATION_PASSWORD_HASH' + postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' + postgresql['listen_address'] = '0.0.0.0' # You can use a public or VPC address here instead + + gitlab_rails['dbpassword'] = 'POSTGRESQL_PASSWORD' + gitlab_rails['enable'] = true + gitlab_rails['auto_migrate'] = false ``` 1. Reconfigure GitLab for the changes to take effect. @@ -708,33 +741,11 @@ For each Patroni instance on the secondary site: gitlab-ctl reconfigure ``` -1. Remove the PostgreSQL data directory: - - WARNING: - If you are converting a secondary site to a Patroni Cluster, you must skip - this step on the PostgreSQL instance. - - ```shell - rm -rf /var/opt/gitlab/postgresql/data - ``` - -1. Edit `/etc/gitlab/gitlab.rb` to enable Patroni: - - ```ruby - patroni['enable'] = true - ``` - -1. Reconfigure GitLab for the changes to take effect: - - ```shell - gitlab-ctl reconfigure - ``` - ### Migrating from repmgr to Patroni 1. Before migrating, it is recommended that there is no replication lag between the primary and secondary sites and that replication is paused. In GitLab 13.2 and later, you can pause and resume replication with `gitlab-ctl geo-replication-pause` and `gitlab-ctl geo-replication-resume` on a Geo secondary database node. 1. Follow the [instructions to migrate repmgr to Patroni](../../postgresql/replication_and_failover.md#switching-from-repmgr-to-patroni). When configuring Patroni on each primary site database node, add `patroni['replication_slots'] = { '<slot_name>' => 'physical' }` -to `gitlab.rb` where `<slot_name>` is the name of the replication slot for your Geo secondary. This will ensure that Patroni recognizes the replication slot as permanent and will not drop it upon restarting. +to `gitlab.rb` where `<slot_name>` is the name of the replication slot for your Geo secondary. This ensures that Patroni recognizes the replication slot as permanent and not drop it upon restarting. 1. If database replication to the secondary was paused before migration, resume replication once Patroni is confirmed working on the primary. ### Migrating a single PostgreSQL node to Patroni @@ -750,14 +761,14 @@ With Patroni it's now possible to support that. In order to migrate the existing 1. [Configure a Standby Cluster](#step-4-configure-a-standby-cluster-on-the-secondary-site) on that single node machine. -You will end up with a "Standby Cluster" with a single node. That allows you to later on add additional Patroni nodes +You end up with a "Standby Cluster" with a single node. That allows you to later on add additional Patroni nodes by following the same instructions above. ### Configuring Patroni cluster for the tracking PostgreSQL database Secondary sites use a separate PostgreSQL installation as a tracking database to keep track of replication status and automatically recover from potential replication issues. -Omnibus automatically configures a tracking database when `roles ['geo_secondary_role']` is set. +Omnibus automatically configures a tracking database when `roles(['geo_secondary_role'])` is set. If you want to run this database in a highly available configuration, follow the instructions below. A production-ready and secure setup requires at least three Consul nodes, three @@ -782,7 +793,7 @@ Follow the minimal configuration for the PgBouncer node for the tracking databas ```ruby # Disable all components except Pgbouncer and Consul agent - roles ['pgbouncer_role'] + roles(['pgbouncer_role']) # PgBouncer configuration pgbouncer['users'] = { @@ -844,7 +855,7 @@ For each Patroni instance on the secondary site for the tracking database: ```ruby # Disable all components except PostgreSQL, Patroni, and Consul - roles ['patroni_role'] + roles(['patroni_role']) # Consul configuration consul['services'] = %w(postgresql) @@ -875,6 +886,7 @@ For each Patroni instance on the secondary site for the tracking database: # GitLab database settings gitlab_rails['db_database'] = 'gitlabhq_geo_production' gitlab_rails['db_username'] = 'gitlab_geo' + gitlab_rails['enable'] = true # Disable automatic database migrations gitlab_rails['auto_migrate'] = false @@ -934,8 +946,8 @@ Patroni implementation on Omnibus that do not allow us to manage two different clusters on the same machine, we recommend setting up a new Patroni cluster for the tracking database by following the same instructions above. -The secondary nodes will backfill the new tracking database, and no data -synchronization will be required. +The secondary nodes backfill the new tracking database, and no data +synchronization is required. ## Troubleshooting diff --git a/doc/administration/geo/setup/external_database.md b/doc/administration/geo/setup/external_database.md index 1b0082687e6..9e187424afa 100644 --- a/doc/administration/geo/setup/external_database.md +++ b/doc/administration/geo/setup/external_database.md @@ -24,11 +24,19 @@ developed and tested. We aim to be compatible with most external sudo -i ``` -1. Edit `/etc/gitlab/gitlab.rb` and add a **unique** ID for your node (arbitrary value): +1. Edit `/etc/gitlab/gitlab.rb` and add: ```ruby - # The unique identifier for the Geo node. - gitlab_rails['geo_node_name'] = '<node_name_here>' + ## + ## Geo Primary role + ## - configure dependent flags automatically to enable Geo + ## + roles ['geo_primary_role'] + + ## + ## The unique identifier for the Geo site. + ## + gitlab_rails['geo_node_name'] = '<geo_site_name_here>' ``` 1. Reconfigure the **primary** node for the change to take effect: @@ -49,7 +57,7 @@ developed and tested. We aim to be compatible with most external To set up an external database, you can either: -- Set up [streaming replication](https://www.postgresql.org/docs/11/warm-standby.html#STREAMING-REPLICATION-SLOTS) yourself (for example AWS RDS, bare metal not managed by Omnibus, etc.). +- Set up [streaming replication](https://www.postgresql.org/docs/12/warm-standby.html#STREAMING-REPLICATION-SLOTS) yourself (for example AWS RDS, bare metal not managed by Omnibus, etc.). - Perform the Omnibus configuration manually as follows. #### Leverage your cloud provider's tools to replicate the primary database @@ -64,8 +72,9 @@ cloud providers: - Amazon RDS - [Creating a Read Replica](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Create) - Azure Database for PostgreSQL - [Create and manage read replicas in Azure Database for PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/howto-read-replicas-portal) +- Google Cloud SQL - [Creating read replicas](https://cloud.google.com/sql/docs/postgres/replication/create-replica) -Once your read-only replica is set up, you can skip to [configure you secondary application node](#configure-secondary-application-nodes-to-use-the-external-read-replica). +Once your read-only replica is set up, you can skip to [configure your secondary application node](#configure-secondary-application-nodes-to-use-the-external-read-replica). #### Manually configure the primary database for replication @@ -182,9 +191,12 @@ to grant additional roles to your tracking database user (by default, this is - Amazon RDS requires the [`rds_superuser`](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.html#Appendix.PostgreSQL.CommonDBATasks.Roles) role. - Azure Database for PostgreSQL requires the [`azure_pg_admin`](https://docs.microsoft.com/en-us/azure/postgresql/howto-create-users#how-to-create-additional-admin-users-in-azure-database-for-postgresql) role. +- Google Cloud SQL requires the [`cloudsqlsuperuser`](https://cloud.google.com/sql/docs/postgres/users#default-users) role. -If you have an external database ready to be used as the tracking database, -follow the instructions below to use it: +This is for the installation of extensions during installation and upgrades. As an alternative, +[ensure the extensions are installed manually, and read about the problems that may arise during future GitLab upgrades](../../../install/postgresql_extensions.md). + +To setup an external tracking database, follow the instructions below: NOTE: If you want to use AWS RDS as a tracking database, make sure it has access to @@ -193,8 +205,12 @@ outbound rules do not apply to RDS PostgreSQL databases. Therefore, you need to rule to the read-replica's security group allowing any TCP traffic from the tracking database on port 5432. -1. Ensure that your secondary node can communicate with your tracking database by - manually changing the `pg_hba.conf` that is associated with your tracking database. +1. Set up PostgreSQL according to the + [database requirements document](../../../install/requirements.md#database). +1. Set up a `gitlab_geo` user with a password of your choice, create the `gitlabhq_geo_production` database, and make the user an owner of the database. You can see an example of this setup in the [installation from source documentation](../../../install/installation.md#6-database). +1. If you are **not** using a cloud-managed PostgreSQL database, ensure that your secondary + node can communicate with your tracking database by manually changing the + `pg_hba.conf` that is associated with your tracking database. Remember to restart PostgreSQL afterwards for the changes to take effect: ```plaintext @@ -226,9 +242,14 @@ the tracking database on port 5432. 1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) -1. Run the tracking database migrations: +1. The reconfigure should automatically create the database. If needed, you can perform this task manually. Note that this task (whether run by itself or during reconfigure) requires the database user to be a superuser. ```shell gitlab-rake geo:db:create + ``` + +1. The reconfigure should automatically migrate the database. You can migrate the database manually if needed, for example if `gitlab_rails['auto_migrate'] = false`: + + ```shell gitlab-rake geo:db:migrate ``` |