diff options
Diffstat (limited to 'doc/administration/geo')
34 files changed, 296 insertions, 129 deletions
diff --git a/doc/administration/geo/disaster_recovery/background_verification.md b/doc/administration/geo/disaster_recovery/background_verification.md index 89a93e45b1d..f2854d0538b 100644 --- a/doc/administration/geo/disaster_recovery/background_verification.md +++ b/doc/administration/geo/disaster_recovery/background_verification.md @@ -1,13 +1,13 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- # Automatic background verification **(PREMIUM ONLY)** -NOTE: **Note:** +NOTE: Automatic background verification of repositories and wikis was added in GitLab EE 10.6 but is enabled by default only on GitLab EE 11.1. You can disable or enable this feature manually by following diff --git a/doc/administration/geo/disaster_recovery/bring_primary_back.md b/doc/administration/geo/disaster_recovery/bring_primary_back.md index b73d3ba52a2..3f86e03bd7f 100644 --- a/doc/administration/geo/disaster_recovery/bring_primary_back.md +++ b/doc/administration/geo/disaster_recovery/bring_primary_back.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -13,7 +13,7 @@ restore your original configuration. This process consists of two steps: 1. Making the old **primary** node a **secondary** node. 1. Promoting a **secondary** node to a **primary** node. -CAUTION: **Caution:** +WARNING: If you have any doubts about the consistency of the data on this node, we recommend setting it up from scratch. ## Configure the former **primary** node to be a **secondary** node @@ -32,14 +32,14 @@ To bring the former **primary** node up to date: sudo gitlab-ctl start ``` - NOTE: **Note:** + NOTE: If you [disabled the **primary** node permanently](index.md#step-2-permanently-disable-the-primary-node), you need to undo those steps now. For Debian/Ubuntu you just need to run `sudo systemctl enable gitlab-runsvdir`. For CentOS 6, you need to install the GitLab instance from scratch and set it up as a **secondary** node by following [Setup instructions](../setup/index.md). In this case, you don't need to follow the next step. - NOTE: **Note:** + NOTE: If you [changed the DNS records](index.md#step-4-optional-updating-the-primary-domain-dns-record) for this node during disaster recovery procedure you may need to [block all the writes to this node](planned_failover.md#prevent-updates-to-the-primary-node) diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index 1e9b430834c..be84b260127 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -13,7 +13,7 @@ failover with minimal effort, in a disaster situation. See [Geo limitations](../index.md#limitations) for more information. -CAUTION: **Warning:** +WARNING: Disaster recovery for multi-secondary configurations is in **Alpha**. For the latest updates, check the [Disaster Recovery epic for complete maturity](https://gitlab.com/groups/gitlab-org/-/epics/590). Multi-secondary configurations require the complete re-synchronization and re-configuration of all non-promoted secondaries and @@ -36,7 +36,7 @@ order to avoid unnecessary data loss. ### Step 2. Permanently disable the **primary** node -CAUTION: **Warning:** +WARNING: If the **primary** node goes offline, there may be data saved on the **primary** node that has not been replicated to the **secondary** node. This data should be treated as lost if you proceed. @@ -58,7 +58,7 @@ must disable the **primary** node. sudo systemctl disable gitlab-runsvdir ``` - NOTE: **Note:** + NOTE: (**CentOS only**) In CentOS 6 or older, there is no easy way to prevent GitLab from being started if the machine reboots isn't available (see [Omnibus GitLab issue #3058](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3058)). It may be safest to uninstall the GitLab package completely: @@ -67,7 +67,7 @@ must disable the **primary** node. yum remove gitlab-ee ``` - NOTE: **Note:** + NOTE: (**Ubuntu 14.04 LTS**) If you are using an older version of Ubuntu or any other distribution based on the Upstart init system, you can prevent GitLab from starting if the machine reboots by doing the following: @@ -133,13 +133,14 @@ Note the following when promoting a secondary: ``` 1. Promote the **secondary** node to the **primary** node. - DANGER: **Warning:** + + WARNING: In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the secondary is paused fails. Do not pause replication before promoting a secondary. If the node is paused, be sure to resume before promoting. This issue has been fixed in GitLab 13.4 and later. - CAUTION: **Caution:** + WARNING: If the secondary node [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs a point-in-time recovery to the last known state. Data that was created on the primary while the secondary was paused will be lost. @@ -173,13 +174,13 @@ conjunction with multiple servers, as it can only perform changes on a **secondary** with only a single machine. Instead, you must do this manually. -DANGER: **Warning:** +WARNING: In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the secondary is paused fails. Do not pause replication before promoting a secondary. If the node is paused, be sure to resume before promoting. This issue has been fixed in GitLab 13.4 and later. -CAUTION: **Caution:** +WARNING: If the secondary node [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs a point-in-time recovery to the last known state. Data that was created on the primary while the secondary was paused will be lost. @@ -300,7 +301,7 @@ secondary domain, like changing Git remotes and API URLs. external_url 'https://<new_external_url>' ``` - NOTE: **Note:** + NOTE: Changing `external_url` won't prevent access via the old secondary URL, as long as the secondary DNS records are still intact. diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index 1238c4d8e2a..1468c5cd39d 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- diff --git a/doc/administration/geo/disaster_recovery/promotion_runbook.md b/doc/administration/geo/disaster_recovery/promotion_runbook.md index 7eb6ef01aee..eef1e7ae9dd 100644 --- a/doc/administration/geo/disaster_recovery/promotion_runbook.md +++ b/doc/administration/geo/disaster_recovery/promotion_runbook.md @@ -3,3 +3,6 @@ redirect_to: runbooks/planned_failover_single_node.md --- This document was moved to [another location](runbooks/planned_failover_single_node.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md index 3864445bbf4..f17990ce5f8 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md @@ -1,11 +1,11 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- -CAUTION: **Caution:** +WARNING: This runbook is in **alpha**. For complete, production-ready documentation, see the [disaster recovery documentation](../index.md). @@ -58,7 +58,7 @@ What is not covered: ### Preparation -NOTE: **Note:** +NOTE: Before following any of those steps, make sure you have `root` access to the **secondary** to promote it, since there isn't provided an automated way to promote a Geo replica and perform a failover. @@ -134,7 +134,7 @@ follow these steps to avoid unnecessary data loss: 1. Finish replicating and verifying all data: - CAUTION: **Caution:** + WARNING: Not all data is automatically replicated. Read more about [what is excluded](../planned_failover.md#not-all-data-is-automatically-replicated). @@ -163,12 +163,12 @@ follow these steps to avoid unnecessary data loss: 1. In this final step, you need to permanently disable the **primary** node. - CAUTION: **Caution:** + WARNING: When the **primary** node goes offline, there may be data saved on the **primary** node that has not been replicated to the **secondary** node. This data should be treated as lost if you proceed. - TIP: **Tip:** + NOTE: If you plan to [update the **primary** domain DNS record](../index.md#step-4-optional-updating-the-primary-domain-dns-record), you may wish to lower the TTL now to speed up propagation. @@ -188,12 +188,12 @@ follow these steps to avoid unnecessary data loss: sudo systemctl disable gitlab-runsvdir ``` - NOTE: **Note:** + NOTE: (**CentOS only**) In CentOS 6 or older, there is no easy way to prevent GitLab from being started if the machine reboots isn't available (see [Omnibus GitLab issue #3058](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3058)). It may be safest to uninstall the GitLab package completely with `sudo yum remove gitlab-ee`. - NOTE: **Note:** + NOTE: (**Ubuntu 14.04 LTS**) If you are using an older version of Ubuntu or any other distribution based on the Upstart init system, you can prevent GitLab from starting if the machine reboots as `root` with @@ -213,12 +213,12 @@ follow these steps to avoid unnecessary data loss: ### Promoting the **secondary** node -NOTE: **Note:** +NOTE: A new **secondary** should not be added at this time. If you want to add a new **secondary**, do this after you have completed the entire process of promoting the **secondary** to the **primary**. -CAUTION: **Caution:** +WARNING: If you encounter an `ActiveRecord::RecordInvalid: Validation failed: Name has already been taken` error during this process, read [the troubleshooting advice](../../replication/troubleshooting.md#fixing-errors-during-a-failover-or-when-promoting-a-secondary-to-a-primary-node). @@ -227,13 +227,13 @@ conjunction with multiple servers, as it can only perform changes on a **secondary** with only a single machine. Instead, you must do this manually. -DANGER: **Warning:** +WARNING: In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the secondary is paused fails. Do not pause replication before promoting a secondary. If the node is paused, be sure to resume before promoting. This issue has been fixed in GitLab 13.4 and later. -CAUTION: **Caution:** +WARNING: If the secondary node [has been paused](../../../geo/index.md#pausing-and-resuming-replication), this performs a point-in-time recovery to the last known state. Data that was created on the primary while the secondary was paused will be lost. diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md index 5e847030077..a2a9350e411 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md @@ -1,11 +1,11 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- -CAUTION: **Caution:** +WARNING: This runbook is in **alpha**. For complete, production-ready documentation, see the [disaster recovery documentation](../index.md). @@ -46,7 +46,7 @@ What is not covered: ### Preparation -NOTE: **Note:** +NOTE: Before following any of those steps, make sure you have `root` access to the **secondary** to promote it, since there isn't provided an automated way to promote a Geo replica and perform a failover. @@ -122,7 +122,7 @@ follow these steps to avoid unnecessary data loss: 1. Finish replicating and verifying all data: - CAUTION: **Caution:** + WARNING: Not all data is automatically replicated. Read more about [what is excluded](../planned_failover.md#not-all-data-is-automatically-replicated). @@ -151,12 +151,12 @@ follow these steps to avoid unnecessary data loss: 1. In this final step, you need to permanently disable the **primary** node. - CAUTION: **Caution:** + WARNING: When the **primary** node goes offline, there may be data saved on the **primary** node that has not been replicated to the **secondary** node. This data should be treated as lost if you proceed. - TIP: **Tip:** + NOTE: If you plan to [update the **primary** domain DNS record](../index.md#step-4-optional-updating-the-primary-domain-dns-record), you may wish to lower the TTL now to speed up propagation. @@ -176,12 +176,12 @@ follow these steps to avoid unnecessary data loss: sudo systemctl disable gitlab-runsvdir ``` - NOTE: **Note:** + NOTE: (**CentOS only**) In CentOS 6 or older, there is no easy way to prevent GitLab from being started if the machine reboots isn't available (see [Omnibus GitLab issue #3058](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3058)). It may be safest to uninstall the GitLab package completely with `sudo yum remove gitlab-ee`. - NOTE: **Note:** + NOTE: (**Ubuntu 14.04 LTS**) If you are using an older version of Ubuntu or any other distribution based on the Upstart init system, you can prevent GitLab from starting if the machine reboots as `root` with diff --git a/doc/administration/geo/glossary.md b/doc/administration/geo/glossary.md new file mode 100644 index 00000000000..e9d57284dd2 --- /dev/null +++ b/doc/administration/geo/glossary.md @@ -0,0 +1,112 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: howto +--- + + +# Geo Glossary + +NOTE: +We are updating the Geo documentation, user interface and commands to reflect these changes. Not all pages comply with +these definitions yet. + + These are the defined terms to describe all aspects of Geo. Using a set of clearly + defined terms helps us to communicate efficiently and avoids confusion. The language + on this page aims to be [ubiquitous](https://about.gitlab.com/handbook/communication/#ubiquitous-language) + and [as simple as possible](https://about.gitlab.com/handbook/communication/#simple-language). + + We provide example diagrams and statements to demonstrate correct usage of terms. + +| Term | Definition | Scope | Discouraged synonyms | +|---------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------|-------------------------------------------------| +| Node | An individual server that runs GitLab either with a specific role or as a whole (e.g. a Rails application node). In a cloud context this can be a specific machine type. | GitLab | instance, server | +| Site | One or a collection of nodes running a single GitLab application. A site can be single-node or multi-node. | GitLab | deployment, installation instance | +| Single-node site | A specific configuration of GitLab that uses exactly one node. | GitLab | single-server, single-instance +| Multi-node site | A specific configuration of GitLab that uses more than one node. | GitLab | multi-server, multi-instance, high availability | +| Primary site | A GitLab site that is configured to be read and writable. There can only be a single primary site. | Geo-specific | Geo deployment, Primary node | +| Secondary site(s) | GitLab site that is configured to be read-only. There can be one or more secondary sites. | Geo-specific | Geo deployment, Secondary node | +| Geo deployment | A collection of two or more GitLab sites with exactly one primary site being replicated by one or more secondary sites. | Geo-specific | | +| Reference architecture(s) | A [specified configuration of GitLab for a number of users](../reference_architectures/index.md), possibly including multiple nodes and multiple sites. | GitLab | | +| Promoting | Changing the role of a site from secondary to primary. | Geo-specific | | +| Demoting | Changing the role of a site from primary to secondary. | Geo-specific | | +| Failover | The entire process that shifts users from a primary Site to a secondary site. This includes promoting a secondary, but contains other parts as well e.g. scheduling maintenance. | Geo-specific | | + +## Examples + +### Single-node site + +```mermaid + graph TD + subgraph S-Site[Single-node site] + Node_3[GitLab node] + end +``` + +### Multi-node site + +```mermaid + graph TD + subgraph MN-Site[Multi-node site] + Node_1[Application node] + Node_2[Database node] + Node_3[Gitaly node] + end +``` + +### Geo deployment - Single-node sites + +This Geo deployment has a single-node primary site, a single-node secondary site: + +```mermaid + graph TD + subgraph Geo deployment + subgraph Primary[Primary site, single-node] + Node_1[GitLab node] + end + subgraph Secondary1[Secondary site 1, single-node] + Node_2[GitLab node] + end + end +``` + +### Geo deployment - Multi-node sites + +This Geo deployment has a multi-node primary site, a multi-node secondary site: + +```mermaid + graph TD + subgraph Geo deployment + subgraph Primary[Primary site, multi-node] + Node_1[Application node] + Node_2[Database node] + end + subgraph Secondary1[Secondary site 1, multi-node] + Node_5[Application node] + Node_6[Database node] + end + end +``` + +### Geo deployment - Mixed sites + +This Geo deployment has a multi-node primary site, a multi-node secondary site and another single-node secondary site: + +```mermaid + graph TD + subgraph Geo deployment + subgraph Primary[Primary site, multi-node] + Node_1[Application node] + Node_2[Database node] + Node_3[Gitaly node] + end + subgraph Secondary1[Secondary site 1, multi-node] + Node_5[Application node] + Node_6[Database node] + end + subgraph Secondary2[Secondary site 2, single-node] + Node_7[Single GitLab node] + end + end +``` diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index 02b907ae237..334f05ef3ce 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -17,7 +17,7 @@ Geo is the solution for widely distributed development teams and for providing a ## Overview -CAUTION: **Caution:** +WARNING: Geo undergoes significant changes from release to release. Upgrades **are** supported and [documented](#updating-geo), but you should ensure that you're using the right version of the documentation for your installation. Fetching large repositories can take a long time for teams located far from a single GitLab instance. @@ -51,6 +51,11 @@ Geo provides: - Authentication system hooks: **Secondary** nodes receives all authentication data (like user accounts and logins) from the **primary** instance. - An intuitive UI: **Secondary** nodes use the same web interface your team has grown accustomed to. In addition, there are visual notifications that block write operations and make it clear that a user is on a **secondary** node. +### Gitaly Cluster + +Geo should not be confused with [Gitaly Cluster](../gitaly/praefect.md). For more information about +the difference between Geo and Gitaly Cluster, see [Gitaly Cluster compared to Geo](../gitaly/praefect.md#gitaly-cluster-compared-to-geo). + ## How it works Your Geo instance can be used for cloning and fetching projects, in addition to reading any data. This will make working with large repositories over large distances much faster. @@ -119,7 +124,7 @@ The following are required to run Geo: - Git-lfs 2.4.2+ on the user side when using LFS - All nodes must run the same GitLab version. -Additionally, check GitLab's [minimum requirements](../../install/requirements.md), +Additionally, check the GitLab [minimum requirements](../../install/requirements.md), and we recommend you use: - At least GitLab Enterprise Edition 10.0 for basic Geo features. @@ -138,11 +143,11 @@ The following table lists basic ports that must be open between the **primary** See the full list of ports used by GitLab in [Package defaults](https://docs.gitlab.com/omnibus/package-information/defaults.html) -NOTE: **Note:** +NOTE: [Web terminal](../../ci/environments/index.md#web-terminals) support requires your load balancer to correctly handle WebSocket connections. When using HTTP or HTTPS proxying, your load balancer must be configured to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the [web terminal](../integration/terminal.md) integration guide for more details. -NOTE: **Note:** +NOTE: When using HTTPS protocol for port 443, you will need to add an SSL certificate to the load balancers. If you wish to terminate SSL at the GitLab application server instead, use TCP protocol. @@ -150,7 +155,7 @@ If you wish to terminate SSL at the GitLab application server instead, use TCP p We recommend that if you use LDAP on your **primary** node, you also set up secondary LDAP servers on each **secondary** node. Otherwise, users will not be able to perform Git operations over HTTP(s) on the **secondary** node using HTTP Basic Authentication. However, Git via SSH and personal access tokens will still work. -NOTE: **Note:** +NOTE: It is possible for all **secondary** nodes to share an LDAP server, but additional latency can be an issue. Also, consider what LDAP server will be available in a [disaster recovery](disaster_recovery/index.md) scenario if a **secondary** node is promoted to be a **primary** node. Check for instructions on how to set up replication in your LDAP service. Instructions will be different depending on the software or service used. For example, OpenLDAP provides [these instructions](https://www.openldap.org/doc/admin24/replication.html). @@ -196,19 +201,21 @@ For information on how to update your Geo nodes to the latest GitLab version, se > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35913) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2. -DANGER: **Warning:** +WARNING: In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the secondary is paused fails. Do not pause replication before promoting a secondary. If the node is paused, be sure to resume before promoting. This issue has been fixed in GitLab 13.4 and later. -CAUTION: **Caution:** +WARNING: Pausing and resuming of replication is currently only supported for Geo installations using an Omnibus GitLab-managed database. External databases are currently not supported. In some circumstances, like during [upgrades](replication/updating_the_geo_nodes.md) or a [planned failover](disaster_recovery/planned_failover.md), it is desirable to pause replication between the primary and secondary. -Pausing and resuming replication is done via a command line tool from the secondary node. +Pausing and resuming replication is done via a command line tool from the secondary node where the `postgresql` service is enabled. + +If `postgresql` is on a standalone database node, ensure that `gitlab.rb` on that node contains the configuration line `gitlab_rails['geo_node_name'] = 'node_name'`, where `node_name` is the same as the `geo_name_name` on the application node. **To Pause: (from secondary)** @@ -250,6 +257,16 @@ For more information on tuning Geo, see [Tuning Geo](replication/tuning.md). For an example of how to set up a location-aware Git remote URL with AWS Route53, see [Location-aware Git remote URL with AWS Route53](replication/location_aware_git_url.md). +### Backfill + +Once a **secondary** node is set up, it will start replicating missing data from +the **primary** node in a process known as **backfill**. You can monitor the +synchronization process on each Geo node from the **primary** node's **Geo Nodes** +dashboard in your browser. + +Failures that happen during a backfill are scheduled to be retried at the end +of the backfill. + ## Remove Geo node For more information on removing a Geo node, see [Removing **secondary** Geo nodes](replication/remove_geo_node.md). @@ -260,7 +277,7 @@ To find out how to disable Geo, see [Disabling Geo](replication/disable_geo.md). ## Limitations -CAUTION: **Caution:** +WARNING: This list of limitations only reflects the latest version of GitLab. If you are using an older version, extra limitations may be in place. - Pushing directly to a **secondary** node redirects (for HTTP) or proxies (for SSH) the request to the **primary** node instead of [handling it directly](https://gitlab.com/gitlab-org/gitlab/-/issues/1381), except when using Git over HTTP with credentials embedded within the URI. For example, `https://user:password@secondary.tld`. @@ -270,7 +287,6 @@ This list of limitations only reflects the latest version of GitLab. If you are - Real-time updates of issues/merge requests (for example, via long polling) doesn't work on the **secondary** node. - [Selective synchronization](replication/configuration.md#selective-synchronization) applies only to files and repositories. Other datasets are replicated to the **secondary** node in full, making it inappropriate for use as an access control mechanism. - Object pools for forked project deduplication work only on the **primary** node, and are duplicated on the **secondary** node. -- [External merge request diffs](../merge_request_diffs.md) will not be replicated if they are on-disk, and viewing merge requests will fail. However, external MR diffs in object storage **are** supported. The default configuration (in-database) does work. - GitLab Runners cannot register with a **secondary** node. Support for this is [planned for the future](https://gitlab.com/gitlab-org/gitlab/-/issues/3294). - Geo **secondary** nodes can not be configured to [use high-availability configurations of PostgreSQL](https://gitlab.com/groups/gitlab-org/-/epics/2536). diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index 43a422c4bc3..42501e16f5f 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -9,7 +9,7 @@ type: howto ## Configuring a new **secondary** node -NOTE: **Note:** +NOTE: This is the final step in setting up a **secondary** Geo node. Stages of the setup process must be completed in the documented order. Before attempting the steps in this stage, [complete all prior stages](../setup/index.md#using-omnibus-gitlab). @@ -23,7 +23,7 @@ The basic steps of configuring a **secondary** node are to: You are encouraged to first read through all the steps before executing them in your testing/production environment. -NOTE: **Note:** +NOTE: **Do not** set up any custom authentication for the **secondary** nodes. This will be handled by the **primary** node. Any change that requires access to the **Admin Area** needs to be done in the **primary** node because the **secondary** node is a read-only replica. @@ -157,7 +157,7 @@ keys must be manually replicated to the **secondary** node. for file in /etc/ssh/ssh_host_*_key.pub; do ssh-keygen -lf $file; done ``` - NOTE: **Note:** + NOTE: The output for private keys and public keys command should generate the same fingerprint. 1. Restart `sshd` on your **secondary** node: diff --git a/doc/administration/geo/replication/database.md b/doc/administration/geo/replication/database.md index da5376190b9..099a73e93d8 100644 --- a/doc/administration/geo/replication/database.md +++ b/doc/administration/geo/replication/database.md @@ -3,3 +3,6 @@ redirect_to: '../setup/database.md' --- This document was moved to [another location](../setup/index.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index 9e896f2c07c..0d2922c3347 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -28,13 +28,13 @@ verification methods: | Database | Application data in PostgreSQL | Native | Native | | Database | Redis | _N/A_ (*1*) | _N/A_ | | Database | Elasticsearch | Native | Native | -| Database | Personal snippets | PostgreSQL Replication | PostgreSQL Replication | -| Database | Project snippets | PostgreSQL Replication | PostgreSQL Replication | | Database | SSH public keys | PostgreSQL Replication | PostgreSQL Replication | | Git | Project repository | Geo with Gitaly | Gitaly Checksum | | Git | Project wiki repository | Geo with Gitaly | Gitaly Checksum | | Git | Project designs repository | Geo with Gitaly | Gitaly Checksum | | Git | Object pools for forked project deduplication | Geo with Gitaly | _Not implemented_ | +| Git | Project Snippets | Geo with Gitaly | _Not implemented_ | +| Git | Personal Snippets | Geo with Gitaly | _Not implemented_ | | Blobs | User uploads _(filesystem)_ | Geo with API | _Not implemented_ | | Blobs | User uploads _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | LFS objects _(filesystem)_ | Geo with API | _Not implemented_ | @@ -81,6 +81,9 @@ Each project can have at most 3 different repositories: They all live in the same shard and share the same base name with a `-wiki` and `-design` suffix for Wiki and Design Repository cases. +Besides that, there are snippet repositories. They can be connected to a project or to some specific user. +Both types will be synced to a secondary node. + ### Blobs GitLab stores files and blobs such as Issue attachments or LFS objects into either: @@ -160,7 +163,7 @@ To enable, such as for package file replication: Feature.enable(:geo_package_file_replication) ``` -DANGER: **Warning:** +WARNING: Features not on this list, or with **No** in the **Replicated** column, are not replicated on the **secondary** node. Failing over without manually replicating data from those features will cause the data to be **lost**. @@ -192,7 +195,7 @@ successfully, you must replicate their data using some other means. | [Generic packages](../../../user/packages/generic_packages/index.md) | **Yes** (13.5) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | | [Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_terraform_state_version_replication`, enabled by default | | [External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_merge_request_diff_replication`, enabled by default | -| [Versioned snippets](../../../user/snippets.md#versioned-snippets) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2810) | No | | +| [Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2810) | No | | | [Server-side Git hooks](../../server_hooks.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1867) | No | No | | | [Elasticsearch integration](../../../integration/elasticsearch.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | | | [GitLab Pages](../../pages/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | No | | diff --git a/doc/administration/geo/replication/disable_geo.md b/doc/administration/geo/replication/disable_geo.md index dabf4499f9d..e8767a56266 100644 --- a/doc/administration/geo/replication/disable_geo.md +++ b/doc/administration/geo/replication/disable_geo.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- diff --git a/doc/administration/geo/replication/docker_registry.md b/doc/administration/geo/replication/docker_registry.md index 5403edc69da..f92a878662e 100644 --- a/doc/administration/geo/replication/docker_registry.md +++ b/doc/administration/geo/replication/docker_registry.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -17,7 +17,7 @@ distributed storage (`azure`, `gcs`, `s3`, `swift`, or `oss`) for your Docker Registry on the **primary** node, you can use the same storage for a **secondary** Docker Registry as well. For more information, read the [Load balancing considerations](https://docs.docker.com/registry/deploying/#load-balancing-considerations) -when deploying the Registry, and how to set up the storage driver for GitLab's +when deploying the Registry, and how to set up the storage driver for the GitLab integrated [Container Registry](../../packages/container_registry.md#use-object-storage). ## Replicating Docker Registry @@ -64,17 +64,17 @@ We need to make Docker Registry send notification events to the ] ``` - NOTE: **Note:** + NOTE: Replace `<replace_with_a_secret_token>` with a case sensitive alphanumeric string that starts with a letter. You can generate one with `< /dev/urandom tr -dc _A-Z-a-z-0-9 | head -c 32 | sed "s/^[0-9]*//"; echo` - NOTE: **Note:** + NOTE: If you use an external Registry (not the one integrated with GitLab), you must add these settings to its configuration yourself. In this case, you will also have to specify notification secret in `registry.notification_secret` section of `/etc/gitlab/gitlab.rb` file. - NOTE: **Note:** + NOTE: If you use GitLab HA, you will also have to specify the notification secret in `registry.notification_secret` section of `/etc/gitlab/gitlab.rb` file for every web node. diff --git a/doc/administration/geo/replication/external_database.md b/doc/administration/geo/replication/external_database.md index 0d0cd8a37d5..dfaf6819fa0 100644 --- a/doc/administration/geo/replication/external_database.md +++ b/doc/administration/geo/replication/external_database.md @@ -3,3 +3,6 @@ redirect_to: '../setup/external_database.md' --- This document was moved to [another location](../setup/external_database.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/geo/replication/faq.md b/doc/administration/geo/replication/faq.md index f7f391b360e..ab8199c3717 100644 --- a/doc/administration/geo/replication/faq.md +++ b/doc/administration/geo/replication/faq.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md index c28688930b5..2d701458e66 100644 --- a/doc/administration/geo/replication/geo_validation_tests.md +++ b/doc/administration/geo/replication/geo_validation_tests.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- diff --git a/doc/administration/geo/replication/high_availability.md b/doc/administration/geo/replication/high_availability.md index 214f15b7565..1da4246db1f 100644 --- a/doc/administration/geo/replication/high_availability.md +++ b/doc/administration/geo/replication/high_availability.md @@ -3,3 +3,6 @@ redirect_to: 'multiple_servers.md' --- This document was moved to [another location](multiple_servers.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/geo/replication/img/geo_node_dashboard.png b/doc/administration/geo/replication/img/geo_node_dashboard.png Binary files differindex 53ca4767c5b..8b9aceba825 100644 --- a/doc/administration/geo/replication/img/geo_node_dashboard.png +++ b/doc/administration/geo/replication/img/geo_node_dashboard.png diff --git a/doc/administration/geo/replication/img/geo_node_healthcheck.png b/doc/administration/geo/replication/img/geo_node_healthcheck.png Binary files differdeleted file mode 100644 index 33a31f7ab49..00000000000 --- a/doc/administration/geo/replication/img/geo_node_healthcheck.png +++ /dev/null diff --git a/doc/administration/geo/replication/index.md b/doc/administration/geo/replication/index.md index 5e9dd73ecc5..955e05491d2 100644 --- a/doc/administration/geo/replication/index.md +++ b/doc/administration/geo/replication/index.md @@ -3,3 +3,6 @@ redirect_to: '../index.md' --- This document was moved to [another location](../index.md). + +<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/geo/replication/location_aware_git_url.md b/doc/administration/geo/replication/location_aware_git_url.md index 82fda8b0fc2..f2cf8c9bda0 100644 --- a/doc/administration/geo/replication/location_aware_git_url.md +++ b/doc/administration/geo/replication/location_aware_git_url.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -18,7 +18,7 @@ Though these instructions use [AWS Route53](https://aws.amazon.com/route53/), other services such as [Cloudflare](https://www.cloudflare.com/) could be used as well. -NOTE: **Note:** +NOTE: You can also use a load balancer to distribute web UI or API traffic to [multiple Geo **secondary** nodes](../../../user/admin_area/geo_nodes.md#multiple-secondary-nodes-behind-a-load-balancer). Importantly, the **primary** node cannot yet be included. See the feature request diff --git a/doc/administration/geo/replication/multiple_servers.md b/doc/administration/geo/replication/multiple_servers.md index 7d65d2165c5..0ab972c9077 100644 --- a/doc/administration/geo/replication/multiple_servers.md +++ b/doc/administration/geo/replication/multiple_servers.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -27,7 +27,7 @@ network topology of your deployment. The only external way to access the two Geo deployments is by HTTPS at `gitlab.us.example.com` and `gitlab.eu.example.com` in the example above. -NOTE: **Note:** +NOTE: The **primary** and **secondary** Geo deployments must be able to communicate to each other over HTTPS. ## Redis and PostgreSQL for multiple nodes @@ -37,7 +37,7 @@ Geo supports: - Redis and PostgreSQL on the **primary** node configured for multiple nodes. - Redis on **secondary** nodes configured for multiple nodes. -NOTE: **Note:** +NOTE: Support for PostgreSQL on **secondary** nodes in multi-node configuration [is planned](https://gitlab.com/groups/gitlab-org/-/epics/2536). @@ -48,7 +48,7 @@ For more information about setting up a multi-node PostgreSQL cluster and Redis [PostgreSQL](../../postgresql/replication_and_failover.md) and [Redis](../../redis/replication_and_failover.md), respectively. -NOTE: **Note:** +NOTE: It is possible to use cloud hosted services for PostgreSQL and Redis, but this is beyond the scope of this document. ## Prerequisites: Two working GitLab multi-node clusters @@ -90,7 +90,7 @@ The following steps enable a GitLab cluster to serve as the **primary** node. After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. -NOTE: **Note:** +NOTE: PostgreSQL and Redis should have already been disabled on the application servers, and connections from the application servers to those services on the backend servers configured, during normal GitLab multi-node set up. See @@ -136,13 +136,13 @@ documentation: - [Gitaly](../../gitaly/index.md), which will store data that is synchronized from the **primary** node. -NOTE: **Note:** +NOTE: [NFS](../../nfs.md) can be used in place of Gitaly but is not recommended. ### Step 2: Configure the main read-only replica PostgreSQL database on the **secondary** node -NOTE: **Note:** +NOTE: The following documentation assumes the database will be run on a single node only. Multi-node PostgreSQL on **secondary** nodes is [not currently supported](https://gitlab.com/groups/gitlab-org/-/epics/2536). @@ -174,6 +174,12 @@ the **primary** database. Use the following as a guide. roles ['geo_secondary_role', 'postgres_role'] ## + ## The unique identifier for the Geo node. + ## This should match the secondary's application node. + ## + gitlab_rails['geo_node_name'] = '<node_name_here>' + + ## ## Secondary address ## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node ## - replace '<tracking_database_ip>' with the public or VPC address of your Geo tracking database node @@ -226,7 +232,7 @@ If using an external PostgreSQL instance, refer also to ### Step 3: Configure the tracking database on the **secondary** node -NOTE: **Note:** +NOTE: This documentation assumes the tracking database will be run on only a single machine, rather than as a PostgreSQL cluster. @@ -359,14 +365,14 @@ then make the following modifications: registry['gid'] = 9002 ``` -NOTE: **Note:** +NOTE: If you had set up PostgreSQL cluster using the omnibus package and you had set up `postgresql['sql_user_password'] = 'md5 digest of secret'` setting, keep in mind that `gitlab_rails['db_password']` and `geo_secondary['db_password']` mentioned above contains the plaintext passwords. This is used to let the Rails servers connect to the databases. -NOTE: **Note:** +NOTE: Make sure that current node IP is listed in `postgresql['md5_auth_cidr_addresses']` setting of your remote database. After making these changes [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. diff --git a/doc/administration/geo/replication/object_storage.md b/doc/administration/geo/replication/object_storage.md index 3f5ba1939b8..ce791d66b34 100644 --- a/doc/administration/geo/replication/object_storage.md +++ b/doc/administration/geo/replication/object_storage.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -25,7 +25,7 @@ To have: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10586) in GitLab 12.4. -CAUTION: **Caution:** +WARNING: This is a [**beta** feature](https://about.gitlab.com/handbook/product/#beta) and is not ready yet for production use at any scale. The main limitations are a lack of testing at scale and no verification of any replicated data. **Secondary** nodes can replicate files stored on the **primary** node regardless of @@ -35,8 +35,8 @@ To enable GitLab replication, you must: 1. Go to **Admin Area > Geo**. 1. Press **Edit** on the **secondary** node. -1. Enable the **Allow this secondary node to replicate content on Object Storage** - checkbox. +1. In the **Synchronization Settings** section, find the **Allow this secondary node to replicate content on Object Storage** + checkbox to enable it. For LFS, follow the documentation to [set up LFS object storage](../../lfs/index.md#storing-lfs-objects-in-remote-object-storage). diff --git a/doc/administration/geo/replication/remove_geo_node.md b/doc/administration/geo/replication/remove_geo_node.md index 2b01231241c..519b93b447b 100644 --- a/doc/administration/geo/replication/remove_geo_node.md +++ b/doc/administration/geo/replication/remove_geo_node.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -42,7 +42,7 @@ Once GitLab has been uninstalled from the **secondary** node, the replication sl sudo gitlab-psql ``` - NOTE: **Note:** + NOTE: Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. 1. Find the name of the relevant replication slot. This is the slot that is specified with `--slot-name` when running the replicate command: `gitlab-ctl replicate-geo-database`. diff --git a/doc/administration/geo/replication/security_review.md b/doc/administration/geo/replication/security_review.md index 28b8c8a9d51..7c15662340c 100644 --- a/doc/administration/geo/replication/security_review.md +++ b/doc/administration/geo/replication/security_review.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -33,7 +33,7 @@ from [owasp.org](https://owasp.org/). ### How can the data be classified into categories according to its sensitivity? -- GitLab’s model of sensitivity is centered around public vs. internal vs. +- The GitLab model of sensitivity is centered around public vs. internal vs. private projects. Geo replicates them all indiscriminately. “Selective sync” exists for files and repositories (but not database content), which would permit only less-sensitive projects to be replicated to a **secondary** node if desired. diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 15e3d3ff0a8..2c0160ed02a 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -35,7 +35,7 @@ to help identify if something is wrong: - Is the node's secondary tracking database connected? - Is the node's secondary tracking database up-to-date? - + For information on how to resolve common errors reported from the UI, see [Fixing Common Errors](#fixing-common-errors). @@ -308,7 +308,8 @@ log data to build up in `pg_xlog`. Removing the unused slots can reduce the amou sudo gitlab-psql ``` - Note: **Note:** Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. + NOTE: + Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. 1. View your replication slots with: @@ -425,6 +426,11 @@ GitLab you are running. GitLab versions 11.11.x or 12.0.x are affected by To resolve the issue, upgrade to GitLab 12.1 or newer. +### Failures during backfill + +During a [backfill](../index.md#backfill), failures are scheduled to be retried at the end +of the backfill queue, therefore these failures only clear up **after** the backfill completes. + ### Resetting Geo **secondary** node replication If you get a **secondary** node in a broken state and want to reset the replication state, @@ -461,13 +467,13 @@ to start again from scratch, there are a few steps that can help you: chown git:git /var/opt/gitlab/git-data/repositories ``` - TIP: **Tip:** + NOTE: You may want to remove the `/var/opt/gitlab/git-data/repositories.old` in the future as soon as you confirmed that you don't need it anymore, to save disk space. 1. _(Optional)_ Rename other data folders and create new ones - CAUTION: **Caution:** + WARNING: You may still have files on the **secondary** node that have been removed from **primary** node but removal have not been reflected. If you skip this step, they will never be removed from this Geo node. diff --git a/doc/administration/geo/replication/tuning.md b/doc/administration/geo/replication/tuning.md index fe0b189863c..183180e5ade 100644 --- a/doc/administration/geo/replication/tuning.md +++ b/doc/administration/geo/replication/tuning.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- diff --git a/doc/administration/geo/replication/updating_the_geo_nodes.md b/doc/administration/geo/replication/updating_the_geo_nodes.md index af59e45250c..ff3a1e8689b 100644 --- a/doc/administration/geo/replication/updating_the_geo_nodes.md +++ b/doc/administration/geo/replication/updating_the_geo_nodes.md @@ -1,13 +1,13 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- # Updating the Geo nodes **(PREMIUM ONLY)** -CAUTION: **Warning:** +WARNING: Read these sections carefully before updating your Geo nodes. Not following version-specific update steps may result in unexpected downtime. If you have any specific questions, [contact Support](https://about.gitlab.com/support/#contact-support). @@ -20,7 +20,7 @@ Updating Geo nodes involves performing: ## General update steps -NOTE: **Note:** +NOTE: These general update steps are not intended for [high-availability deployments](https://docs.gitlab.com/omnibus/update/README.html#multi-node--ha-deployment), and will cause downtime. If you want to avoid downtime, consider using [zero downtime updates](https://docs.gitlab.com/omnibus/update/README.html#zero-downtime-updates). To update the Geo nodes when a new GitLab version is released, update **primary** diff --git a/doc/administration/geo/replication/using_a_geo_server.md b/doc/administration/geo/replication/using_a_geo_server.md index aeed5a44b30..743b1b384db 100644 --- a/doc/administration/geo/replication/using_a_geo_server.md +++ b/doc/administration/geo/replication/using_a_geo_server.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- diff --git a/doc/administration/geo/replication/version_specific_updates.md b/doc/administration/geo/replication/version_specific_updates.md index 9ea5283812e..0d8eba555ab 100644 --- a/doc/administration/geo/replication/version_specific_updates.md +++ b/doc/administration/geo/replication/version_specific_updates.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -11,6 +11,10 @@ Check this document if it includes instructions for the version you are updating These steps go together with the [general steps](updating_the_geo_nodes.md#general-update-steps) for updating Geo nodes. +## Updating to GitLab 13.5 + +In GitLab 13.5, there is a [regression that prevents viewing a list of container repositories and registries](https://gitlab.com/gitlab-org/gitlab/-/issues/285475) on Geo secondaries. This issue is fixed in GitLab 13.6.1 and later. + ## Updating to GitLab 13.3 In GitLab 13.3, Geo removed the PostgreSQL [Foreign Data Wrapper](https://www.postgresql.org/docs/11/postgres-fdw.html) dependency for the tracking database. @@ -24,7 +28,7 @@ DROP SERVER gitlab_secondary CASCADE; DROP EXTENSION IF EXISTS postgres_fdw; ``` -DANGER: **Warning:** +WARNING: In GitLab 13.3, promoting a secondary node to a primary while the secondary is paused fails. Do not pause replication before promoting a secondary. If the node is paused, be sure to resume before promoting. To avoid this issue, @@ -50,7 +54,7 @@ the recommended procedure, see the ## Updating to GitLab 12.9 -CAUTION: **Warning:** +WARNING: GitLab 12.9.0 through GitLab 12.9.3 are affected by [a bug that stops repository verification](https://gitlab.com/gitlab-org/gitlab/-/issues/213523). The issue is fixed in GitLab 12.9.4. Upgrade to GitLab 12.9.4 or later. @@ -81,7 +85,7 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade ## Updating to GitLab 12.7 -DANGER: **Warning:** +WARNING: Only upgrade to GitLab 12.7.5 or later. Do not upgrade to versions 12.7.0 through 12.7.4 because there is [an initialization order bug](https://gitlab.com/gitlab-org/gitlab/-/issues/199672) that causes Geo @@ -141,7 +145,7 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade ## Updating to GitLab 12.3 -DANGER: **Warning:** +WARNING: If the existing PostgreSQL server version is 9.6.x, it is recommended to upgrade to GitLab 12.4 or later. By default, GitLab 12.3 attempts to update the embedded PostgreSQL server from 9.6 to 10.9. In certain circumstances, it will @@ -155,7 +159,7 @@ For the recommended procedure, see the ## Updating to GitLab 12.2 -DANGER: **Warning:** +WARNING: If the existing PostgreSQL server version is 9.6.x, it is recommended to upgrade to GitLab 12.4 or later. By default, GitLab 12.2 attempts to update the embedded PostgreSQL server from 9.6 to 10.9. In certain circumstances, it will @@ -185,7 +189,7 @@ The restart avoids a version mismatch when PostgreSQL tries to load the FDW exte ## Updating to GitLab 12.1 -DANGER: **Warning:** +WARNING: If the existing PostgreSQL server version is 9.6.x, it is recommended to upgrade to GitLab 12.4 or later. By default, GitLab 12.1 attempts to update the embedded PostgreSQL server from 9.6 to 10.9. In certain circumstances, it will @@ -199,14 +203,14 @@ For the recommended procedure, see the ## Updating to GitLab 12.0 -CAUTION: **Warning:** +WARNING: This version is affected by a [bug that results in new LFS objects not being replicated to Geo secondary nodes](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). The issue is fixed in GitLab 12.1; be sure to upgrade to GitLab 12.1 or later. ## Updating to GitLab 11.11 -CAUTION: **Warning:** +WARNING: This version is affected by a [bug that results in new LFS objects not being replicated to Geo secondary nodes](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). The issue is fixed in GitLab 12.1; be sure to upgrade to GitLab 12.1 or later. @@ -372,7 +376,7 @@ the now-unused SSH keys from your secondaries, as they may cause problems if the ### Hashed Storage -CAUTION: **Warning:** +WARNING: Hashed storage is in **Alpha**. It is considered experimental and not production-ready. See [Hashed Storage](../../repository_storage_types.md) for more detail. @@ -383,7 +387,7 @@ migrated we recommend leaving Hashed Storage enabled. ## Updating to GitLab 10.1 -CAUTION: **Warning:** +WARNING: Hashed storage is in **Alpha**. It is considered experimental and not production-ready. See [Hashed Storage](../../repository_storage_types.md) for more detail. diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index 24e55d26997..9778e08a30b 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -1,19 +1,19 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- # Geo database replication **(PREMIUM ONLY)** -NOTE: **Note:** +NOTE: If your GitLab installation uses external (not managed by Omnibus) PostgreSQL instances, the Omnibus roles will not be able to perform all necessary configuration steps. In this case, [follow the Geo with external PostgreSQL instances document instead](external_database.md). -NOTE: **Note:** +NOTE: The stages of the setup process must be completed in the documented order. Before attempting the steps in this stage, [complete all prior stages](../setup/index.md#using-omnibus-gitlab). @@ -44,7 +44,7 @@ The following guide assumes that: you have a new **secondary** server set up with the same versions of the OS, PostgreSQL, and GitLab on all nodes. -CAUTION: **Warning:** +WARNING: Geo works with streaming replication. Logical replication is not supported at this time. There is an [issue where support is being discussed](https://gitlab.com/gitlab-org/gitlab/-/issues/7420). @@ -79,7 +79,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o 1. GitLab 10.4 and up only: Do the following to make sure the `gitlab` database user has a password defined: - NOTE: **Note:** + NOTE: Until FDW settings are removed in GitLab version 14.0, avoid using single or double quotes in the password for PostgreSQL as that will lead to errors when reconfiguring. @@ -134,7 +134,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o connect to the **primary** node's database. For this reason, we need the address of each node. - NOTE: **Note:** + NOTE: For external PostgreSQL instances, see [additional instructions](external_database.md). If you are using a cloud provider, you can lookup the addresses for each @@ -151,7 +151,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ## ## Public address ## - echo "External address: $(curl --silent ipinfo.io/ip)" + echo "External address: $(curl --silent "ipinfo.io/ip")" ``` In most cases, the following addresses will be used to configure GitLab @@ -171,7 +171,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o corresponding to the given address. See [the PostgreSQL documentation](https://www.postgresql.org/docs/11/runtime-config-connection.html) for more details. - NOTE: **Note:** + NOTE: If you need to use `0.0.0.0` or `*` as the listen_address, you will also need to add `127.0.0.1/32` to the `postgresql['md5_auth_cidr_addresses']` setting, to allow Rails to connect through `127.0.0.1`. For more information, see [omnibus-5258](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5258). @@ -290,7 +290,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o gitlab-ctl stop sidekiq ``` - NOTE: **Note:** + NOTE: This step is important so we don't try to execute anything before the node is fully configured. 1. [Check TCP connectivity](../../raketasks/maintenance.md) to the **primary** node's PostgreSQL server: @@ -299,7 +299,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o gitlab-rake gitlab:tcp_check[<primary_node_ip>,5432] ``` - NOTE: **Note:** + NOTE: If this step fails, you may be using the wrong IP address, or a firewall may be preventing access to the server. Check the IP address, paying close attention to the difference between public and private addresses and ensure @@ -404,7 +404,7 @@ needed files for streaming replication. The directories used are the defaults that are set up in Omnibus. If you have changed any defaults, configure it as you see fit replacing the directories and paths. -CAUTION: **Warning:** +WARNING: Make sure to run this on the **secondary** server as it removes all PostgreSQL's data before running `pg_basebackup`. @@ -421,7 +421,7 @@ data before running `pg_basebackup`. 1. Execute the command below to start a backup/restore and begin the replication - CAUTION: **Warning:** + WARNING: Each Geo **secondary** node must have its own unique replication slot name. Using the same slot name between two secondaries will break PostgreSQL replication. @@ -431,14 +431,9 @@ data before running `pg_basebackup`. --host=<primary_node_ip> ``` - NOTE: **Note:** + NOTE: Replication slot names must only contain lowercase letters, numbers, and the underscore character. - NOTE: **Note:** - In GitLab 13.4, a seed project is added when GitLab is first installed. This makes it necessary to pass `--force` even - on a new Geo secondary node. There is an [issue to account for seed projects](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5618) - when checking the database. - When prompted, enter the _plaintext_ password you set up for the `gitlab_replicator` user in the first step. @@ -499,6 +494,8 @@ This experimental implementation has the following limitations: For instructions about how to set up Patroni on the primary node, see the [PostgreSQL replication and failover with Omnibus GitLab](../../postgresql/replication_and_failover.md#patroni) page. +If you are currently using `repmgr` on your Geo primary, see [these instructions](#migrating-from-repmgr-to-patroni) for migrating from `repmgr` to Patroni. + A production-ready and secure setup requires at least three Patroni instances on the primary, and a similar configuration on the secondary nodes. Be sure to use password credentials and other database best practices. @@ -549,6 +546,13 @@ patroni['standby_cluster']['primary_slot_name'] = 'geo_secondary' # or the uniqu patroni['replication_password'] = 'PLAIN_TEXT_POSTGRESQL_REPLICATION_PASSWORD' ``` +## Migrating from repmgr to Patroni + +1. Before migrating, it is recommended that there is no replication lag between the primary and secondary sites and that replication is paused. In GitLab 13.2 and later, you can pause and resume replication with `gitlab-ctl geo-replication-pause` and `gitlab-ctl geo-replication-resume` on a Geo secondary database node. +1. Follow the [instructions to migrate repmgr to Patroni](../../postgresql/replication_and_failover.md#switching-from-repmgr-to-patroni). When configuring Patroni on each primary site database node, add `patroni['replicaton_slots'] = { '<slot_name>' => 'physical' }` +to `gitlab.rb` where `<slot_name>` is the name of the replication slot for your Geo secondary. This will ensure that Patroni recognizes the replication slot as permanent and will not drop it upon restarting. +1. If database replication to the secondary was paused before migration, resume replication once Patroni is confirmed working on the primary. + ## Troubleshooting Read the [troubleshooting document](../replication/troubleshooting.md). diff --git a/doc/administration/geo/setup/external_database.md b/doc/administration/geo/setup/external_database.md index 33a7bc38b6e..1fb41fbb53a 100644 --- a/doc/administration/geo/setup/external_database.md +++ b/doc/administration/geo/setup/external_database.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -11,7 +11,7 @@ This document is relevant if you are using a PostgreSQL instance that is *not managed by Omnibus*. This includes cloud-managed instances like AWS RDS, or manually installed and configured PostgreSQL instances. -NOTE: **Note:** +NOTE: We strongly recommend running Omnibus-managed instances as they are actively developed and tested. We aim to be compatible with most external (not managed by Omnibus) databases but we do not guarantee compatibility. @@ -186,7 +186,7 @@ to grant additional roles to your tracking database user (by default, this is If you have an external database ready to be used as the tracking database, follow the instructions below to use it: -NOTE: **Note:** +NOTE: If you want to use AWS RDS as a tracking database, make sure it has access to the secondary database. Unfortunately, just assigning the same security group is not enough as outbound rules do not apply to RDS PostgreSQL databases. Therefore, you need to explicitly add an inbound diff --git a/doc/administration/geo/setup/index.md b/doc/administration/geo/setup/index.md index a4d9fcba14d..8308cbfa82e 100644 --- a/doc/administration/geo/setup/index.md +++ b/doc/administration/geo/setup/index.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -12,7 +12,7 @@ These instructions assume you have a working instance of GitLab. They guide you 1. Making your existing instance the **primary** node. 1. Adding **secondary** nodes. -CAUTION: **Caution:** +WARNING: The steps below should be followed in the order they appear. **Make sure the GitLab version is the same on all nodes.** ## Using Omnibus GitLab |
