diff options
Diffstat (limited to 'doc/administration/geo')
15 files changed, 701 insertions, 192 deletions
diff --git a/doc/administration/geo/disaster_recovery/bring_primary_back.md b/doc/administration/geo/disaster_recovery/bring_primary_back.md index d06f11bbe09..64673b9ee8d 100644 --- a/doc/administration/geo/disaster_recovery/bring_primary_back.md +++ b/doc/administration/geo/disaster_recovery/bring_primary_back.md @@ -5,27 +5,27 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Bring a demoted primary node back online **(PREMIUM SELF)** +# Bring a demoted primary site back online **(PREMIUM SELF)** -After a failover, it is possible to fail back to the demoted **primary** node to +After a failover, it is possible to fail back to the demoted **primary** site to restore your original configuration. This process consists of two steps: -1. Making the old **primary** node a **secondary** node. -1. Promoting a **secondary** node to a **primary** node. +1. Making the old **primary** site a **secondary** site. +1. Promoting a **secondary** site to a **primary** site. WARNING: -If you have any doubts about the consistency of the data on this node, we recommend setting it up from scratch. +If you have any doubts about the consistency of the data on this site, we recommend setting it up from scratch. -## Configure the former **primary** node to be a **secondary** node +## Configure the former **primary** site to be a **secondary** site -Since the former **primary** node will be out of sync with the current **primary** node, the first step is to bring the former **primary** node up to date. Note, deletion of data stored on disk like -repositories and uploads will not be replayed when bringing the former **primary** node back +Since the former **primary** site will be out of sync with the current **primary** site, the first step is to bring the former **primary** site up to date. Note, deletion of data stored on disk like +repositories and uploads will not be replayed when bringing the former **primary** site back into sync, which may result in increased disk usage. Alternatively, you can [set up a new **secondary** GitLab instance](../setup/index.md) to avoid this. -To bring the former **primary** node up to date: +To bring the former **primary** site up to date: -1. SSH into the former **primary** node that has fallen behind. +1. SSH into the former **primary** site that has fallen behind. 1. Make sure all the services are up: ```shell @@ -33,36 +33,36 @@ To bring the former **primary** node up to date: ``` NOTE: - If you [disabled the **primary** node permanently](index.md#step-2-permanently-disable-the-primary-node), + If you [disabled the **primary** site permanently](index.md#step-2-permanently-disable-the-primary-site), you need to undo those steps now. For Debian/Ubuntu you just need to run `sudo systemctl enable gitlab-runsvdir`. For CentOS 6, you need to install - the GitLab instance from scratch and set it up as a **secondary** node by + the GitLab instance from scratch and set it up as a **secondary** site by following [Setup instructions](../setup/index.md). In this case, you don't need to follow the next step. NOTE: If you [changed the DNS records](index.md#step-4-optional-updating-the-primary-domain-dns-record) - for this node during disaster recovery procedure you may need to [block - all the writes to this node](planned_failover.md#prevent-updates-to-the-primary-node) + for this site during disaster recovery procedure you may need to [block + all the writes to this site](planned_failover.md#prevent-updates-to-the-primary-node) during this procedure. -1. [Set up database replication](../setup/database.md). In this case, the **secondary** node - refers to the former **primary** node. - 1. If [PgBouncer](../../postgresql/pgbouncer.md) was enabled on the **current secondary** node - (when it was a primary node) disable it by editing `/etc/gitlab/gitlab.rb` +1. [Set up database replication](../setup/database.md). In this case, the **secondary** site + refers to the former **primary** site. + 1. If [PgBouncer](../../postgresql/pgbouncer.md) was enabled on the **current secondary** site + (when it was a primary site) disable it by editing `/etc/gitlab/gitlab.rb` and running `sudo gitlab-ctl reconfigure`. - 1. You can then set up database replication on the **secondary** node. + 1. You can then set up database replication on the **secondary** site. -If you have lost your original **primary** node, follow the -[setup instructions](../setup/index.md) to set up a new **secondary** node. +If you have lost your original **primary** site, follow the +[setup instructions](../setup/index.md) to set up a new **secondary** site. -## Promote the **secondary** node to **primary** node +## Promote the **secondary** site to **primary** site -When the initial replication is complete and the **primary** node and **secondary** node are +When the initial replication is complete and the **primary** site and **secondary** site are closely in sync, you can do a [planned failover](planned_failover.md). -## Restore the **secondary** node +## Restore the **secondary** site -If your objective is to have two nodes again, you need to bring your **secondary** -node back online as well by repeating the first step -([configure the former **primary** node to be a **secondary** node](#configure-the-former-primary-node-to-be-a-secondary-node)) -for the **secondary** node. +If your objective is to have two sites again, you need to bring your **secondary** +site back online as well by repeating the first step +([configure the former **primary** site to be a **secondary** site](#configure-the-former-primary-site-to-be-a-secondary-site)) +for the **secondary** site. diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index f6f88e9b193..bf28eb76ffd 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -16,36 +16,36 @@ For the latest updates, check the [Disaster Recovery epic for complete maturity] Multi-secondary configurations require the complete re-synchronization and re-configuration of all non-promoted secondaries and causes downtime. -## Promoting a **secondary** Geo node in single-secondary configurations +## Promoting a **secondary** Geo site in single-secondary configurations We don't currently provide an automated way to promote a Geo replica and do a failover, but you can do it manually if you have `root` access to the machine. -This process promotes a **secondary** Geo node to a **primary** node. To regain -geographic redundancy as quickly as possible, you should add a new **secondary** node +This process promotes a **secondary** Geo site to a **primary** site. To regain +geographic redundancy as quickly as possible, you should add a new **secondary** site immediately after following these instructions. ### Step 1. Allow replication to finish if possible -If the **secondary** node is still replicating data from the **primary** node, follow +If the **secondary** site is still replicating data from the **primary** site, follow [the planned failover docs](planned_failover.md) as closely as possible in order to avoid unnecessary data loss. -### Step 2. Permanently disable the **primary** node +### Step 2. Permanently disable the **primary** site WARNING: -If the **primary** node goes offline, there may be data saved on the **primary** node -that have not been replicated to the **secondary** node. This data should be treated +If the **primary** site goes offline, there may be data saved on the **primary** site +that have not been replicated to the **secondary** site. This data should be treated as lost if you proceed. -If an outage on the **primary** node happens, you should do everything possible to +If an outage on the **primary** site happens, you should do everything possible to avoid a split-brain situation where writes can occur in two different GitLab instances, complicating recovery efforts. So to prepare for the failover, we -must disable the **primary** node. +must disable the **primary** site. - If you have SSH access: - 1. SSH into the **primary** node to stop and disable GitLab: + 1. SSH into the **primary** site to stop and disable GitLab: ```shell sudo gitlab-ctl stop @@ -57,35 +57,35 @@ must disable the **primary** node. sudo systemctl disable gitlab-runsvdir ``` -- If you do not have SSH access to the **primary** node, take the machine offline and +- If you do not have SSH access to the **primary** site, take the machine offline and prevent it from rebooting by any means at your disposal. You might need to: - Reconfigure the load balancers. - Change DNS records (for example, point the primary DNS record to the - **secondary** node to stop usage of the **primary** node). + **secondary** site to stop usage of the **primary** site). - Stop the virtual servers. - Block traffic through a firewall. - - Revoke object storage permissions from the **primary** node. + - Revoke object storage permissions from the **primary** site. - Physically disconnect a machine. If you plan to [update the primary domain DNS record](#step-4-optional-updating-the-primary-domain-dns-record), you may wish to lower the TTL now to speed up propagation. -### Step 3. Promoting a **secondary** node +### Step 3. Promoting a **secondary** site WARNING: -In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the +In GitLab 13.2 and 13.3, promoting a secondary site to a primary while the secondary is paused fails. Do not pause replication before promoting a -secondary. If the node is paused, be sure to resume before promoting. +secondary. If the secondary site is paused, be sure to resume before promoting. This issue has been fixed in GitLab 13.4 and later. Note the following when promoting a secondary: -- If replication was paused on the secondary node (for example as a part of +- If replication was paused on the secondary site (for example as a part of upgrading, while you were running a version of GitLab earlier than 13.4), you - _must_ [enable the node by using the database](../replication/troubleshooting.md#message-activerecordrecordinvalid-validation-failed-enabled-geo-primary-node-cannot-be-disabled) - before proceeding. If the secondary node + _must_ [enable the site by using the database](../replication/troubleshooting.md#message-activerecordrecordinvalid-validation-failed-enabled-geo-primary-node-cannot-be-disabled) + before proceeding. If the secondary site [has been paused](../../geo/index.md#pausing-and-resuming-replication), the promotion performs a point-in-time recovery to the last known state. Data that was created on the primary while the secondary was paused is lost. @@ -99,7 +99,32 @@ Note the following when promoting a secondary: for more information, see this [troubleshooting advice](../replication/troubleshooting.md#errors-when-using---skip-preflight-checks-or---force). -#### Promoting a **secondary** node running on a single machine +#### Promoting a **secondary** site running on a single node running GitLab 14.5 and later + +1. SSH in to your **secondary** node and execute: + + - To promote the secondary node to primary: + + ```shell + sudo gitlab-ctl geo promote + ``` + + - To promote the secondary node to primary **without any further confirmation**: + + ```shell + sudo gitlab-ctl geo promote --force + ``` + +1. Verify you can connect to the newly-promoted **primary** site using the URL used + previously for the **secondary** site. +1. If successful, the **secondary** site is now promoted to the **primary** site. + +#### Promoting a **secondary** site running on a single node running GitLab 14.4 and earlier + +WARNING: +The `gitlab-ctl promote-to-primary-node` and `gitlab-ctl promoted-db` commands are +deprecated in GitLab 14.5 and later, and are scheduled to [be removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/345207). +Use `gitlab-ctl geo promote` instead. 1. SSH in to your **secondary** node and login as root: @@ -116,7 +141,7 @@ Note the following when promoting a secondary: roles ['geo_secondary_role'] ``` -1. Promote the **secondary** node to the **primary** node: +1. Promote the **secondary** site to the **primary** site: - To promote the secondary node to primary along with [preflight checks](planned_failover.md#preflight-checks): @@ -146,18 +171,57 @@ Note the following when promoting a secondary: gitlab-ctl promote-to-primary-node --force ``` -1. Verify you can connect to the newly-promoted **primary** node using the URL used - previously for the **secondary** node. -1. If successful, the **secondary** node is now promoted to the **primary** node. +1. Verify you can connect to the newly-promoted **primary** site using the URL used + previously for the **secondary** site. +1. If successful, the **secondary** site is now promoted to the **primary** site. -#### Promoting a **secondary** node with multiple servers +#### Promoting a **secondary** site with multiple nodes running GitLab 14.5 and later + +1. SSH to every Sidekiq, PostgresSQL, and Gitaly node in the **secondary** site and run one of the following commands: + + - To promote the secondary node to primary: + + ```shell + sudo gitlab-ctl geo promote + ``` + + - To promote the secondary node to primary **without any further confirmation**: + + ```shell + sudo gitlab-ctl geo promote --force + ``` + +1. SSH into each Rails node on your **secondary** site and run one of the following commands: + + - To promote the secondary node to primary: + + ```shell + sudo gitlab-ctl geo promote + ``` + + - To promote the secondary node to primary **without any further confirmation**: + + ```shell + sudo gitlab-ctl geo promote --force + ``` + +1. Verify you can connect to the newly-promoted **primary** site using the URL used + previously for the **secondary** site. +1. If successful, the **secondary** site is now promoted to the **primary** site. + +#### Promoting a **secondary** site with multiple nodes running GitLab 14.4 and earlier + +WARNING: +The `gitlab-ctl promote-to-primary-node` and `gitlab-ctl promoted-db` commands are +deprecated in GitLab 14.5 and later, and are scheduled to [be removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/345207). +Use `gitlab-ctl geo promote` instead. The `gitlab-ctl promote-to-primary-node` command cannot be used yet in -conjunction with multiple servers, as it can only -perform changes on a **secondary** with only a single machine. Instead, you must +conjunction with multiple nodes, as it can only perform changes on +a **secondary** with only a single node. Instead, you must do this manually. -1. SSH in to the database node in the **secondary** and trigger PostgreSQL to +1. SSH in to the database node in the **secondary** site and trigger PostgreSQL to promote to read-write: ```shell @@ -187,16 +251,54 @@ do this manually. 1. Verify you can connect to the newly-promoted **primary** using the URL used previously for the **secondary**. -1. If successful, the **secondary** node is now promoted to the **primary** node. +1. If successful, the **secondary** site is now promoted to the **primary** site. + +#### Promoting a **secondary** site with a Patroni standby cluster running GitLab 14.5 and later + +1. SSH to every Sidekiq, PostgresSQL, and Gitaly node in the **secondary** site and run one of the following commands: + + - To promote the secondary node to primary: + + ```shell + sudo gitlab-ctl geo promote + ``` + + - To promote the secondary node to primary **without any further confirmation**: + + ```shell + sudo gitlab-ctl geo promote --force + ``` + +1. SSH into each Rails node on your **secondary** site and run one of the following commands: -#### Promoting a **secondary** node with a Patroni standby cluster + - To promote the secondary node to primary: + + ```shell + sudo gitlab-ctl geo promote + ``` + + - To promote the secondary node to primary **without any further confirmation**: + + ```shell + sudo gitlab-ctl geo promote --force + ``` + +1. Verify you can connect to the newly-promoted **primary** site using the URL used + previously for the **secondary** site. +1. If successful, the **secondary** site is now promoted to the **primary** site. + +#### Promoting a **secondary** site with a Patroni standby cluster running GitLab 14.4 and earlier + +WARNING: +The `gitlab-ctl promote-to-primary-node` and `gitlab-ctl promoted-db` commands are +deprecated in GitLab 14.5 and later, and are scheduled to [be removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/345207). +Use `gitlab-ctl geo promote` instead. The `gitlab-ctl promote-to-primary-node` command cannot be used yet in -conjunction with a Patroni standby cluster, as it can only -perform changes on a **secondary** with only a single machine. Instead, you must -do this manually. +conjunction with a Patroni standby cluster, as it can only perform changes on +a **secondary** with only a single node. Instead, you must do this manually. -1. SSH in to the Standby Leader database node in the **secondary** and trigger PostgreSQL to +1. SSH in to the Standby Leader database node in the **secondary** site and trigger PostgreSQL to promote to read-write: ```shell @@ -230,9 +332,81 @@ do this manually. 1. Verify you can connect to the newly-promoted **primary** using the URL used previously for the **secondary**. -1. If successful, the **secondary** node is now promoted to the **primary** node. +1. If successful, the **secondary** site is now promoted to the **primary** site. + +#### Promoting a **secondary** site with an external PostgreSQL database running GitLab 14.5 and later + +The `gitlab-ctl geo promote` command can be used in conjunction with +an external PostgreSQL database, but it can only perform changes on +a **secondary** PostgreSQL database managed by Omnibus. +You must promote the replica database associated with the **secondary** +site first. + +1. Promote the replica database associated with the **secondary** site. This + sets the database to read-write. The instructions vary depending on where your database is hosted: + - [Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Promote) + - [Azure PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/howto-read-replicas-portal#stop-replication) + - [Google Cloud SQL](https://cloud.google.com/sql/docs/mysql/replication/manage-replicas#promote-replica) + - For other external PostgreSQL databases, save the following script in your + secondary node, for example `/tmp/geo_promote.sh`, and modify the connection + parameters to match your environment. Then, execute it to promote the replica: + + ```shell + #!/bin/bash -#### Promoting a **secondary** node with an external PostgreSQL database + PG_SUPERUSER=postgres + + # The path to your pg_ctl binary. You may need to adjust this path to match + # your PostgreSQL installation + PG_CTL_BINARY=/usr/lib/postgresql/10/bin/pg_ctl + + # The path to your PostgreSQL data directory. You may need to adjust this + # path to match your PostgreSQL installation. You can also run + # `SHOW data_directory;` from PostgreSQL to find your data directory + PG_DATA_DIRECTORY=/etc/postgresql/10/main + + # Promote the PostgreSQL database and allow read/write operations + sudo -u $PG_SUPERUSER $PG_CTL_BINARY -D $PG_DATA_DIRECTORY promote + ``` + +1. SSH to every Sidekiq, PostgresSQL, and Gitaly node in the **secondary** site and run one of the following commands: + + - To promote the secondary node to primary: + + ```shell + sudo gitlab-ctl geo promote + ``` + + - To promote the secondary node to primary **without any further confirmation**: + + ```shell + sudo gitlab-ctl geo promote --force + ``` + +1. SSH into each Rails node on your **secondary** site and run one of the following commands: + + - To promote the secondary node to primary: + + ```shell + sudo gitlab-ctl geo promote + ``` + + - To promote the secondary node to primary **without any further confirmation**: + + ```shell + sudo gitlab-ctl geo promote --force + ``` + +1. Verify you can connect to the newly-promoted **primary** site using the URL used + previously for the **secondary** site. +1. If successful, the **secondary** site is now promoted to the **primary** site. + +#### Promoting a **secondary** site with an external PostgreSQL database running GitLab 14.4 and earlier + +WARNING: +The `gitlab-ctl promote-to-primary-node` and `gitlab-ctl promoted-db` commands are +deprecated in GitLab 14.5 and later, and are scheduled to [be removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/345207). +Use `gitlab-ctl geo promote` instead. The `gitlab-ctl promote-to-primary-node` command cannot be used in conjunction with an external PostgreSQL database, as it can only perform changes on a **secondary** @@ -287,23 +461,23 @@ required: 1. Verify you can connect to the newly-promoted **primary** using the URL used previously for the **secondary**. -1. If successful, the **secondary** node is now promoted to the **primary** node. +1. If successful, the **secondary** site is now promoted to the **primary** site. ### Step 4. (Optional) Updating the primary domain DNS record -Updating the DNS records for the primary domain to point to the **secondary** node +Updating the DNS records for the primary domain to point to the **secondary** site to prevent the need to update all references to the primary domain to the secondary domain, like changing Git remotes and API URLs. -1. SSH into the **secondary** node and login as root: +1. SSH into the **secondary** site and login as root: ```shell sudo -i ``` 1. Update the primary domain's DNS record. After updating the primary domain's - DNS records to point to the **secondary** node, edit `/etc/gitlab/gitlab.rb` on the - **secondary** node to reflect the new URL: + DNS records to point to the **secondary** site, edit `/etc/gitlab/gitlab.rb` on the + **secondary** site to reflect the new URL: ```ruby # Change the existing external_url configuration @@ -314,13 +488,13 @@ secondary domain, like changing Git remotes and API URLs. Changing `external_url` does not prevent access via the old secondary URL, as long as the secondary DNS records are still intact. -1. Reconfigure the **secondary** node for the change to take effect: +1. Reconfigure the **secondary** site for the change to take effect: ```shell gitlab-ctl reconfigure ``` -1. Execute the command below to update the newly promoted **primary** node URL: +1. Execute the command below to update the newly promoted **primary** site URL: ```shell gitlab-rake geo:update_primary_node_url @@ -335,14 +509,14 @@ secondary domain, like changing Git remotes and API URLs. To determine if you need to do this, search for the `gitlab_rails["geo_node_name"]` setting in your `/etc/gitlab/gitlab.rb` file. If it is commented out with `#` or not found at all, then you - need to update the **primary** node's name in the database. You can search for it + need to update the **primary** site's name in the database. You can search for it like so: ```shell grep "geo_node_name" /etc/gitlab/gitlab.rb ``` - To update the **primary** node's name in the database: + To update the **primary** site's name in the database: ```shell gitlab-rails runner 'Gitlab::Geo.primary_node.update!(name: GeoNode.current_node_name)' @@ -352,12 +526,12 @@ secondary domain, like changing Git remotes and API URLs. If you updated the DNS records for the primary domain, these changes may not have yet propagated depending on the previous DNS records TTL. -### Step 5. (Optional) Add **secondary** Geo node to a promoted **primary** node +### Step 5. (Optional) Add **secondary** Geo site to a promoted **primary** site -Promoting a **secondary** node to **primary** node using the process above does not enable -Geo on the new **primary** node. +Promoting a **secondary** site to **primary** site using the process above does not enable +Geo on the new **primary** site. -To bring a new **secondary** node online, follow the [Geo setup instructions](../index.md#setup-instructions). +To bring a new **secondary** site online, follow the [Geo setup instructions](../index.md#setup-instructions). ### Step 6. (Optional) Removing the secondary's tracking database @@ -376,13 +550,13 @@ for the changes to take effect. ## Promoting secondary Geo replica in multi-secondary configurations -If you have more than one **secondary** node and you need to promote one of them, we suggest you follow -[Promoting a **secondary** Geo node in single-secondary configurations](#promoting-a-secondary-geo-node-in-single-secondary-configurations) +If you have more than one **secondary** site and you need to promote one of them, we suggest you follow +[Promoting a **secondary** Geo site in single-secondary configurations](#promoting-a-secondary-geo-site-in-single-secondary-configurations) and after that you also need two extra steps. -### Step 1. Prepare the new **primary** node to serve one or more **secondary** nodes +### Step 1. Prepare the new **primary** site to serve one or more **secondary** sites -1. SSH into the new **primary** node and login as root: +1. SSH into the new **primary** site and login as root: ```shell sudo -i @@ -442,13 +616,13 @@ and after that you also need two extra steps. ### Step 2. Initiate the replication process -Now we need to make each **secondary** node listen to changes on the new **primary** node. To do that you need +Now we need to make each **secondary** site listen to changes on the new **primary** site. To do that you need to [initiate the replication process](../setup/database.md#step-3-initiate-the-replication-process) again but this time -for another **primary** node. All the old replication settings are overwritten. +for another **primary** site. All the old replication settings are overwritten. ## Promoting a secondary Geo cluster in GitLab Cloud Native Helm Charts -When updating a Cloud Native Geo deployment, the process for updating any node that is external to the secondary Kubernetes cluster does not differ from the non Cloud Native approach. As such, you can always defer to [Promoting a secondary Geo node in single-secondary configurations](#promoting-a-secondary-geo-node-in-single-secondary-configurations) for more information. +When updating a Cloud Native Geo deployment, the process for updating any node that is external to the secondary Kubernetes cluster does not differ from the non Cloud Native approach. As such, you can always defer to [Promoting a secondary Geo site in single-secondary configurations](#promoting-a-secondary-geo-site-in-single-secondary-configurations) for more information. The following sections assume you are using the `gitlab` namespace. If you used a different namespace when setting up your cluster, you should also replace `--namespace gitlab` with your namespace. @@ -489,13 +663,45 @@ must disable the **primary** site: - Revoke object storage permissions from the **primary** site. - Physically disconnect a machine. -### Step 2. Promote all **secondary** nodes external to the cluster +### Step 2. Promote all **secondary** sites external to the cluster WARNING: If the secondary site [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs a point-in-time recovery to the last known state. Data that was created on the primary while the secondary was paused is lost. +If you are running GitLab 14.5 and later: + +1. SSH to every Sidekiq, PostgresSQL, and Gitaly node in the **secondary** site and run one of the following commands: + + - To promote the secondary node to primary: + + ```shell + sudo gitlab-ctl geo promote + ``` + + - To promote the secondary node to primary **without any further confirmation**: + + ```shell + sudo gitlab-ctl geo promote --force + ``` + +1. SSH into each Rails node on your **secondary** site and run one of the following commands: + + - To promote the secondary node to primary: + + ```shell + sudo gitlab-ctl geo promote + ``` + + - To promote the secondary node to primary **without any further confirmation**: + + ```shell + sudo gitlab-ctl geo promote --force + ``` + +If you are running GitLab 14.4 and earlier: + 1. SSH in to the database node in the **secondary** and trigger PostgreSQL to promote to read-write: @@ -522,8 +728,6 @@ Data that was created on the primary while the secondary was paused is lost. After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) on the database node. -### Step 3. Promote the **secondary** cluster - 1. Find the task runner pod: ```shell @@ -536,6 +740,8 @@ Data that was created on the primary while the secondary was paused is lost. kubectl --namespace gitlab exec -ti gitlab-geo-task-runner-XXX -- gitlab-rake geo:set_secondary_as_primary ``` +### Step 3. Promote the **secondary** cluster + 1. Update the existing cluster configuration. You can retrieve the existing configuration with Helm: diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index 6312ed669ae..939b4aec968 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -204,4 +204,4 @@ in the loss of any data uploaded to the new **primary** in the meantime. Don't forget to remove the broadcast message after the failover is complete. -Finally, you can bring the [old site back as a secondary](bring_primary_back.md#configure-the-former-primary-node-to-be-a-secondary-node). +Finally, you can bring the [old site back as a secondary](bring_primary_back.md#configure-the-former-primary-site-to-be-a-secondary-site). diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md index 3eb7bc2a8e0..3c7af309f78 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md @@ -66,13 +66,13 @@ promote a Geo replica and perform a failover. NOTE: GitLab 13.9 through GitLab 14.3 are affected by a bug in which the Geo secondary site statuses will appear to stop updating and become unhealthy. For more information, see [Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](../../replication/troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). -On the **secondary** node: +On the **secondary** site: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Geo > Nodes** to see its status. Replicated objects (shown in green) should be close to 100%, and there should be no failures (shown in red). If a large proportion of - objects aren't yet replicated (shown in gray), consider giving the node more + objects aren't yet replicated (shown in gray), consider giving the site more time to complete.  @@ -85,20 +85,20 @@ You can use the [Geo status API](../../../../api/geo_nodes.md#retrieve-project-sync-or-verification-failures-that-occurred-on-the-current-node) to review failed objects and the reasons for failure. A common cause of replication failures is the data being missing on the -**primary** node - you can resolve these failures by restoring the data from backup, +**primary** site - you can resolve these failures by restoring the data from backup, or removing references to the missing data. The maintenance window won't end until Geo replication and verification is completely finished. To keep the window as short as possible, you should ensure these processes are close to 100% as possible during active use. -If the **secondary** node is still replicating data from the **primary** node, +If the **secondary** site is still replicating data from the **primary** site, follow these steps to avoid unnecessary data loss: 1. Until a [read-only mode](https://gitlab.com/gitlab-org/gitlab/-/issues/14609) is implemented, updates must be prevented from happening manually to the - **primary**. Your **secondary** node still needs read-only - access to the **primary** node during the maintenance window: + **primary**. Your **secondary** site still needs read-only + access to the **primary** site during the maintenance window: 1. At the scheduled time, using your cloud provider or your node's firewall, block all HTTP, HTTPS and SSH traffic to/from the **primary** node, **except** for your IP and @@ -121,18 +121,18 @@ follow these steps to avoid unnecessary data loss: ``` From this point, users are unable to view their data or make changes on the - **primary** node. They are also unable to log in to the **secondary** node. + **primary** site. They are also unable to log in to the **secondary** site. However, existing sessions need to work for the remainder of the maintenance period, and so public data is accessible throughout. - 1. Verify the **primary** node is blocked to HTTP traffic by visiting it in browser via + 1. Verify the **primary** site is blocked to HTTP traffic by visiting it in browser via another IP. The server should refuse connection. - 1. Verify the **primary** node is blocked to Git over SSH traffic by attempting to pull an + 1. Verify the **primary** site is blocked to Git over SSH traffic by attempting to pull an existing Git repository with an SSH remote URL. The server should refuse connection. - 1. On the **primary** node: + 1. On the **primary** site: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dhasboard, select **Cron**. @@ -150,7 +150,7 @@ follow these steps to avoid unnecessary data loss: 1. If you are manually replicating any [data not managed by Geo](../../replication/datatypes.md#limitations-on-replicationverification), trigger the final replication process now. - 1. On the **primary** node: + 1. On the **primary** site: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all queues except @@ -165,7 +165,7 @@ follow these steps to avoid unnecessary data loss: - Database replication lag is 0ms. - The Geo log cursor is up to date (0 events behind). - 1. On the **secondary** node: + 1. On the **secondary** site: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all the `geo` @@ -173,14 +173,14 @@ follow these steps to avoid unnecessary data loss: 1. [Run an integrity check](../../../raketasks/check.md) to verify the integrity of CI artifacts, LFS objects, and uploads in file storage. - At this point, your **secondary** node contains an up-to-date copy of everything the - **primary** node has, meaning nothing is lost when you fail over. + At this point, your **secondary** site contains an up-to-date copy of everything the + **primary** site has, meaning nothing is lost when you fail over. -1. In this final step, you need to permanently disable the **primary** node. +1. In this final step, you need to permanently disable the **primary** site. WARNING: - When the **primary** node goes offline, there may be data saved on the **primary** node - that has not been replicated to the **secondary** node. This data should be treated + When the **primary** site goes offline, there may be data saved on the **primary** site + that has not been replicated to the **secondary** site. This data should be treated as lost if you proceed. NOTE: @@ -189,9 +189,9 @@ follow these steps to avoid unnecessary data loss: When performing a failover, we want to avoid a split-brain situation where writes can occur in two different GitLab instances. So to prepare for the - failover, you must disable the **primary** node: + failover, you must disable the **primary** site: - - If you have SSH access to the **primary** node, stop and disable GitLab: + - If you have SSH access to the **primary** site, stop and disable GitLab: ```shell sudo gitlab-ctl stop @@ -214,19 +214,58 @@ follow these steps to avoid unnecessary data loss: from starting if the machine reboots as `root` with `initctl stop gitlab-runsvvdir && echo 'manual' > /etc/init/gitlab-runsvdir.override && initctl reload-configuration`. - - If you do not have SSH access to the **primary** node, take the machine offline and + - If you do not have SSH access to the **primary** site, take the machine offline and prevent it from rebooting. Since there are many ways you may prefer to accomplish this, we avoid a single recommendation. You may need to: - Reconfigure the load balancers. - Change DNS records (for example, point the **primary** DNS record to the - **secondary** node to stop using the **primary** node). + **secondary** site to stop using the **primary** site). - Stop the virtual servers. - Block traffic through a firewall. - - Revoke object storage permissions from the **primary** node. + - Revoke object storage permissions from the **primary** site. - Physically disconnect a machine. -### Promoting the **secondary** node +### Promoting the **secondary** site running GitLab 14.5 and later + +1. SSH to every Sidekiq, PostgresSQL, and Gitaly node in the **secondary** site and run one of the following commands: + + - To promote the secondary node to primary: + + ```shell + sudo gitlab-ctl geo promote + ``` + + - To promote the secondary node to primary **without any further confirmation**: + + ```shell + sudo gitlab-ctl geo promote --force + ``` + +1. SSH into each Rails node on your **secondary** site and run one of the following commands: + + - To promote the secondary node to primary: + + ```shell + sudo gitlab-ctl geo promote + ``` + + - To promote the secondary node to primary **without any further confirmation**: + + ```shell + sudo gitlab-ctl geo promote --force + ``` + +1. Verify you can connect to the newly promoted **primary** site using the URL used + previously for the **secondary** site. +1. If successful, the **secondary** site is now promoted to the **primary** site. + +### Promoting the **secondary** site running GitLab 14.4 and earlier + +WARNING: +The `gitlab-ctl promote-to-primary-node` and `gitlab-ctl promoted-db` commands are +deprecated in GitLab 14.5 and later, and are scheduled to [be removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/345207). +Use `gitlab-ctl geo promote` instead. NOTE: A new **secondary** should not be added at this time. If you want to add a new @@ -243,13 +282,13 @@ perform changes on a **secondary** with only a single machine. Instead, you must do this manually. WARNING: -In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the +In GitLab 13.2 and 13.3, promoting a secondary site to a primary while the secondary is paused fails. Do not pause replication before promoting a -secondary. If the node is paused, be sure to resume before promoting. This +secondary. If the site is paused, be sure to resume before promoting. This issue has been fixed in GitLab 13.4 and later. WARNING: - If the secondary node [has been paused](../../../geo/index.md#pausing-and-resuming-replication), this performs +If the secondary site [has been paused](../../../geo/index.md#pausing-and-resuming-replication), this performs a point-in-time recovery to the last known state. Data that was created on the primary while the secondary was paused is lost. @@ -291,6 +330,6 @@ Data that was created on the primary while the secondary was paused is lost. ### Next steps To regain geographic redundancy as quickly as possible, you should -[add a new **secondary** node](../../setup/index.md). To +[add a new **secondary** site](../../setup/index.md). To do that, you can re-add the old **primary** as a new secondary and bring it back online. diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md index d4782144df8..8a4f2ed4306 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md @@ -54,10 +54,10 @@ promote a Geo replica and perform a failover. NOTE: GitLab 13.9 through GitLab 14.3 are affected by a bug in which the Geo secondary site statuses will appear to stop updating and become unhealthy. For more information, see [Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](../../replication/troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). -On the **secondary** node, navigate to the **Admin Area > Geo** dashboard to +On the **secondary** site, navigate to the **Admin Area > Geo** dashboard to review its status. Replicated objects (shown in green) should be close to 100%, and there should be no failures (shown in red). If a large proportion of -objects aren't yet replicated (shown in gray), consider giving the node more +objects aren't yet replicated (shown in gray), consider giving the site more time to complete.  @@ -70,20 +70,20 @@ You can use the [Geo status API](../../../../api/geo_nodes.md#retrieve-project-sync-or-verification-failures-that-occurred-on-the-current-node) to review failed objects and the reasons for failure. A common cause of replication failures is the data being missing on the -**primary** node - you can resolve these failures by restoring the data from backup, +**primary** site - you can resolve these failures by restoring the data from backup, or removing references to the missing data. The maintenance window won't end until Geo replication and verification is completely finished. To keep the window as short as possible, you should ensure these processes are close to 100% as possible during active use. -If the **secondary** node is still replicating data from the **primary** node, +If the **secondary** site is still replicating data from the **primary** site, follow these steps to avoid unnecessary data loss: 1. Until a [read-only mode](https://gitlab.com/gitlab-org/gitlab/-/issues/14609) is implemented, updates must be prevented from happening manually to the - **primary**. Your **secondary** node still needs read-only - access to the **primary** node during the maintenance window: + **primary**. Your **secondary** site still needs read-only + access to the **primary** site during the maintenance window: 1. At the scheduled time, using your cloud provider or your node's firewall, block all HTTP, HTTPS and SSH traffic to/from the **primary** node, **except** for your IP and @@ -106,18 +106,18 @@ follow these steps to avoid unnecessary data loss: ``` From this point, users are unable to view their data or make changes on the - **primary** node. They are also unable to log in to the **secondary** node. + **primary** site. They are also unable to log in to the **secondary** site. However, existing sessions need to work for the remainder of the maintenance period, and so public data is accessible throughout. - 1. Verify the **primary** node is blocked to HTTP traffic by visiting it in browser via + 1. Verify the **primary** site is blocked to HTTP traffic by visiting it in browser via another IP. The server should refuse connection. - 1. Verify the **primary** node is blocked to Git over SSH traffic by attempting to pull an + 1. Verify the **primary** site is blocked to Git over SSH traffic by attempting to pull an existing Git repository with an SSH remote URL. The server should refuse connection. - 1. On the **primary** node: + 1. On the **primary** site: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dhasboard, select **Cron**. @@ -135,7 +135,7 @@ follow these steps to avoid unnecessary data loss: 1. If you are manually replicating any [data not managed by Geo](../../replication/datatypes.md#limitations-on-replicationverification), trigger the final replication process now. - 1. On the **primary** node: + 1. On the **primary** site: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all queues except @@ -143,14 +143,14 @@ follow these steps to avoid unnecessary data loss: These queues contain work that has been submitted by your users; failing over before it is completed, causes the work to be lost. 1. On the left sidebar, select **Geo > Nodes** and wait for the - following conditions to be true of the **secondary** node you are failing over to: + following conditions to be true of the **secondary** site you are failing over to: - All replication meters reach 100% replicated, 0% failures. - All verification meters reach 100% verified, 0% failures. - Database replication lag is 0ms. - The Geo log cursor is up to date (0 events behind). - 1. On the **secondary** node: + 1. On the **secondary** site: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all the `geo` @@ -158,14 +158,14 @@ follow these steps to avoid unnecessary data loss: 1. [Run an integrity check](../../../raketasks/check.md) to verify the integrity of CI artifacts, LFS objects, and uploads in file storage. - At this point, your **secondary** node contains an up-to-date copy of everything the - **primary** node has, meaning nothing is lost when you fail over. + At this point, your **secondary** site contains an up-to-date copy of everything the + **primary** site has, meaning nothing is lost when you fail over. -1. In this final step, you need to permanently disable the **primary** node. +1. In this final step, you need to permanently disable the **primary** site. WARNING: - When the **primary** node goes offline, there may be data saved on the **primary** node - that has not been replicated to the **secondary** node. This data should be treated + When the **primary** site goes offline, there may be data saved on the **primary** site + that has not been replicated to the **secondary** site. This data should be treated as lost if you proceed. NOTE: @@ -174,9 +174,9 @@ follow these steps to avoid unnecessary data loss: When performing a failover, we want to avoid a split-brain situation where writes can occur in two different GitLab instances. So to prepare for the - failover, you must disable the **primary** node: + failover, you must disable the **primary** site: - - If you have SSH access to the **primary** node, stop and disable GitLab: + - If you have SSH access to the **primary** site, stop and disable GitLab: ```shell sudo gitlab-ctl stop @@ -199,19 +199,19 @@ follow these steps to avoid unnecessary data loss: from starting if the machine reboots as `root` with `initctl stop gitlab-runsvvdir && echo 'manual' > /etc/init/gitlab-runsvdir.override && initctl reload-configuration`. - - If you do not have SSH access to the **primary** node, take the machine offline and + - If you do not have SSH access to the **primary** site, take the machine offline and prevent it from rebooting. Since there are many ways you may prefer to accomplish this, we avoid a single recommendation. You may need to: - Reconfigure the load balancers. - Change DNS records (for example, point the **primary** DNS record to the - **secondary** node to stop using the **primary** node). + **secondary** site to stop using the **primary** site). - Stop the virtual servers. - Block traffic through a firewall. - - Revoke object storage permissions from the **primary** node. + - Revoke object storage permissions from the **primary** site. - Physically disconnect a machine. -### Promoting the **secondary** node +### Promoting the **secondary** site Note the following when promoting a secondary: @@ -222,9 +222,35 @@ Note the following when promoting a secondary: error during this process, read [the troubleshooting advice](../../replication/troubleshooting.md#fixing-errors-during-a-failover-or-when-promoting-a-secondary-to-a-primary-node). -To promote the secondary node: +To promote the secondary site running GitLab 14.5 and later: -1. SSH in to your **secondary** node and login as root: +1. SSH in to your **secondary** node and run one of the following commands: + + - To promote the secondary node to primary: + + ```shell + sudo gitlab-ctl geo promote + ``` + + - To promote the secondary node to primary **without any further confirmation**: + + ```shell + sudo gitlab-ctl geo promote --force + ``` + +1. Verify you can connect to the newly promoted **primary** site using the URL used + previously for the **secondary** site. + + If successful, the **secondary** site is now promoted to the **primary** site. + +To promote the secondary site running GitLab 14.4 and earlier: + +WARNING: +The `gitlab-ctl promote-to-primary-node` and `gitlab-ctl promoted-db` commands are +deprecated in GitLab 14.5 and later, and are scheduled to [be removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/345207). +Use `gitlab-ctl geo promote` instead. + +1. SSH in to your **secondary** site and login as root: ```shell sudo -i @@ -275,20 +301,20 @@ To promote the secondary node: gitlab-ctl promote-to-primary-node --skip-preflight-check ``` - You can also promote the secondary node to primary **without any further confirmation**, even when preflight checks fail: + You can also promote the secondary site to primary **without any further confirmation**, even when preflight checks fail: ```shell sudo gitlab-ctl promote-to-primary-node --force ``` -1. Verify you can connect to the newly promoted **primary** node using the URL used - previously for the **secondary** node. +1. Verify you can connect to the newly promoted **primary** site using the URL used + previously for the **secondary** site. - If successful, the **secondary** node has now been promoted to the **primary** node. + If successful, the **secondary** site is now promoted to the **primary** site. ### Next steps To regain geographic redundancy as quickly as possible, you should -[add a new **secondary** node](../../setup/index.md). To +[add a new **secondary** site](../../setup/index.md). To do that, you can re-add the old **primary** as a new secondary and bring it back online. diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index 48091967189..30d8d765dc5 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -129,25 +129,38 @@ and we recommend you use: ### Firewall rules -The following table lists basic ports that must be open between the **primary** and **secondary** sites for Geo. +The following table lists basic ports that must be open between the **primary** and **secondary** sites for Geo. To simplify failovers, we recommend opening ports in both directions. -| **Primary** site | **Secondary** site | Protocol | -|:-----------------|:-------------------|:-------------| -| 80 | 80 | HTTP | -| 443 | 443 | TCP or HTTPS | -| 22 | 22 | TCP | -| 5432 | | PostgreSQL | +| Source site | Source port | Destination site | Destination port | Protocol | +|-------------|-------------|------------------|------------------|-------------| +| Primary | Any | Secondary | 80 | TCP (HTTP) | +| Primary | Any | Secondary | 443 | TCP (HTTPS) | +| Secondary | Any | Primary | 80 | TCP (HTTP) | +| Secondary | Any | Primary | 443 | TCP (HTTPS) | +| Secondary | Any | Primary | 5432 | TCP | See the full list of ports used by GitLab in [Package defaults](../package_information/defaults.md) NOTE: -[Web terminal](../../ci/environments/index.md#web-terminals) support requires your load balancer to correctly handle WebSocket connections. +[Web terminal](../../ci/environments/index.md#web-terminals-deprecated) support requires your load balancer to correctly handle WebSocket connections. When using HTTP or HTTPS proxying, your load balancer must be configured to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the [web terminal](../integration/terminal.md) integration guide for more details. NOTE: When using HTTPS protocol for port 443, you need to add an SSL certificate to the load balancers. If you wish to terminate SSL at the GitLab application server instead, use TCP protocol. +#### Internal URL + +HTTP requests from any Geo secondary site to the primary Geo site use the Internal URL of the primary +Geo site. If this is not explicitly defined in the primary Geo site settings in the Admin Area, the +public URL of the primary site will be used. + +To update the internal URL of the primary Geo site: + +1. On the top bar, go to **Menu > Admin > Geo > Sites**. +1. Select **Edit** on the primary site. +1. Change the **Internal URL**, then select **Save changes**. + ### LDAP We recommend that if you use LDAP on your **primary** site, you also set up secondary LDAP servers on each **secondary** site. Otherwise, users are unable to perform Git operations over HTTP(s) on the **secondary** site using HTTP Basic Authentication. However, Git via SSH and personal access tokens still works. @@ -258,6 +271,10 @@ For information on using Geo in disaster recovery situations to mitigate data-lo For more information on how to replicate the Container Registry, see [Docker Registry for a **secondary** site](replication/docker_registry.md). +### Geo secondary proxy + +For more information on using Geo proxying on secondary nodes, see [Geo proxying for secondary sites](secondary_proxy/index.md). + ### Security Review For more information on Geo security, see [Geo security review](replication/security_review.md). diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index e8e87f92481..c98436157fc 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -183,14 +183,13 @@ successfully, you must replicate their data using some other means. |[Application data in PostgreSQL](../../postgresql/index.md) | **Yes** (10.2) | **Yes** (10.2) | No | | |[Project repository](../../../user/project/repository/) | **Yes** (10.2) | **Yes** (10.7) | No | | |[Project wiki repository](../../../user/project/wiki/) | **Yes** (10.2) | **Yes** (10.7) | No | | -|[Group wiki repository](../../../user/project/wiki/index.md#group-wikis) | [**Yes** (13.10)](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | No | No | Behind feature flag `geo_group_wiki_repository_replication`, enabled by default. | +|[Group wiki repository](../../../user/project/wiki/group.md) | [**Yes** (13.10)](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | No | No | Behind feature flag `geo_group_wiki_repository_replication`, enabled by default. | |[Uploads](../../uploads.md) | **Yes** (10.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | No | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. | |[LFS objects](../../lfs/index.md) | **Yes** (10.2) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8922) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696).<br /><br />Behind feature flag `geo_lfs_object_replication`, enabled by default. | |[Personal snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | No | | |[Project snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | No | | -|[CI job artifacts (other than Job Logs)](../../../ci/pipelines/job_artifacts.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. | +|[CI job artifacts](../../../ci/pipelines/job_artifacts.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. Job logs also verified on transfer. | |[CI Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/pipeline_artifact.rb) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | Via Object Storage provider if supported. Native Geo support (Beta). | Persists additional artifacts after a pipeline completes | -|[Job logs](../../job_logs.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. | |[Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | No | Disabled by default. See [instructions](docker_registry.md) to enable. | |[Content in object storage (beta)](object_storage.md) | **Yes** (12.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) | No | | |[Infrastructure Registry](../../../user/packages/infrastructure_registry/index.md) | **Yes** (14.0) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (14.0) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | diff --git a/doc/administration/geo/replication/disable_geo.md b/doc/administration/geo/replication/disable_geo.md index ae2597ad649..02a65f0b8e1 100644 --- a/doc/administration/geo/replication/disable_geo.md +++ b/doc/administration/geo/replication/disable_geo.md @@ -28,7 +28,7 @@ To disable Geo, you need to first remove all your secondary Geo sites, which mea anymore on these sites. You can follow our docs to [remove your secondary Geo sites](remove_geo_site.md). If the current site that you want to keep using is a secondary site, you need to first promote it to primary. -You can use our steps on [how to promote a secondary site](../disaster_recovery/#step-3-promoting-a-secondary-node) +You can use our steps on [how to promote a secondary site](../disaster_recovery/#step-3-promoting-a-secondary-site) to do that. ## Remove the primary site from the UI diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index b7370d32056..432d042608c 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -455,7 +455,7 @@ To solve this: 1. Back up [the `.git` folder](../../repository_storage_types.md#translate-hashed-storage-paths). -1. Optional: [Spot-check](../../troubleshooting/log_parsing.md#find-all-projects-affected-by-a-fatal-git-problem)) +1. Optional: [Spot-check](../../troubleshooting/log_parsing.md#find-all-projects-affected-by-a-fatal-git-problem) a few of those IDs whether they indeed correspond to a project with known Geo replication failures. Use `fatal: 'geo'` as the `grep` term and the following API call: @@ -683,7 +683,7 @@ when promoting a secondary to a primary node with strategies to resolve them. ### Message: ActiveRecord::RecordInvalid: Validation failed: Name has already been taken -When [promoting a **secondary** node](../disaster_recovery/index.md#step-3-promoting-a-secondary-node), +When [promoting a **secondary** site](../disaster_recovery/index.md#step-3-promoting-a-secondary-site), you might encounter the following error: ```plaintext @@ -723,21 +723,35 @@ If you disabled a secondary node, either with the [replication pause task](../in (13.2) or by using the user interface (13.1 and earlier), you must first re-enable the node before you can continue. This is fixed in 13.4. -Run the following command, replacing `https://<secondary url>/` with the URL -for your secondary server, using either `http` or `https`, and ensuring that you -end the URL with a slash (`/`): +This can be fixed in the database. -```shell -sudo gitlab-rails dbconsole +1. Start a database console: -UPDATE geo_nodes SET enabled = true WHERE url = 'https://<secondary url>/' AND enabled = false;" -``` + In [GitLab 14.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/341210): + + ```shell + sudo gitlab-rails dbconsole --database main + ``` + + In GitLab 14.1 and earlier: + + ```shell + sudo gitlab-rails dbconsole + ``` + +1. Run the following command, replacing `https://<secondary url>/` with the URL + for your secondary server. You can use either `http` or `https`, but ensure that you + end the URL with a slash (`/`): + + ```sql + UPDATE geo_nodes SET enabled = true WHERE url = 'https://<secondary url>/' AND enabled = false;" + ``` -This should update 1 row. + This should update 1 row. ### Message: ``NoMethodError: undefined method `secondary?' for nil:NilClass`` -When [promoting a **secondary** node](../disaster_recovery/index.md#step-3-promoting-a-secondary-node), +When [promoting a **secondary** site](../disaster_recovery/index.md#step-3-promoting-a-secondary-site), you might encounter the following error: ```plaintext @@ -753,13 +767,13 @@ Tasks: TOP => geo:set_secondary_as_primary (See full trace by running task with --trace) ``` -This command is intended to be executed on a secondary node only, and this error -is displayed if you attempt to run this command on a primary node. +This command is intended to be executed on a secondary site only, and this error +is displayed if you attempt to run this command on a primary site. ### Message: `sudo: gitlab-pg-ctl: command not found` When -[promoting a **secondary** node with multiple servers](../disaster_recovery/index.md#promoting-a-secondary-node-with-multiple-servers), +[promoting a **secondary** site with multiple nodes](../disaster_recovery/index.md#promoting-a-secondary-site-with-multiple-nodes-running-gitlab-144-and-earlier), you need to run the `gitlab-pg-ctl` command to promote the PostgreSQL read-replica database. diff --git a/doc/administration/geo/secondary_proxy/img/single_url_add_traffic_policy_endpoints.png b/doc/administration/geo/secondary_proxy/img/single_url_add_traffic_policy_endpoints.png Binary files differnew file mode 100644 index 00000000000..d1c4afceb04 --- /dev/null +++ b/doc/administration/geo/secondary_proxy/img/single_url_add_traffic_policy_endpoints.png diff --git a/doc/administration/geo/secondary_proxy/img/single_url_create_policy_records_with_traffic_policy.png b/doc/administration/geo/secondary_proxy/img/single_url_create_policy_records_with_traffic_policy.png Binary files differnew file mode 100644 index 00000000000..3bad391f220 --- /dev/null +++ b/doc/administration/geo/secondary_proxy/img/single_url_create_policy_records_with_traffic_policy.png diff --git a/doc/administration/geo/secondary_proxy/index.md b/doc/administration/geo/secondary_proxy/index.md new file mode 100644 index 00000000000..2b8c0d1e6fa --- /dev/null +++ b/doc/administration/geo/secondary_proxy/index.md @@ -0,0 +1,127 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: howto +--- + +# Geo proxying for secondary sites **(PREMIUM SELF)** + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5914) in GitLab 14.4 [with a flag](../../feature_flags.md) named `geo_secondary_proxy`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. See below to [Set up a unified URL for Geo sites](#set-up-a-unified-url-for-geo-sites). +The feature is not ready for production use. + +Use Geo proxying to: + +- Have secondary sites serve read-write traffic by proxying to the primary site. +- Selectively accelerate replicated data types by directing read-only operations to the local site instead. + +When enabled, users of the secondary site can use the WebUI as if they were accessing the +primary site's UI. This significantly improves the overall user experience of secondary sites. + +With secondary proxying, web requests to secondary Geo sites are +proxied directly to the primary, and appear to act as a read-write site. + +Proxying is done by the [`gitlab-workhorse`](https://gitlab.com/gitlab-org/gitlab-workhorse) component. +Traffic usually sent to the Rails application on the Geo secondary site is proxied +to the [internal URL](../index.md#internal-url) of the primary Geo site instead. + +Use secondary proxying for use-cases including: + +- Having all Geo sites behind a single URL. +- Geographically load-balancing traffic without worrying about write access. + +## Set up a unified URL for Geo sites + +Secondary sites can transparently serve read-write traffic. You can +use a single external URL so that requests can hit either the primary Geo site +or any secondary Geo sites that use Geo proxying. + +### Configure an external URL to send traffic to both sites + +Follow the [Location-aware public URL](location_aware_external_url.md) steps to create +a single URL used by all Geo sites, including the primary. + +### Update the Geo sites to use the same external URL + +1. On your Geo sites, SSH **into each node running Rails (Puma, Sidekiq, Log-Cursor) + and change the `external_url` to that of the single URL: + + ```shell + sudo editor /etc/gitlab/gitlab.rb + ``` + +1. Reconfigure the updated nodes for the change to take effect if the URL was different than the one already set: + + ```shell + gitlab-ctl reconfigure + ``` + +1. To match the new external URL set on the secondary Geo sites, the primary database + needs to reflect this change. + + In the Geo administration page of the **primary** site, edit each Geo secondary that + is using the secondary proxying and set the `URL` field to the single URL. + Make sure the primary site is also using this URL. + +### Enable secondary proxying + +1. SSH into each application node (serving user traffic directly) on your secondary Geo site + and add the following environment variable: + + ```shell + sudo editor /etc/gitlab/gitlab.rb + ``` + + ```ruby + gitlab_workhorse['env'] = { + "GEO_SECONDARY_PROXY" => "1" + } + ``` + +1. Reconfigure the updated nodes for the change to take effect: + + ```shell + gitlab-ctl reconfigure + ``` + +1. SSH into one node running Rails on your primary Geo site and enable the Geo secondary proxy feature flag: + + ```shell + sudo gitlab-rails runner "Feature.enable(:geo_secondary_proxy)" + ``` + +## Enable Geo proxying with Separate URLs + +The ability to use proxying with separate URLs is still in development. You can follow the +["Geo secondary proxying with separate URLs" epic](https://gitlab.com/groups/gitlab-org/-/epics/6865) +for progress. + +## Features accelerated by secondary Geo sites + +Most HTTP traffic sent to a secondary Geo site can be proxied to the primary Geo site. With this architecture, +secondary Geo sites are able to support write requests. Certain **read** requests are handled locally by secondary +sites for improved latency and bandwidth nearby. All write requests are proxied to the primary site. + +The following table details the components currently tested through the Geo secondary site Workhorse proxy. +It does not cover all data types, more will be added in the future as they are tested. + +| Feature / component | Accelerated reads? | +|:----------------------------------------------------|:-----------------------| +| Project, wiki, design repository (using the web UI) | **{dotted-circle}** No | +| Project, wiki repository (using Git) | **{check-circle}** Yes <sup>1</sup> | +| Project, Personal Snippet (using the web UI) | **{dotted-circle}** No | +| Project, Personal Snippet (using Git) | **{check-circle}** Yes <sup>1</sup> | +| Group wiki repository (using the web UI) | **{dotted-circle}** No | +| Group wiki repository (using Git) | **{check-circle}** Yes <sup>1</sup> | +| User uploads | **{dotted-circle}** No | +| LFS objects (using the web UI) | **{dotted-circle}** No | +| LFS objects (using Git) | **{check-circle}** Yes | +| Pages | **{dotted-circle}** No <sup>2</sup> | +| Advanced search (using the web UI) | **{dotted-circle}** No | + +1. Git reads are served from the local secondary while pushes get proxied to the primary. + Selective sync or cases where repositories don't exist locally on the Geo secondary throw a "not found" error. +1. Pages can use the same URL (without access control), but must be configured separately and are not proxied. diff --git a/doc/administration/geo/secondary_proxy/location_aware_external_url.md b/doc/administration/geo/secondary_proxy/location_aware_external_url.md new file mode 100644 index 00000000000..b5a8d4baa1f --- /dev/null +++ b/doc/administration/geo/secondary_proxy/location_aware_external_url.md @@ -0,0 +1,83 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: howto +--- + +# Location-aware public URL **(PREMIUM SELF)** + +With [Geo proxying for secondary sites](index.md), you can provide GitLab users +with a single URL that automatically uses the Geo site closest to them. +Users don't need to use different URLs or worry about read-only operations to take +advantage of closer Geo sites as they move. + +With [Geo proxying for secondary sites](index.md) web and Git requests are proxied +from **secondary** sites to the **primary**. + +Though these instructions use [AWS Route53](https://aws.amazon.com/route53/), +other services such as [Cloudflare](https://www.cloudflare.com/) can be used +as well. + +## Prerequisites + +This example creates a `gitlab.example.com` subdomain that automatically directs +requests: + +- From Europe to a **secondary** site. +- From all other locations to the **primary** site. + +The URLs to access each node by itself are: + +- `primary.example.com` as a Geo **primary** site. +- `secondary.example.com` as a Geo **secondary** site. + +For this example, you need: + +- A working GitLab **primary** site that is accessible at `gitlab.example.com` _and_ `primary.example.com`. +- A working GitLab **secondary** site. +- A Route53 Hosted Zone managing your domain for the Route53 setup. + +If you haven't yet set up a Geo _primary_ site and _secondary_ site, see the +[Geo setup instructions](../index.md#setup-instructions). + +## AWS Route53 + +### Create a traffic policy + +In a Route53 Hosted Zone, traffic policies can be used to set up a variety of +routing configurations. + +1. Go to the +[Route53 dashboard](https://console.aws.amazon.com/route53/home) and select +**Traffic policies**. + +1. Select **Create traffic policy**. +1. Fill in the **Policy Name** field with `Single Git Host` and select **Next**. +1. Leave **DNS type** as `A: IP Address in IPv4 format`. +1. Select **Connect to...**, then **Geolocation rule**. +1. For the first **Location**: + 1. Leave it as `Default`. + 1. Select **Connect to...**, then **New endpoint**. + 1. Choose **Type** `value` and fill it in with `<your **primary** IP address>`. + +1. For the second **Location**: + 1. Choose `Europe`. + 1. Select **Connect to...** and select **New endpoint**. + 1. Choose **Type** `value` and fill it in with `<your **secondary** IP address>`. + +  + +1. Select **Create traffic policy**. +1. Fill in **Policy record DNS name** with `gitlab`. + +  + +1. Select **Create policy records**. + +You have successfully set up a single host, like `gitlab.example.com`, which +distributes traffic to your Geo sites by geolocation. + +## Enable Geo proxying for secondary sites + +After setting up a single URL to use for all Geo sites, continue with the [steps to enable Geo proxying for secondary sites](index.md). diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index d72bb0737ae..7f4c771c962 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -7,23 +7,21 @@ type: howto # Geo database replication **(PREMIUM SELF)** -NOTE: -If your GitLab installation uses external (not managed by Omnibus) PostgreSQL -instances, the Omnibus roles are unable to perform all necessary -configuration steps. In this case, -[follow the Geo with external PostgreSQL instances document instead](external_database.md). +This document describes the minimal required steps to replicate your primary +GitLab database to a secondary node's database. You may have to change some +values, based on attributes including your database's setup and size. NOTE: -The stages of the setup process must be completed in the documented order. -Before attempting the steps in this stage, [complete all prior stages](../setup/index.md#using-omnibus-gitlab). +If your GitLab installation uses external (not managed by Omnibus GitLab) +PostgreSQL instances, the Omnibus roles cannot perform all necessary +configuration steps. In this case, use the [Geo with external PostgreSQL instances](external_database.md) +process instead. -This document describes the minimal steps you have to take to replicate your -**primary** GitLab database to a **secondary** node's database. You may have to -change some values, based on attributes including your database's setup and -size. +The stages of the setup process must be completed in the documented order. +Before you attempt the steps in this stage, [complete all prior stages](../setup/index.md#using-omnibus-gitlab). -You are encouraged to first read through all the steps before executing them -in your testing/production environment. +Be sure to read and review all of these steps before you execute them in your +testing or production environments. ## Single instance database replication @@ -214,7 +212,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ## - Prevents automatic upgrade of Postgres since it requires downtime of ## streaming replication to Geo secondary sites ## - Enables standard single-node GitLab services like NGINX, Puma, Redis, - ## Sidekiq, etc. If you are segregating services, then you will need to + ## or Sidekiq. If you are segregating services, then you will need to ## explicitly disable unwanted services. ## roles(['geo_primary_role']) @@ -690,8 +688,8 @@ could do it with [HAProxy](https://www.haproxy.org/). The following IPs and names are used as an example: - `10.6.0.21`: Patroni 1 (`patroni1.internal`) -- `10.6.0.21`: Patroni 2 (`patroni2.internal`) -- `10.6.0.22`: Patroni 3 (`patroni3.internal`) +- `10.6.0.22`: Patroni 2 (`patroni2.internal`) +- `10.6.0.23`: Patroni 3 (`patroni3.internal`) ```plaintext global @@ -716,7 +714,7 @@ backend postgresql server patroni1.internal 10.6.0.21:5432 maxconn 100 check port 8008 server patroni2.internal 10.6.0.22:5432 maxconn 100 check port 8008 - server patroni3.internal 10.6.0.23.195:5432 maxconn 100 check port 8008 + server patroni3.internal 10.6.0.23:5432 maxconn 100 check port 8008 ``` Refer to your preferred Load Balancer's documentation for further guidance. diff --git a/doc/administration/geo/setup/external_database.md b/doc/administration/geo/setup/external_database.md index 13dbf9d1434..756e73f022f 100644 --- a/doc/administration/geo/setup/external_database.md +++ b/doc/administration/geo/setup/external_database.md @@ -8,7 +8,7 @@ type: howto # Geo with external PostgreSQL instances **(PREMIUM SELF)** This document is relevant if you are using a PostgreSQL instance that is *not -managed by Omnibus*. This includes cloud-managed instances like AWS RDS, or +managed by Omnibus*. This includes cloud-managed instances like Amazon RDS, or manually installed and configured PostgreSQL instances. NOTE: @@ -58,7 +58,7 @@ developed and tested. We aim to be compatible with most external To set up an external database, you can either: -- Set up [streaming replication](https://www.postgresql.org/docs/12/warm-standby.html#STREAMING-REPLICATION-SLOTS) yourself (for example AWS RDS, bare metal not managed by Omnibus, and so on). +- Set up [streaming replication](https://www.postgresql.org/docs/12/warm-standby.html#STREAMING-REPLICATION-SLOTS) yourself (for example Amazon RDS, bare metal not managed by Omnibus, and so on). - Perform the Omnibus configuration manually as follows. #### Leverage your cloud provider's tools to replicate the primary database @@ -200,7 +200,7 @@ This is for the installation of extensions during installation and upgrades. As To setup an external tracking database, follow the instructions below: NOTE: -If you want to use AWS RDS as a tracking database, make sure it has access to +If you want to use Amazon RDS as a tracking database, make sure it has access to the secondary database. Unfortunately, just assigning the same security group is not enough as outbound rules do not apply to RDS PostgreSQL databases. Therefore, you need to explicitly add an inbound rule to the read-replica's security group allowing any TCP traffic from |
