diff options
Diffstat (limited to 'doc/administration/geo/replication')
8 files changed, 140 insertions, 95 deletions
diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index 7666a450648..39d1f5ae602 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -202,7 +202,7 @@ keys must be manually replicated to the **secondary** site. ``` 1. Navigate to the Primary Node GitLab Instance: - 1. On the top bar, select **Menu > Admin**. + 1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Geo > Sites**. 1. Select **Add site**.  @@ -222,19 +222,23 @@ keys must be manually replicated to the **secondary** site. gitlab-ctl restart ``` - Check if there are any common issue with your Geo setup by running: + Check if there are any common issues with your Geo setup by running: ```shell gitlab-rake gitlab:geo:check ``` + If any of the checks fail, check the [troubleshooting documentation](troubleshooting.md). + 1. SSH into a **Rails or Sidekiq server on your primary** site and login as root to verify the - **secondary** site is reachable or there are any common issue with your Geo setup: + **secondary** site is reachable or there are any common issues with your Geo setup: ```shell gitlab-rake gitlab:geo:check ``` + If any of the checks fail, check the [troubleshooting documentation](troubleshooting.md). + Once added to the Geo administration page and restarted, the **secondary** site automatically starts replicating missing data from the **primary** site in a process known as **backfill**. Meanwhile, the **primary** site starts to notify each **secondary** site of any changes, so @@ -305,7 +309,7 @@ method to be enabled. This is enabled by default, but if converting an existing On the **primary** site: -1. On the top bar, select **Menu > Admin**. +1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Settings > General**. 1. Expand **Visibility and access controls**. 1. Ensure "Enabled Git access protocols" is set to either "Both SSH and HTTP(S)" or "Only HTTP(S)". @@ -315,7 +319,7 @@ On the **primary** site: You can sign in to the **secondary** site with the same credentials you used with the **primary** site. After you sign in: -1. On the top bar, select **Menu > Admin**. +1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Geo > Sites**. 1. Verify that it's correctly identified as a **secondary** Geo site, and that Geo is enabled. diff --git a/doc/administration/geo/replication/container_registry.md b/doc/administration/geo/replication/container_registry.md index b425e5dcc0d..01ba81b6dbe 100644 --- a/doc/administration/geo/replication/container_registry.md +++ b/doc/administration/geo/replication/container_registry.md @@ -12,7 +12,7 @@ You can set up a Container Registry on your **secondary** Geo site that mirrors ## Supported container registries Geo supports the following types of container registries: - + - [Docker](https://docs.docker.com/registry/) - [OCI](https://github.com/opencontainers/distribution-spec/blob/main/spec.md) @@ -26,7 +26,7 @@ The following container image formats are support by Geo: In addition, Geo also supports [BuildKit cache images](https://github.com/moby/buildkit). -## Supported storage +## Supported storage ### Docker @@ -34,7 +34,7 @@ For more information on supported registry storage drivers see [Docker registry storage drivers](https://docs.docker.com/registry/storage-drivers/) Read the [Load balancing considerations](https://docs.docker.com/registry/deploying/#load-balancing-considerations) -when deploying the Registry, and how to set up the storage driver for the GitLab integrated +when deploying the Registry, and how to set up the storage driver for the GitLab integrated [Container Registry](../../packages/container_registry.md#use-object-storage). ### Registries that support OCI artifacts @@ -160,7 +160,7 @@ For each application and Sidekiq node on the **secondary** site: To verify Container Registry replication is working, on the **secondary** site: -1. On the top bar, select **Menu > Admin**. +1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Geo > Nodes**. The initial replication, or "backfill", is probably still in progress. diff --git a/doc/administration/geo/replication/disable_geo.md b/doc/administration/geo/replication/disable_geo.md index f0658ae45a2..84bc2e034b9 100644 --- a/doc/administration/geo/replication/disable_geo.md +++ b/doc/administration/geo/replication/disable_geo.md @@ -36,7 +36,7 @@ to do that. To remove the **primary** site: 1. [Remove all secondary Geo sites](remove_geo_site.md) -1. On the top bar, select **Menu > Admin**. +1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Geo > Nodes**. 1. Select **Remove** for the **primary** node. 1. Confirm by selecting **Remove** when the prompt appears. diff --git a/doc/administration/geo/replication/object_storage.md b/doc/administration/geo/replication/object_storage.md index d2e10678f8c..0336a1669f9 100644 --- a/doc/administration/geo/replication/object_storage.md +++ b/doc/administration/geo/replication/object_storage.md @@ -41,7 +41,7 @@ whether they are stored on the local file system or in object storage. To enable GitLab replication: -1. On the top bar, select **Menu > Admin**. +1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Geo > Nodes**. 1. Select **Edit** on the **secondary** site. 1. In the **Synchronization Settings** section, find the **Allow this secondary node to replicate content on Object Storage** diff --git a/doc/administration/geo/replication/remove_geo_site.md b/doc/administration/geo/replication/remove_geo_site.md index 0d6715a93b7..b136f6cc8b8 100644 --- a/doc/administration/geo/replication/remove_geo_site.md +++ b/doc/administration/geo/replication/remove_geo_site.md @@ -9,7 +9,7 @@ type: howto **Secondary** sites can be removed from the Geo cluster using the Geo administration page of the **primary** site. To remove a **secondary** site: -1. On the top bar, select **Menu > Admin**. +1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Geo > Nodes**. 1. Select the **Remove** button for the **secondary** site you want to remove. 1. Confirm by selecting **Remove** when the prompt appears. diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 26d192f62cd..d64ad2549e8 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -19,24 +19,24 @@ Here is a list of steps you should take to attempt to fix problem: Before attempting more advanced troubleshooting: -- Check [the health of the **secondary** node](#check-the-health-of-the-secondary-node). +- Check [the health of the **secondary** site](#check-the-health-of-the-secondary-site). - Check [if PostgreSQL replication is working](#check-if-postgresql-replication-is-working). -### Check the health of the **secondary** node +### Check the health of the **secondary** site -On the **primary** node: +On the **primary** site: -1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **Geo > Nodes**. +1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, select **Geo > Sites**. -We perform the following health checks on each **secondary** node +We perform the following health checks on each **secondary** site to help identify if something is wrong: -- Is the node running? -- Is the node's secondary database configured for streaming replication? -- Is the node's secondary tracking database configured? -- Is the node's secondary tracking database connected? -- Is the node's secondary tracking database up-to-date? +- Is the site running? +- Is the secondary site's database configured for streaming replication? +- Is the secondary site's tracking database configured? +- Is the secondary site's tracking database connected? +- Is the secondary site's tracking database up-to-date?  @@ -48,8 +48,8 @@ health check manually to get this information and a few more details. #### Health check Rake task -This Rake task can be run on an app node in the **primary** or **secondary** -Geo nodes: +This Rake task can be run on a **Rails** node in the **primary** or **secondary** +Geo sites: ```shell sudo gitlab-rake gitlab:geo:check @@ -275,11 +275,11 @@ sudo gitlab-rake gitlab:geo:check Checking Geo ... Finished ``` - Ensure you have added the secondary node in the Admin Area of the **primary** node. - Also ensure you entered the `external_url` or `gitlab_rails['geo_node_name']` - when adding the secondary node in the Admin Area of the **primary** node. - In GitLab 12.3 and earlier, edit the secondary node in the Admin Area of the **primary** - node and ensure that there is a trailing `/` in the `Name` field. + Ensure you have added the secondary site in the **Main menu > Admin > Geo > Sites** on the web interface for the **primary** site. + Also ensure you entered the `gitlab_rails['geo_node_name']` + when adding the secondary site in the Admin Area of the **primary** site. + In GitLab 12.3 and earlier, edit the secondary site in the Admin Area of the **primary** + site and ensure that there is a trailing `/` in the `Name` field. - Check returns `Exception: PG::UndefinedTable: ERROR: relation "geo_nodes" does not exist`. @@ -321,7 +321,7 @@ error messages (indicated by `Database replication working? ... no` in the This means that the `max_replication_slots` PostgreSQL variable needs to be set on the **primary** database. This setting defaults to 1. You may need to -increase this value if you have more **secondary** nodes. +increase this value if you have more **secondary** sites. Be sure to restart PostgreSQL for this to take effect. See the [PostgreSQL replication setup](../setup/database.md#postgresql-replication) guide for more details. @@ -329,13 +329,13 @@ Be sure to restart PostgreSQL for this to take effect. See the ### Message: `FATAL: could not start WAL streaming: ERROR: replication slot "geo_secondary_my_domain_com" does not exist`? This occurs when PostgreSQL does not have a replication slot for the -**secondary** node by that name. +**secondary** site by that name. -You may want to rerun the [replication process](../setup/database.md) on the **secondary** node . +You may want to rerun the [replication process](../setup/database.md) on the **secondary** site . ### Message: "Command exceeded allowed execution time" when setting up replication? -This may happen while [initiating the replication process](../setup/database.md#step-3-initiate-the-replication-process) on the **secondary** node, +This may happen while [initiating the replication process](../setup/database.md#step-3-initiate-the-replication-process) on the **secondary** site, and indicates your initial dataset is too large to be replicated in the default timeout (30 minutes). Re-run `gitlab-ctl replicate-geo-database`, but include a larger value for @@ -374,8 +374,8 @@ log data to build up in `pg_xlog`. Removing the unused slots can reduce the amou Slots where `active` is `f` are not active. -- When this slot should be active, because you have a **secondary** node configured using that slot, - sign in to that **secondary** node and check the [PostgreSQL logs](../../logs/index.md#postgresql-logs) +- When this slot should be active, because you have a **secondary** site configured using that slot, + sign in on the web interface for the **secondary** site and check the [PostgreSQL logs](../../logs/index.md#postgresql-logs) to view why the replication is not running. - If you are no longer using the slot (for example, you no longer have Geo enabled), you can remove it with in the @@ -398,12 +398,12 @@ These long-running queries are [planned to be removed in the future](https://gitlab.com/gitlab-org/gitlab/-/issues/34269), but as a workaround, we recommend enabling [`hot_standby_feedback`](https://www.postgresql.org/docs/10/hot-standby.html#HOT-STANDBY-CONFLICT). -This increases the likelihood of bloat on the **primary** node as it prevents +This increases the likelihood of bloat on the **primary** site as it prevents `VACUUM` from removing recently-dead rows. However, it has been used successfully in production on GitLab.com. To enable `hot_standby_feedback`, add the following to `/etc/gitlab/gitlab.rb` -on the **secondary** node: +on the **secondary** site: ```ruby postgresql['hot_standby_feedback'] = 'on' @@ -463,14 +463,14 @@ This happens if data is detected in the `projects` table. When one or more proje is aborted to prevent accidental data loss. To bypass this message, pass the `--force` option to the command. In GitLab 13.4, a seed project is added when GitLab is first installed. This makes it necessary to pass `--force` even -on a new Geo secondary node. There is an [issue to account for seed projects](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5618) +on a new Geo secondary site. There is an [issue to account for seed projects](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5618) when checking the database. ### Message: `Synchronization failed - Error syncing repository` WARNING: If large repositories are affected by this problem, -their resync may take a long time and cause significant load on your Geo nodes, +their resync may take a long time and cause significant load on your Geo sites, storage and network systems. If you see the error message `Synchronization failed - Error syncing repository` along with `fatal: fsck error in packed object`, this indicates @@ -483,7 +483,7 @@ it's possible to override the consistency checks instead. To do that, follow [the instructions in the Gitaly docs](../../gitaly/configure_gitaly.md#repository-consistency-checks). You can also get the error message `Synchronization failed - Error syncing repository` along with the following log messages, this indicates that the expected `geo` remote is not present in the `.git/config` file -of a repository on the secondary Geo node's file system: +of a repository on the secondary Geo site's file system: ```json { @@ -505,7 +505,7 @@ of a repository on the secondary Geo node's file system: To solve this: -1. Sign in to the secondary Geo node. +1. Sign in on the web interface for the secondary Geo site. 1. Back up [the `.git` folder](../../repository_storage_types.md#translate-hashed-storage-paths). @@ -538,7 +538,7 @@ To solve this: end ``` -### Very large repositories never successfully synchronize on the **secondary** node +### Very large repositories never successfully synchronize on the **secondary** site GitLab places a timeout on all repository clones, including project imports and Geo synchronization operations. If a fresh `git clone` of a repository @@ -546,7 +546,8 @@ on the **primary** takes more than the default three hours, you may be affected To increase the timeout: -1. On the **secondary** node, add the following line to `/etc/gitlab/gitlab.rb`: +1. On the **Sidekiq nodes on your secondary** site, +add the following line to `/etc/gitlab/gitlab.rb`: ```ruby gitlab_rails['gitlab_shell_git_timeout'] = 14400 @@ -563,9 +564,9 @@ long enough to accommodate a full clone of your largest repositories. ### New LFS objects are never replicated -If new LFS objects are never replicated to secondary Geo nodes, check the version of +If new LFS objects are never replicated to secondary Geo sites, check the version of GitLab you are running. GitLab versions 11.11.x or 12.0.x are affected by -[a bug that results in new LFS objects not being replicated to Geo secondary nodes](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). +[a bug that results in new LFS objects not being replicated to Geo secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). To resolve the issue, upgrade to GitLab 12.1 or later. @@ -574,9 +575,9 @@ To resolve the issue, upgrade to GitLab 12.1 or later. During a [backfill](../index.md#backfill), failures are scheduled to be retried at the end of the backfill queue, therefore these failures only clear up **after** the backfill completes. -### Resetting Geo **secondary** node replication +### Resetting Geo **secondary** site replication -If you get a **secondary** node in a broken state and want to reset the replication state, +If you get a **secondary** site in a broken state and want to reset the replication state, to start again from scratch, there are a few steps that can help you: 1. Stop Sidekiq and the Geo LogCursor. @@ -617,8 +618,8 @@ to start again from scratch, there are a few steps that can help you: 1. Optional. Rename other data folders and create new ones. WARNING: - You may still have files on the **secondary** node that have been removed from the **primary** node, but this - removal has not been reflected. If you skip this step, these files are not removed from the Geo node. + You may still have files on the **secondary** site that have been removed from the **primary** site, but this + removal has not been reflected. If you skip this step, these files are not removed from the Geo **secondary** site. Any uploaded content (like file attachments, avatars, or LFS objects) is stored in a subfolder in one of these paths: @@ -667,7 +668,7 @@ to start again from scratch, there are a few steps that can help you: ### Design repository failures on mirrored projects and project imports -On the top bar, under **Menu > Admin > Geo > Nodes**, +On the top bar, under **Main menu > Admin > Geo > Sites**, if the Design repositories progress bar shows `Synced` and `Failed` greater than 100%, and negative `Queued`, the instance is likely affected by @@ -714,7 +715,7 @@ Counts: {"synced"=>3} ``` -#### If you are promoting a Geo secondary site running on a single server +#### If you are promoting a Geo secondary site running on a single node `gitlab-ctl promotion-preflight-checks` fails due to the existence of `failed` rows in the `geo_design_registry` table. Use the @@ -831,10 +832,10 @@ We recommend transferring each failing repository individually and checking for after each transfer. Follow the [single target `rsync` instructions](../../operations/moving_repositories.md#single-rsync-to-another-server) to transfer each affected repository from the primary to the secondary site. -## Fixing errors during a failover or when promoting a secondary to a primary node +## Fixing errors during a failover or when promoting a secondary to a primary site The following are possible error messages that might be encountered during failover or -when promoting a secondary to a primary node with strategies to resolve them. +when promoting a secondary to a primary site with strategies to resolve them. ### Message: `ActiveRecord::RecordInvalid: Validation failed: Name has already been taken` @@ -868,14 +869,14 @@ or `gitlab-ctl promote-to-primary-node`, either: ``` - Upgrade to GitLab 12.6.3 or later if it is safe to do so. For example, - if the failover was just a test. A + if the failover was just a test. A [caching-related bug](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22021) was fixed. ### Message: `ActiveRecord::RecordInvalid: Validation failed: Enabled Geo primary node cannot be disabled` -If you disabled a secondary node, either with the [replication pause task](../index.md#pausing-and-resuming-replication) +If you disabled a secondary site, either with the [replication pause task](../index.md#pausing-and-resuming-replication) (GitLab 13.2) or by using the user interface (GitLab 13.1 and earlier), you must first -re-enable the node before you can continue. This is fixed in GitLab 13.4. +re-enable the site before you can continue. This is fixed in GitLab 13.4. This can be fixed in the database. @@ -894,7 +895,7 @@ This can be fixed in the database. ``` 1. Run the following command, replacing `https://<secondary url>/` with the URL - for your secondary server. You can use either `http` or `https`, but ensure that you + for your secondary node. You can use either `http` or `https`, but ensure that you end the URL with a slash (`/`): ```sql @@ -987,32 +988,31 @@ sudo gitlab-rake geo:set_secondary_as_primary ## Expired artifacts If you notice for some reason there are more artifacts on the Geo -secondary node than on the Geo primary node, you can use the Rake task +**secondary** site than on the Geo **primary** site, you can use the Rake task to [cleanup orphan artifact files](../../../raketasks/cleanup.md#remove-orphan-artifact-files). -On a Geo **secondary** node, this command also cleans up all Geo +On a Geo **secondary** site, this command also cleans up all Geo registry record related to the orphan files on disk. ## Fixing sign in errors ### Message: The redirect URI included is not valid -If you are able to sign in to the **primary** node, but you receive this error message -when attempting to sign in to a **secondary**, you should verify the Geo -node's URL matches its external URL. +If you are able to sign in to the web interface for the **primary** site, but you receive this error message +when attempting to sign in to a **secondary** web interface, you should verify the Geo +site's URL matches its external URL. -On the **primary** node: +On the **primary** site: -1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **Geo > Nodes**. +1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, select **Geo > Sites**. 1. Find the affected **secondary** site and select **Edit**. 1. Ensure the **URL** field matches the value found in `/etc/gitlab/gitlab.rb` - in `external_url "https://gitlab.example.com"` on the frontend servers of - the **secondary** node. + in `external_url "https://gitlab.example.com"` on the **Rails nodes of the secondary** site. ## Fixing common errors -This section documents common error messages reported in the Admin Area, and how to fix them. +This section documents common error messages reported in the Admin Area on the web interface, and how to fix them. ### Geo database configuration file is missing @@ -1029,11 +1029,11 @@ has the correct permissions. Geo cannot reuse an existing tracking database. It is safest to use a fresh secondary, or reset the whole secondary by following -[Resetting Geo secondary node replication](#resetting-geo-secondary-node-replication). +[Resetting Geo secondary site replication](#resetting-geo-secondary-site-replication). -### Geo node has a database that is writable which is an indication it is not configured for replication with the primary node +### Geo site has a database that is writable which is an indication it is not configured for replication with the primary site -This error message refers to a problem with the database replica on a **secondary** node, +This error message refers to a problem with the database replica on a **secondary** site, which Geo expects to have access to. It usually means, either: - An unsupported replication method was used (for example, logical replication). @@ -1043,24 +1043,24 @@ which Geo expects to have access to. It usually means, either: Geo **secondary** sites require two separate PostgreSQL instances: -- A read-only replica of the **primary** node. +- A read-only replica of the **primary** site. - A regular, writable instance that holds replication metadata. That is, the Geo tracking database. This error message indicates that the replica database in the **secondary** site is misconfigured and replication has stopped. To restore the database and resume replication, you can do one of the following: -- [Reset the Geo secondary site replication](#resetting-geo-secondary-node-replication). +- [Reset the Geo secondary site replication](#resetting-geo-secondary-site-replication). - [Set up a new secondary Geo Omnibus instance](../setup/index.md#using-omnibus-gitlab). If you set up a new secondary from scratch, you must also [remove the old site from the Geo cluster](remove_geo_site.md#removing-secondary-geo-sites). -### Geo node does not appear to be replicating the database from the primary node +### Geo site does not appear to be replicating the database from the primary site The most common problems that prevent the database from replicating correctly are: -- **Secondary** nodes cannot reach the **primary** node. Check credentials, firewall rules, and so on. -- SSL certificate problems. Make sure you copied `/etc/gitlab/gitlab-secrets.json` from the **primary** node. +- **Secondary** sites cannot reach the **primary** site. Check credentials, [firewall rules](../index.md#firewall-rules), and so on. +- SSL certificate problems. Make sure you copied `/etc/gitlab/gitlab-secrets.json` from the **primary** site. - Database storage disk is full. - Database replication slot is misconfigured. - Database is not using a replication slot or another alternative and cannot catch-up because WAL files were purged. @@ -1072,26 +1072,26 @@ Make sure you follow the [Geo database replication](../setup/database.md) instru If you are using Omnibus GitLab installation, something might have failed during upgrade. You can: - Run `sudo gitlab-ctl reconfigure`. -- Manually trigger the database migration by running: `sudo gitlab-rake db:migrate:geo` as root on the **secondary** node. +- Manually trigger the database migration by running: `sudo gitlab-rake db:migrate:geo` as root on the **secondary** site. ### GitLab indicates that more than 100% of repositories were synced This can be caused by orphaned records in the project registry. You can clear them [using a Rake task](../../../administration/raketasks/geo.md#remove-orphaned-project-registries). -### Geo Admin Area returns 404 error for a secondary node +### Geo Admin Area returns 404 error for a secondary site -Sometimes `sudo gitlab-rake gitlab:geo:check` indicates that the **secondary** node is -healthy, but a 404 Not Found error message for the **secondary** node is returned in the Geo Admin Area on -the **primary** node. +Sometimes `sudo gitlab-rake gitlab:geo:check` indicates that **Rails nodes of the secondary** sites are +healthy, but a 404 Not Found error message for the **secondary** site is returned in the Geo Admin Area on the web interface for +the **primary** site. To resolve this issue: -- Try restarting the **secondary** using `sudo gitlab-ctl restart`. -- Check `/var/log/gitlab/gitlab-rails/geo.log` to see if the **secondary** node is - using IPv6 to send its status to the **primary** node. If it is, add an entry to - the **primary** node using IPv4 in the `/etc/hosts` file. Alternatively, you should - [enable IPv6 on the **primary** node](https://docs.gitlab.com/omnibus/settings/nginx.html#setting-the-nginx-listen-address-or-addresses). +- Try restarting **each Rails, Sidekiq and Gitaly nodes on your secondary site** using `sudo gitlab-ctl restart`. +- Check `/var/log/gitlab/gitlab-rails/geo.log` on Sidekiq nodes to see if the **secondary** site is + using IPv6 to send its status to the **primary** site. If it is, add an entry to + the **primary** site using IPv4 in the `/etc/hosts` file. Alternatively, you should + [enable IPv6 on the **primary** site](https://docs.gitlab.com/omnibus/settings/nginx.html#setting-the-nginx-listen-address-or-addresses). ### Secondary site returns 502 errors with Geo proxying @@ -1167,7 +1167,7 @@ To fix this issue, set the primary site's internal URL to a URL that is: You may have problems if you're running a version of [Git LFS](https://git-lfs.github.com/) before 2.4.2. As noted in [this authentication issue](https://github.com/git-lfs/git-lfs/issues/3025), -requests redirected from the secondary to the primary node do not properly send the +requests redirected from the secondary to the primary site do not properly send the Authorization header. This may result in either an infinite `Authorization <-> Redirect` loop, or Authorization error messages. @@ -1194,13 +1194,13 @@ The partial failover to a secondary Geo *site* may be the result of a temporary/ 1. SSH into every Sidekiq, PostgresSQL, Gitaly, and Rails node in the **secondary** site and run one of the following commands: - - To promote the secondary node to primary: + - To promote the secondary site to primary: ```shell sudo gitlab-ctl geo promote ``` - - To promote the secondary node to primary **without any further confirmation**: + - To promote the secondary site to primary **without any further confirmation**: ```shell sudo gitlab-ctl geo promote --force @@ -1230,3 +1230,37 @@ If the above steps are **not successful**, proceed through the next steps: 1. Verify you can connect to the newly-promoted **primary** site using the URL used previously for the **secondary** site. 1. If successful, the **secondary** site is now promoted to the **primary** site. + +## Additional tools + +There are useful snippets for manipulating Geo internals in the [GitLab Rails Cheat Sheet](../../troubleshooting/gitlab_rails_cheat_sheet.md#geo). For example, you can find how to manually sync or verify a replicable in Rails console. + +## Check OS locale data compatibility + +If different operating systems or different operating system versions are deployed across Geo sites, we recommend that you perform a locale data compatibility check setting up Geo. + +Geo uses PostgreSQL and Streaming Replication to replicate data across Geo sites. PostgreSQL uses locale data provided by the operating system’s C library for sorting text. If the locale data in the C library is incompatible across Geo sites, erroneous query results that lead to [incorrect behavior on secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/360723). See [here](https://wiki.postgresql.org/wiki/Locale_data_changes) for more details. + +On all hosts running PostgreSQL, across all Geo sites, run the following shell command: + +```shell +( echo "1-1"; echo "11" ) | LC_COLLATE=en_US.UTF-8 sort +``` + +The output will either look like: + +```plaintext +1-1 +11 +``` + +or the reverse order: + +```plaintext +11 +1-1 +``` + +If the output is identical on all hosts, then they running compatible versions of locale data. + +If the output differs on some hosts, then PostgreSQL replication will not work properly. We advise that you select operating system versions that are compatible. diff --git a/doc/administration/geo/replication/tuning.md b/doc/administration/geo/replication/tuning.md index 755ab45a76c..370c50c93db 100644 --- a/doc/administration/geo/replication/tuning.md +++ b/doc/administration/geo/replication/tuning.md @@ -14,7 +14,7 @@ in the background. On the **primary** site: -1. On the top bar, select **Menu > Admin**. +1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Geo > Sites**. 1. Select **Edit** of the secondary site you want to tune. 1. Under **Tuning settings**, there are several variables that can be tuned to diff --git a/doc/administration/geo/replication/version_specific_upgrades.md b/doc/administration/geo/replication/version_specific_upgrades.md index 350310c7076..6211b5689b1 100644 --- a/doc/administration/geo/replication/version_specific_upgrades.md +++ b/doc/administration/geo/replication/version_specific_upgrades.md @@ -157,6 +157,13 @@ to perform the following additional steps for the zero-downtime upgrade: sudo gitlab-rake db:migrate ``` +1. Hot reload `puma` and `sidekiq` services: + + ```shell + sudo gitlab-ctl hup puma + sudo gitlab-ctl restart sidekiq + ``` + If you have already run the final `sudo gitlab-rake db:migrate` command on the deploy node and have encountered the [column rename issue](https://gitlab.com/gitlab-org/gitlab/-/issues/324160), you might see the following error: @@ -183,8 +190,8 @@ GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab each upgraded reference. Delay any upgrade attempts until this is in the [13.7.5 patch release.](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3002). More details are available [in this issue](https://gitlab.com/gitlab-org/git/-/issues/79). -- A new secret is generated in `/etc/gitlab/gitlab-secrets.json`. - In an HA GitLab or GitLab Geo environment, secrets need to be the same on all nodes. +- A new secret is generated in `/etc/gitlab/gitlab-secrets.json`. + In an HA GitLab or GitLab Geo environment, secrets need to be the same on all nodes. Ensure this new secret is also accounted for if you are manually syncing the file across nodes, or manually specifying secrets in `/etc/gitlab/gitlab.rb`. @@ -247,7 +254,7 @@ the recommended procedure, see the ## Upgrading to GitLab 12.9 WARNING: -GitLab 12.9.0 through GitLab 12.9.3 are affected by +GitLab 12.9.0 through GitLab 12.9.3 are affected by [a bug that stops repository verification](https://gitlab.com/gitlab-org/gitlab/-/issues/213523). The issue is fixed in GitLab 12.9.4. Upgrade to GitLab 12.9.4 or later. @@ -401,6 +408,6 @@ For the recommended procedure, see the ## Upgrading to GitLab 12.0 WARNING: -This version is affected by a +This version is affected by a [bug that results in new LFS objects not being replicated to Geo secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). The issue is fixed in GitLab 12.1. Be sure to upgrade to GitLab 12.1 or later. |