diff options
-rw-r--r-- | doc/administration/high_availability/redis.md | 648 | ||||
-rw-r--r-- | doc/administration/high_availability/redis_source.md | 14 |
2 files changed, 480 insertions, 182 deletions
diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index f49bf1d4ab5..6b16f9e791f 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -1,9 +1,14 @@ # Configuring Redis for GitLab HA -You can choose to install and manage Redis yourself, or you can use the one +High Availability with Redis is possible using a **Master** x **Slave** +topology with **Sentinel** service to watch and automatically start +failover proceedings. + +You can choose to install and manage Redis and Sentinel yourself, use +a hosted, managed clouse solution or you can use or you can use the one that comes bundled with Omnibus GitLab packages. -> **Note:** Redis does not require authentication by default. See +> **Note:** Redis requires authentication for High Availability. See [Redis Security](http://redis.io/topics/security) documentation for more information. We recommend using a combination of a Redis password and tight firewall rules to secure your Redis service. @@ -12,20 +17,26 @@ that comes bundled with Omnibus GitLab packages. <!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --> **Table of Contents** -- [Configure your own Redis server](#configure-your-own-redis-server) -- [Configure Redis using Omnibus](#configure-redis-using-omnibus) -- [Experimental Redis Sentinel support](#experimental-redis-sentinel-support) -- [Redis Sentinel support](#redis-sentinel-support) +- [Using an external Redis server](#using-an-external-redis-server) +- [High Availability with Sentinel](#high-availability-with-sentinel) - [Prerequisites](#prerequisites) - [Redis setup](#redis-setup) - - [Existing single-machine installation](#existing-single-machine-installation) - - [Omnibus packages](#omnibus-packages) - - [Configuring Sentinel](#configuring-sentinel) - - [How sentinel handles a failover](#how-sentinel-handles-a-failover) - - [Sentinel setup](#sentinel-setup) + - [Experimental Redis Sentinel support](#experimental-redis-sentinel-support) + - [Sentinel setup](#sentinel-setup) + - [Recommended setup](#recommended-setup) +- [Configuring instances using Omnibus](#configuring-instances-using-omnibus) + - [Existing single-machine installation](#existing-single-machine-installation) + - [Configuring Master Redis instance](#configuring-master-redis-instance) + - [Configuring Slave Redis instances](#configuring-slave-redis-instances) + - [Configuring Sentinel instances](#configuring-sentinel-instances) - [Community Edition](#community-edition) - [Enterprise Edition](#enterprise-edition) - [GitLab setup](#gitlab-setup) +- [Example Configurations](#example-configurations) + - [Configuration for Redis Master](#configuration-for-redis-master) + - [Configuration for Redis Slave](#configuration-for-redis-slave) + - [Configuration for Sentinel (EE only)](#configuration-for-sentinel-ee-only) + - [Control running services](#control-running-services) - [Troubleshooting](#troubleshooting) - [Redis replication](#redis-replication) - [Sentinel](#sentinel) @@ -33,75 +44,57 @@ that comes bundled with Omnibus GitLab packages. <!-- END doctoc generated TOC please keep comment here to allow auto update --> -## Configure your own Redis server +## Using an external Redis server If you're hosting GitLab on a cloud provider, you can optionally use a managed service for Redis. For example, AWS offers a managed ElastiCache service that runs Redis. -## Configure Redis using Omnibus +Managed services can provide High Availability using their own proprietary +technology and provide a transparent proxy, which means that GitLab doesn't +need any additional change, or will use Sentinel and manage it for you. -If you don't want to bother setting up your own Redis server, you can use the -one bundled with Omnibus. In this case, you should disable all services except -Redis. +If your provider, uses Sentinel method, see [GitLab Setup](#gitlab-setup) +to understant where you need to provide the list of servers and credentials. -1. Download/install Omnibus GitLab using **steps 1 and 2** from - [GitLab downloads](https://about.gitlab.com/downloads). Do not complete other - steps on the download page. -1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration. - Be sure to change the `external_url` to match your eventual GitLab front-end - URL: +If you want to setup Redis by yourself, without using Omnibus, you can +read our documentation: [Configuring Redis for GitLab HA (Source Install)](redis_source.md) - ```ruby - external_url 'https://gitlab.example.com' - - # Disable all services except Redis - redis['enable'] = true - bootstrap['enable'] = false - nginx['enable'] = false - postgresql['enable'] = false - gitlab_rails['enable'] = false - - # Redis configuration - redis['port'] = 6379 - redis['bind'] = '0.0.0.0' - redis['password'] = 'redis-password-goes-here' - ``` +## High Availability with Sentinel -1. Run `sudo gitlab-ctl reconfigure` to install and configure Redis. +> Since GitLab `8.11`, you can configure a list of Redis Sentinel servers that +will monitor a group of Redis servers to provide failover support. - > **Note**: This `reconfigure` step will result in some errors. - That's OK - don't be alarmed. +> With GitLab `8.14`, we bundled Redis Sentinel as part of Omnibus package and +improved the way you use and configure it. -1. Run `touch /etc/gitlab/skip-auto-migrations` to prevent database migrations - from running on upgrade. Only the primary GitLab application server should - handle migrations. +High Availability with Redis requires a few things: -## Experimental Redis Sentinel support +- Multiple Redis instances +- Run Redis in a **Master** x **Slave** topology +- Multiple Sentinel instances +- Application support and visiblity to all Sentinel and Redis instances - > Experimental Redis Sentinel support was [Introduced][ce-1877] in GitLab 8.11. - Starting with 8.14, Redis Sentinel is no longer experimental. - If you used with versions `< 8.14` before, please check the updated - documentation below. +Redis Sentinel can handle the most important tasks in a HA environment to help +keep servers online with minimal to no downtime: -## Redis Sentinel support +- Monitors **Master** and **Slaves** instances to see if they are available +- Promote a **Slave** to **Master** when the **Master** fails. +- Demote a **Master** to **Slave** when failed **Master** comes back online (to prevent + data-partitioning). +- Can be queried by clients to always connect to the current **Master** server. -Since GitLab 8.11, you can configure a list of Redis Sentinel servers that -will monitor a group of Redis servers to provide you with a standard failover -support. +When a **Master** fails to respond, it's the client responsability to handle timeout +and reconnect (querying a **Sentinel** for a new **Master**). To get a better understanding on how to correctly setup Sentinel, please read the [Redis Sentinel documentation](http://redis.io/topics/sentinel) first, as -failing to configure it correctly can lead to data loss. +failing to configure it correctly can lead to data loss, or can bring your +whole cluster down, invalidating the failover effort. -Redis Sentinel can handle the most important tasks in a HA environment to help -keep servers online with minimal to no downtime: - -- Monitors master and slave instances to see if they are available -- Promote a slave to master when the master fails. -- Demote a master to slave when failed master comes back online (to prevent - data-partitioning). -- Can be queried by clients to always connect to the correct master server. +This documentation will provide you with a minimal and a recommended topology +that can resist to some levels of failure. Usually the more Redis and Sentinel +instances you have provisioned, the better will be your availability. The configuration consists of three parts: @@ -112,44 +105,191 @@ The configuration consists of three parts: ### Prerequisites You need at least `3` independent machines: physical, or VMs running into -distinct physical machines. +distinct physical machines. They must be believed to fail in an +independent way. If you fail to provision the machines in that specific way, any issue with the shared environment can bring your entire setup down. -Read carefully how to configure those components below. +You also need to take in consideration the underlying network topology, +making sure you have redundant connectivity between Redis / Sentinel and +GitLab instances, otherwhise the networks will become a single point of +failure. + +Read carefully how to configure the components below. ### Redis setup -You must have at least `3` Redis servers: `1` Master, `2` Slaves, and they need to -be each in a independent machine (see explanation above). +You must have at least `3` Redis servers: `1` Master, `2` Slaves, and they +need to be each in a independent machine (see explanation above). -They should be configured the same way and with similar server specs, as -in a failover situation, any `Slave` can be elected as the new `Master` by +You can have additional Redis nodes, that will help survive a situation +where more nodes goes down. Whenever there is only `2` nodes online, a failover +will not be initiated. + +As an example, if you have `6` Redis nodes, a maximum of `3` can be +simultaneously down. + +Please note that there are different requirements for Sentinel nodes. +If you host them in the same Redis machines, you may need to take +that restrictions into consideration when calculating the ammount of +nodes to be provisioned. See [Sentinel setup](#sentinel-setup) +documentation for more information. + +All Redis nodes should be configured the same way and with similar server specs, as +in a failover situation, any **Slave** can be promoted as the new **Master** by the Sentinel servers. -With Sentinel, you must define a password to protect the access as both -Sentinel instances and other redis instances should be able to talk to +The replication requires authentication, so you need to define a password to +protect all Redis nodes and the Sentinels. They will all share the same +password, and all instances must be able to talk to each other over the network. -You'll need to define both `requirepass` and `masterauth` in all -nodes. At any time during a failover the Sentinels can reconfigure a node -and change it's status from `Master` to `Slave` and vice versa. +Redis nodes will need the same password defined in `redis['password']` and +`redis['master_password']`, no matter if **Master** or **Slave**. At any time +during a failover the Sentinels can reconfigure a node and change it's status +from **Master** to **Slave** and vice versa. + +Initial **Slave** nodes requires `redis['master']` defined to `false` and +`redis['master_ip']` pointing to the initial **Master**. If you use the +simplified configuration by enabling `redis_slave_role['enable']`, you +just need to fill in the `redis['master_ip']`. + +This values doesn't have to be changed again in `/etc/gitlab/gitlab.rb` after +a failover, as the nodes will be managed by the Sentinels, and even after a +`gitlab-ctl reconfigure`, they will get their configuration restored by +the same Sentinels. + +### Experimental Redis Sentinel support + + > Experimental Redis Sentinel support was [Introduced][ce-1877] in GitLab 8.11. + Starting with 8.14, Redis Sentinel is no longer experimental. + If you used with versions `< 8.14` before, please check the updated + documentation here. + +### Sentinel setup + +Sentinels watches both other sentinels and Redis nodes. Whenever a Sentinel +detects that a Redis node is not responding, it will announce that to the +other sentinels. You have to reach the **quorum**, the minimum ammount of +sentinels that agrees that a node is down, to be able to start a failover. + +Whenver the **quorum** is met, you need the **majority** of all known +Sentinel nodes to be available and reachable, to elect the Sentinel **leader** +who will take all the decisions to restore the service availability by: + +- Promoting a new **Master** +- Reconfiguring the other **Slaves** and make them point to the new **Master** +- Announce the new **Master** to every other Sentinel peer +- Reconfigure the old **Master** and demote to **Slave** when it comes back online + +You must have at least `3` Redis Sentinel servers, and they need to +be each in a independent machine (that are believed to fail independently). + +You can configure them in the same machines where you've configured the other +Redis servers, but understand that if a whole node goes down, you loose both +a Sentinel and a Redis instance. -Initial `Slave` nodes require an additional `slaveof` setting in `redis.conf` -pointing to the initial `Master`. +The number of sentinels should ideally always be an **odd** number, for the +consensus algorithm to be effective in the case of a failure. -#### Existing single-machine installation +In a `3` nodes topology, you can only afford `1` Sentinel node going down. +Whenever the **majority** of the Sentinels goes down, the network partition +protection prevents destructive actions and a failover **will not be started**. + +Here are some examples: + +- With `5` or `6` sentinels, a maximum of `2` can go down for a failover begin. +- With `7` sentinels, a maximum of `3` nodes can go down. + +The **Leader** election can sometimes fail the voting round when **consensus**, +is not achieved (see the odd number of nodes requirement above). In that case, +a new attempt will be made after the amount of time defined in +`sentinel['failover_timeout']` (in milliseconds). + +The `failover_time` variable have a lot of different usages, according to +official documentation: + +- The time needed to re-start a failover after a previous failover was + already tried against the same master by a given Sentinel, is two + times the failover timeout. + +- The time needed for a slave replicating to a wrong master according + to a Sentinel current configuration, to be forced to replicate + with the right master, is exactly the failover timeout (counting since + the moment a Sentinel detected the misconfiguration). + +- The time needed to cancel a failover that is already in progress but + did not produced any configuration change (SLAVEOF NO ONE yet not + acknowledged by the promoted slave). + +- The maximum time a failover in progress waits for all the slaves to be + reconfigured as slaves of the new master. However even after this time + the slaves will be reconfigured by the Sentinels anyway, but not with + the exact parallel-syncs progression as specified. + +### Recommended setup + +For a minimal setup, you will install the Omnibus GitLab package in `3` +independent machines, both with **Redis** and **Sentinel**: + +- Redis Master + Sentinel +- Redis Slave + Sentinel +- Redis Slave + Sentinel + +Make sure you've read [Redis Setup](#redis-setup) and [Sentinel Setup](#sentinel-setup) +before, to understant how and why the ammount of nodes came from. + +For a recommended setup, that can resist more failures, you will install +the Omnibus GitLab package in `5` independent machines, both with +**Redis** and **Sentinel**: + +- Redis Master + Sentinel +- Redis Slave + Sentinel +- Redis Slave + Sentinel +- Redis Slave + Sentinel +- Redis Slave + Sentinel + +## Configuring instances using Omnibus + +This is a summary of what are we going to do: + +1. Provision the required number of instances specified previously + - You can opt to install Redis and Sentinel in the same machine or each in + independent ones. + - Don't install Redis and Sentinel in the same machines your GitLab instance + is running on. + - All machines must be able to talk to each other and accept incomming + connection over Redis (`6379`) and Sentinel (`26379`) ports. + - GitLab machines must be able to access these machines and with the same + permissions. + - Protected them from indiscriminated access from external networks (Internet), + to harden the security. + +1. Download/install Omnibus GitLab using **steps 1 and 2** from + [GitLab downloads](https://about.gitlab.com/downloads) in each node. + - Do not complete other steps on the download page. + - Make sure you select the correct Omnibus package, with the same version + and type (Community, Enterprise editions) of your current install. + +1. Run `touch /etc/gitlab/skip-auto-migrations` to prevent database migrations + from running on upgrade. Only the primary GitLab application server should + handle migrations. + +1. Create/edit `/etc/gitlab/gitlab.rb` and make the changes based on the + [Example Configurations](#example-configurations). + +### Existing single-machine installation If you already have a single-machine GitLab install running, you will need to replicate from this machine first, before de-activating the Redis instance inside it. -Your single-machine install will be the initial `Master`, and the `3` others -should be configured as `Slave` pointing to this machine. +Your single-machine install will be the initial **Master**, and the `3` others +should be configured as **Slave** pointing to this machine. After replication catchs-up, you will need to stop services in the -single-machine install, to rotate the `Master` to one of the new nodes. +single-machine install, to rotate the **Master** to one of the new nodes. Make the required changes in configuration and restart the new nodes again. @@ -159,130 +299,317 @@ To disable redis in the single install, edit `/etc/gitlab/gitlab.rb`: redis['enable'] = false ``` -#### Omnibus packages +If you fail to replicate first, you may loose data (unprocessed background jobs). + +### Configuring Master Redis instance + +You will need to configure the following in `/etc/gitlab/gitlab.rb`: + +1. Define `redis_master_role['enable']` to `true`, to disable other services + in the machine (you can still enable Sentinel) + +1. Define a `redis['bind']` address pointing to a local IP that your other machines + can reach you. + - If you really need to bind to an external acessible IP, make + sure you add extra firewall rules to prevent unauthorized access. + - You can also set bind to `0.0.0.0` which listen in all interfaces. + +1. Define a `redis['port']` so redis can listen for TCP requests which will + allow other machines to connect to it. + +1. Set up a password authentication with `redis['password']` and + `redis['master_password']` (use the same password in all nodes). + +Reconfigure Omnibus GitLab for the changes to take effect: `sudo gitlab-ctl reconfigure` -You need to install the Omnibus GitLab package in `3` independent machines. +### Configuring Slave Redis instances -**Configuring Master Redis instance** +You will need to configure the following in `/etc/gitlab/gitlab.rb`: -You will need to configure the following: +1. Define `redis_slaves_role['enable']` to `true`, to disable other services + in the machine (you can still enable Sentinel) + - This will also set automatically `redis['master'] = false`. 1. Define a `redis['bind']` address pointing to a local IP that your other machines - can reach you. If you really need to bind to an external acessible IP, make + can reach you. + - If you really need to bind to an external acessible IP, make sure you add extra firewall rules to prevent unauthorized access. + - You can also set bind to `0.0.0.0` which listen in all interfaces. + 1. Define a `redis['port']` so redis can listen for TCP requests which will allow other machines to connect to it. -1. Set up a password authentication with `redis['master_password']` (use the same - password in all nodes). + +1. Set up a password authentication with `redis['password']` and + `redis['master_password']` (use the same password in all nodes). + +1. Define `redis['master_ip']` with the IP of the **Master** Redis. + +1. Define `redis['master_port']` with the port of the **Master** Redis (default to `6379`). + +### Configuring Sentinel instances + +Now that the Redis servers are all set up, let's configure the Sentinel +servers. + +If you are not sure if your Redis servers are working and replicating +correctly, please read the [Troubleshooting Replication](#troubleshooting-replication) +and fix it before proceeding with Sentinel setup. + +You must have at least `3` Redis Sentinel servers, and they need to +be each in a independent machine. You can configure them in the same +machines where you've configured the other Redis servers. + +##### Community Edition + +With GitLab Community Edition, you need to install, configure, execute and +monitor Sentinel from source. Omnibus GitLab Community Edition package does +not support Sentinel configuration. + +See [documentation for Source Install](redis_source.md). + +##### Enterprise Edition + +With GitLab Enterprise Edition, you can use Omnibus package to setup multiple +machines with Sentinel daemon. + +See [example configuration](#configuration-for-sentinel-ee-only) below. + +### GitLab setup + +The final part is to inform the main GitLab application server of the Redis +Sentinels servers and authentication credentials. + +You can enable or disable Sentinel support at any time in new or existing +installations. From the GitLab application perspective, all it requires is +the correct credentials for the Sentinel nodes. + +While it doesn't require a list of all Sentinel nodes, in case of a failure, +it needs to access at least one of listeds. + +>**Note:** +The following steps should be performed in the [GitLab application server](gitlab.md) +which ideally should not have Redis or Sentinels in the same machine for a HA setup. + +1. Edit `/etc/gitlab/gitlab.rb` and add/change the following lines: + + - `redis['master_name']` - this is the `master-group-name` from sentinel (default: `gitlab-redis`) + - `redis['master_password']` - the same password you've defined before for Redis and Sentinels + - `gitlab_rails['redis_sentinels']` - a list of sentinels with `host` and `port` + +1. [Reconfigure] GitLab for the changes to take effect. + +See [example configuration](#configuration-for-gitlab) below. + +## Example Configurations + +In this example we consider that all servers have an internal network +interface with IPs in the `10.0.0.x` range, and that they can connect +to each other using these IPs. + +In a real world usage, you would also setup firewall rules to prevent +unauthorized access from other machines, and block traffic from the +outside (Internet). + +We will use the same `3` nodes with **Redis** + **Sentinel** topology +discussed in [Redis Setup](#redis-setup) and [Sentinel Setup](#sentinel-setup) +documentation. + +Here is a list and description of each **machine** and the assigned **IP**: + +* `10.0.0.1`: Redis Master + Sentinel 1 +* `10.0.0.2`: Redis Slave 1 + Sentinel 2 +* `10.0.0.2`: Redis Slave 2 + Sentinel 3 + +Please note that after the initial configuration, if a failover is initiated +by the Sentinel nodes, the Redis nodes will be reconfigured and the **Master** +will change permanently (including in `redis.conf`) from one node to the other, +until a new failover is initiated again. + +The same thing will happen with `sentinel.conf` that will be overriten after the +initial execution, after any new sentinel node starts watching the **Master**, +or a failover promotes a different **Master** node. + +### Configuration for Redis Master + +**Example configation for Redis Master:** In `/etc/gitlab/gitlab.rb`: ```ruby -## Redis TCP support (will disable UNIX socket transport) -redis['bind'] = '0.0.0.0' # or specify an IP to bind to a single one +redis_master_role['enable'] = true + +redis['bind'] = '10.0.0.1' redis['port'] = 6379 redis['password'] = 'redis-password-goes-here' redis['master_password'] = 'redis-password-goes-here' ``` - Reconfigure Omnibus GitLab for the changes to take effect: `sudo gitlab-ctl reconfigure` -**Configuring Slave Redis instances** +### Configuration for Redis Slave + +**Example configation for Slave 1:** -You need to make the same changes listed for the `Master` instance, -with an additional `Slave` section as in the example below: +In `/etc/gitlab/gitlab.rb`: ```ruby -redis['bind'] = '0.0.0.0' # or specify an IP to bind to a single one +redis_slave_role['enable'] = true + +redis['bind'] = '10.0.0.2' redis['port'] = 6379 redis['password'] = 'redis-password-goes-here' redis['master_password'] = 'redis-password-goes-here' -## Slave redis instance -redis['master'] = false -redis['master_ip'] = '10.10.10.10' # IP of master Redis server -redis['master_port'] = 6379 # Port of master Redis server +redis['master_ip'] = '10.0.0.1' # IP of master Redis server +#redis['master_port'] = 6379 # Port of master Redis server, uncomment to change to non default ``` Reconfigure Omnibus GitLab for the changes to take effect: `sudo gitlab-ctl reconfigure` ---- - -Now that the Redis servers are all set up, let's configure the Sentinel -servers. +**Example configation for Slave 2:** -If you are not sure if your Redis servers are working and replicating -correctly, please read the [Troubleshooting Replication](#troubleshooting-replication) -and fix it before proceeding with Sentinel setup. +In `/etc/gitlab/gitlab.rb`: -### Configuring Sentinel +```ruby +redis_slave_role['enable'] = true -You must have at least `3` Redis Sentinel servers, and they need to -be each in a independent machine. You can configure them in the same -machines where you've configured the other Redis servers. +redis['bind'] = '10.0.0.3' +redis['port'] = 6379 +redis['password'] = 'redis-password-goes-here' +redis['master_password'] = 'redis-password-goes-here' -This number is required for the consensus algorithm to be effective -in the case of a failure. **You should always have an `odd` number -of Sentinel nodes provisioned**. +redis['master_ip'] = '10.0.0.1' # IP of master Redis server +#redis['master_port'] = 6379 # Port of master Redis server, uncomment to change to non default +``` -#### How sentinel handles a failover +Reconfigure Omnibus GitLab for the changes to take effect: `sudo gitlab-ctl reconfigure` -If (`quorum` value of) Sentinels agree the fact the `master` is not reachable, -Sentinels will try to elect a temporary `Leader`. The **Majority** of the -Sentinels must agree to start a failover. +### Configuration for Sentinel (EE only) -If you don't have the **Majority** of the Sentinels online (for example if you -are under a network partitioning), a failover **will not be started**. +Please note that some of the variables are already configured previously +as they are required for Redis replication. -For example, for a cluster of `3` Sentinels, at least `2` must agree on a -`Leader`. If you have total of `5` at least `3` must agree on a `Leader`. +**Example configation for Sentinel 1:** -The `quorum` is only used to detect failure, not to elect the `Leader`. +In `/etc/gitlab/gitlab.rb`: -Official [Sentinel documentation](http://redis.io/topics/sentinel#example-sentinel-deployments) -also lists different network topologies and warns againts situations like -network partition and how it can affect the state of the HA solution. Make -sure you read it carefully and understand the implications in your current -setup. +```ruby +redis_sentinel_role['enable'] = true -GitLab Enterprise Edition provides [automated way to setup and run](#sentinel-setup-ee-only) the Sentinel daemon. +redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node +redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance +#redis['master_port'] = 6379 # port of the initial master redis instance, uncomment to change to non default +redis['master_password'] = 'redis-password-goes-here' # the same value defined in redis['password'] in the master instance -#### Sentinel setup +## Configure Sentinel +sentinel['bind'] = '10.0.0.1' +# sentinel['port'] = 26379 # uncomment to change default port -##### Community Edition +## Quorum must reflect the amount of voting sentinels it take to start a failover. +## Value must NOT be greater then the ammount of sentinels. +## +## The quorum can be used to tune Sentinel in two ways: +## 1. If a the quorum is set to a value smaller than the majority of Sentinels +## we deploy, we are basically making Sentinel more sensible to master failures, +## triggering a failover as soon as even just a minority of Sentinels is no longer +## able to talk with the master. +## 1. If a quorum is set to a value greater than the majority of Sentinels, we are +## making Sentinel able to failover only when there are a very large number (larger +## than majority) of well connected Sentinels which agree about the master being down.s +sentinel['quorum'] = 2 -With GitLab Community Edition, you need to install, configure, execute and -monitor Sentinel from source. Omnibus GitLab Community Edition package does -not support Sentinel configuration. +## Consider unresponsive server down after x amount of ms. +# sentinel['down_after_milliseconds'] = 10000 -See documentation for Source Install [here](redis_source.md). +## Specifies the failover timeout in milliseconds. It is used in many ways: +## +## - The time needed to re-start a failover after a previous failover was +## already tried against the same master by a given Sentinel, is two +## times the failover timeout. +## +## - The time needed for a slave replicating to a wrong master according +## to a Sentinel current configuration, to be forced to replicate +## with the right master, is exactly the failover timeout (counting since +## the moment a Sentinel detected the misconfiguration). +## +## - The time needed to cancel a failover that is already in progress but +## did not produced any configuration change (SLAVEOF NO ONE yet not +## acknowledged by the promoted slave). +## +## - The maximum time a failover in progress waits for all the slaves to be +## reconfigured as slaves of the new master. However even after this time +## the slaves will be reconfigured by the Sentinels anyway, but not with +## the exact parallel-syncs progression as specified. +# sentinel['failover_timeout'] = 60000 +``` -##### Enterprise Edition +**Example configation for Sentinel 2:** -To setup sentinel, edit `/etc/gitlab/gitlab.rb` file: +In `/etc/gitlab/gitlab.rb`: ```ruby +redis_sentinel_role['enable'] = true -## When you install Sentinel in a separate machine, you need to control which -## other services will be running in it. -## We've simplified the choice using special "roles" settings: +redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node +redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance +#redis['master_port'] = 6379 # port of the initial master redis instance, uncomment to change to non default +redis['master_password'] = 'redis-password-goes-here' # the same value defined in redis['password'] in the master instance -## Enabled Sentinel and Redis Master services -redis_sentinel_role['enable'] = true -redis_master_role['enable'] = true +## Configure Sentinel +sentinel['bind'] = '10.0.0.2' +# sentinel['port'] = 26379 # uncomment to change default port + +## Quorum must reflect the amount of voting sentinels it take to start a failover. +## Value must NOT be greater then the ammount of sentinels. +## +## The quorum can be used to tune Sentinel in two ways: +## 1. If a the quorum is set to a value smaller than the majority of Sentinels +## we deploy, we are basically making Sentinel more sensible to master failures, +## triggering a failover as soon as even just a minority of Sentinels is no longer +## able to talk with the master. +## 1. If a quorum is set to a value greater than the majority of Sentinels, we are +## making Sentinel able to failover only when there are a very large number (larger +## than majority) of well connected Sentinels which agree about the master being down.s +sentinel['quorum'] = 2 -## Enabled Sentinel and Redis Slave services +## Consider unresponsive server down after x amount of ms. +# sentinel['down_after_milliseconds'] = 10000 + +## Specifies the failover timeout in milliseconds. It is used in many ways: +## +## - The time needed to re-start a failover after a previous failover was +## already tried against the same master by a given Sentinel, is two +## times the failover timeout. +## +## - The time needed for a slave replicating to a wrong master according +## to a Sentinel current configuration, to be forced to replicate +## with the right master, is exactly the failover timeout (counting since +## the moment a Sentinel detected the misconfiguration). +## +## - The time needed to cancel a failover that is already in progress but +## did not produced any configuration change (SLAVEOF NO ONE yet not +## acknowledged by the promoted slave). +## +## - The maximum time a failover in progress waits for all the slaves to be +## reconfigured as slaves of the new master. However even after this time +## the slaves will be reconfigured by the Sentinels anyway, but not with +## the exact parallel-syncs progression as specified. +# sentinel['failover_timeout'] = 60000 +``` + +**Example configation for Sentinel 3:** + +In `/etc/gitlab/gitlab.rb`: + +```ruby redis_sentinel_role['enable'] = true -redis_slave_role['enable'] = true -## Configure Redis redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance -redis['master_port'] = 6379 # port of the initial master redis instance +#redis['master_port'] = 6379 # port of the initial master redis instance, uncomment to change to non default redis['master_password'] = 'redis-password-goes-here' # the same value defined in redis['password'] in the master instance ## Configure Sentinel -# sentinel['bind'] = '0.0.0.0' # bind to all interfaces, uncomment to specify an IP and bind to a single one +sentinel['bind'] = '10.0.0.3' # sentinel['port'] = 26379 # uncomment to change default port ## Quorum must reflect the amount of voting sentinels it take to start a failover. @@ -323,6 +650,8 @@ sentinel['quorum'] = 2 # sentinel['failover_timeout'] = 60000 ``` +### Control running services + In the example above we've used `redis_sentinel_role` and `redis_master_role` which simplify the ammount of configuration changes. @@ -365,37 +694,6 @@ mailroom['enable'] = false # Redis Slave role also change this setting from default 'true' to 'false': redis['master'] = false ``` ---- - -The final part is to inform the main GitLab application server of the Redis -master and the new sentinels servers. - -### GitLab setup - -You can enable or disable Sentinel support at any time in new or existing -installations. From the GitLab application perspective, all it requires is -the correct credentials for the Sentinel nodes. - -While it doesn't require a list of all Sentinel nodes, in case of a failure, -it needs to access at one of listed ones. - ->**Note:** -The following steps should be performed in the [GitLab application server](gitlab.md) -which ideally should not have Redis or Sentinels in the same machine for a HA setup. - -1. Edit `/etc/gitlab/gitlab.rb` and add/change the following lines: - - ```ruby - redis['master_name'] = "gitlab-redis" - redis['master_password'] = 'redis-password-goes-here' - gitlab_rails['redis_sentinels'] = [ - {'host' => '10.10.10.1', 'port' => 26379}, - {'host' => '10.10.10.2', 'port' => 26379}, - {'host' => '10.10.10.3', 'port' => 26379} - ] - ``` - -1. [Reconfigure] GitLab for the changes to take effect. ## Troubleshooting diff --git a/doc/administration/high_availability/redis_source.md b/doc/administration/high_availability/redis_source.md index 7e8c8c2b4b9..ab5bb13f070 100644 --- a/doc/administration/high_availability/redis_source.md +++ b/doc/administration/high_availability/redis_source.md @@ -153,7 +153,7 @@ outside (Internet). We will use the same `3` nodes with **Redis** + **Sentinel** topology discussed in the [Configuring Redis for GitLab HA](redis.md) documentation. -Here is a list and description of each **machine** and the assined **ip**: +Here is a list and description of each **machine** and the assigned **IP**: * `10.0.0.1`: Redis Master + Sentinel 1 * `10.0.0.2`: Redis Slave 1 + Sentinel 2 @@ -170,7 +170,7 @@ or a failover promotes a different **Master** node. ### Configuring Redis Master -`redis.conf`: +**Example configation for Redis Master - `redis.conf`:** ```conf bind 10.0.0.1 @@ -181,7 +181,7 @@ masterauth redis-password-goes-here ### Configuring Redis Slaves -**Slave 1 - `redis.conf`:** +**Example configation for Slave 1 - `redis.conf`:** ```conf bind 10.0.0.2 @@ -193,7 +193,7 @@ masterauth redis-password-goes-here slaveof 10.0.0.1 6379 ``` -**Slave 2 - `redis.conf`:** +**Example configation for Slave 2 - `redis.conf`:** ```conf bind 10.0.0.3 @@ -211,7 +211,7 @@ For this example, **Sentinel 1** will be configured in the same machine as the **Redis Master**, **Sentinel 2** and **Sentinel 3** in the same machines as the **Slave 1** and **Slave 2** respectively. -Sentinel 1 - `sentinel.conf` +**Example configation for Sentinel 1 - `sentinel.conf`:** ```conf bind 10.0.0.1 @@ -222,7 +222,7 @@ sentinel down-after-milliseconds gitlab-redis 10000 sentinel failover_timeout 30000 ``` -Sentinel 2 - `sentinel.conf` +**Example configation for Sentinel 2 - `sentinel.conf`:** ```conf bind 10.0.0.2 @@ -233,7 +233,7 @@ sentinel down-after-milliseconds gitlab-redis 10000 sentinel failover_timeout 30000 ``` -Sentinel 3 - `sentinel.conf` +**Example configation for Sentinel 3 - `sentinel.conf`:** ```conf bind 10.0.0.3 |