1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
|
# Configuring Redis for GitLab HA
You can choose to install and manage Redis yourself, or you can use the one
that comes bundled with GitLab Omnibus packages.
> **Note:** Redis does not require authentication by default. See
[Redis Security](http://redis.io/topics/security) documentation for more
information. We recommend using a combination of a Redis password and tight
firewall rules to secure your Redis service.
## Configure your own Redis server
If you're hosting GitLab on a cloud provider, you can optionally use a
managed service for Redis. For example, AWS offers a managed ElastiCache service
that runs Redis.
## Configure Redis using Omnibus
If you don't want to bother setting up your own Redis server, you can use the
one bundled with Omnibus. In this case, you should disable all services except
Redis.
1. Download/install GitLab Omnibus using **steps 1 and 2** from
[GitLab downloads](https://about.gitlab.com/downloads). Do not complete other
steps on the download page.
1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration.
Be sure to change the `external_url` to match your eventual GitLab front-end
URL:
```ruby
external_url 'https://gitlab.example.com'
# Disable all services except Redis
redis['enable'] = true
bootstrap['enable'] = false
nginx['enable'] = false
unicorn['enable'] = false
sidekiq['enable'] = false
postgresql['enable'] = false
gitlab_workhorse['enable'] = false
mailroom['enable'] = false
# Redis configuration
redis['port'] = 6379
redis['bind'] = '0.0.0.0'
# If you wish to use Redis authentication (recommended)
redis['password'] = 'redis-password-goes-here'
```
1. Run `sudo gitlab-ctl reconfigure` to install and configure PostgreSQL.
> **Note**: This `reconfigure` step will result in some errors.
That's OK - don't be alarmed.
1. Run `touch /etc/gitlab/skip-auto-migrations` to prevent database migrations
from running on upgrade. Only the primary GitLab application server should
handle migrations.
## Experimental Redis Sentinel support
> [Introduced][ce-1877] in GitLab 8.11.
Since GitLab 8.11, you can configure a list of Redis Sentinel servers that
will monitor a group of Redis servers to provide you with a standard failover
support.
There is currently one exception to the Sentinel support: `mail_room`, the
component that processes incoming emails. It doesn't support Sentinel yet, but
we hope to integrate a future release that does support it.
To get a better understanding on how to correctly setup Sentinel, please read
the [Redis Sentinel documentation](http://redis.io/topics/sentinel) first, as
failing to configure it correctly can lead to data loss.
The configuration consists of three parts:
- Redis setup
- Sentinel setup
- GitLab setup
Read carefully how to configure those components below.
### Redis setup
You must have at least 2 Redis servers: 1 Master, 1 or more Slaves.
They should be configured the same way and with similar server specs, as
in a failover situation, any Slave can be elected as the new Master by
the Sentinel servers.
In a minimal setup, the only required change for the slaves in `redis.conf`
is the addition of a `slaveof` line pointing to the initial master.
You can increase the security by defining a `requirepass` configuration in
the master, and `masterauth` in slaves.
---
**Configuring your own Redis server**
1. Add to the slaves' `redis.conf`:
```conf
# IP and port of the master Redis server
slaveof 10.10.10.10 6379
```
1. Optionally, set up password authentication for increased security.
Add the following to master's `redis.conf`:
```conf
# Optional password authentication for increased security
requirepass "<password>"
```
1. Then add this line to all the slave servers' `redis.conf`:
```conf
masterauth "<password>"
```
1. Restart the Redis services for the changes to take effect.
---
**Using Redis via Omnibus**
1. Edit `/etc/gitlab/gitlab.rb` of a master Redis machine (usualy a single machine):
```ruby
## Redis TCP support (will disable UNIX socket transport)
redis['bind'] = '0.0.0.0' # or specify an IP to bind to a single one
redis['port'] = 6379
## Master redis instance
redis['password'] = 'redis-password-goes-here'
```
1. Edit `/etc/gitlab/gitlab.rb` of a slave Redis machine (should be one or more machines):
```ruby
## Redis TCP support (will disable UNIX socket transport)
redis['bind'] = '0.0.0.0' # or specify an IP to bind to a single one
redis['port'] = 6379
## Slave redis instance
redis['master'] = false
redis['master_ip'] = '10.10.10.10' # IP of master Redis server
redis['master_port'] = 6379 # Port of master Redis server
redis['master_password'] = "redis-password-goes-here"
```
1. Reconfigure the GitLab for the changes to take effect: `sudo gitlab-ctl reconfigure`
---
Now that the Redis servers are all set up, let's configure the Sentinel
servers.
### Sentinel setup
We provide an automated way to setup and run the Sentinel daemon
with GitLab EE.
See the instructions below how to setup it by yourself.
Here is an example configuration file (`sentinel.conf`) for a Sentinel node:
```conf
port 26379
sentinel monitor gitlab-redis 10.0.0.1 6379 1
sentinel down-after-milliseconds gitlab-redis 10000
sentinel config-epoch gitlab-redis 0
sentinel leader-epoch gitlab-redis 0
```
---
The final part is to inform the main GitLab application server of the Redis
master and the new sentinels servers.
### GitLab setup
You can enable or disable sentinel support at any time in new or existing
installations. From the GitLab application perspective, all it requires is
the correct credentials for the master Redis and for a few Sentinel nodes.
It doesn't require a list of all Sentinel nodes, as in case of a failure,
the application will need to query only one of them.
>**Note:**
The following steps should be performed in the [GitLab application server](gitlab.md).
**For source based installations**
1. Edit `/home/git/gitlab/config/resque.yml` following the example in
`/home/git/gitlab/config/resque.yml.example`, and uncomment the sentinels
line, changing to the correct server credentials.
1. Restart GitLab for the changes to take effect.
**For Omnibus installations**
1. Edit `/etc/gitlab/gitlab.rb` and add/change the following lines:
```ruby
redis['master_name'] = "gitlab-redis"
redis['master_password'] = 'redis-password-goes-here'
gitlab_rails['redis_sentinels'] = [
{'host' => '10.10.10.1', 'port' => 26379},
{'host' => '10.10.10.2', 'port' => 26379},
{'host' => '10.10.10.3', 'port' => 26379}
]
```
1. [Reconfigure] the GitLab for the changes to take effect.
### Sentinel troubleshooting
#### Omnibus install
If you get an error like: `Redis::CannotConnectError: No sentinels available.`,
there may be something wrong with your configuration files or it can be related
to [this issue][gh-531].
You must make sure you are defining the same value in `redis['master_name']`
and `redis['master_pasword']` as you defined for your sentinel node.
The way the redis connector `redis-rb` works with sentinel is a bit
non-intuitive. We try to hide the complexity in omnibus, but it still requires
a few extra configs.
#### Source install
If you get an error like: `Redis::CannotConnectError: No sentinels available.`,
there may be something wrong with your configuration files or it can be related
to [this issue][gh-531].
It's a bit non-intuitive the way you have to config `resque.yml` and
`sentinel.conf`, otherwise `redis-rb` will not work properly.
The `master-group-name` ('gitlab-redis') defined in (`sentinel.conf`)
**must** be used as the hostname in GitLab (`resque.yml` for source installations
or `gitlab-rails['redis_*']` in Omnibus):
```conf
# sentinel.conf:
sentinel monitor gitlab-redis 10.10.10.10 6379 1
sentinel down-after-milliseconds gitlab-redis 10000
sentinel config-epoch gitlab-redis 0
sentinel leader-epoch gitlab-redis 0
```
```yaml
# resque.yaml
production:
url: redis://:myredispassword@gitlab-redis/
sentinels:
-
host: slave1.example.com # or use ip
port: 26380 # point to sentinel, not to redis port
-
host: slave2.exampl.com # or use ip
port: 26381 # point to sentinel, not to redis port
```
When in doubt, please read [Redis Sentinel documentation](http://redis.io/topics/sentinel)
---
To make sure your configuration is correct:
1. SSH into your GitLab application server
1. Enter the Rails console:
```
# For Omnibus installations
sudo gitlab-rails console
# For source installations
sudo -u git rails console RAILS_ENV=production
```
1. Run in the console:
```ruby
redis = Redis.new(Gitlab::Redis.params)
redis.info
```
Keep this screen open and try to simulate a failover below.
1. To simulate a failover on master Redis, SSH into the Redis server and run:
```bash
# port must match your master redis port
redis-cli -h localhost -p 6379 DEBUG sleep 60
```
1. Then back in the Rails console from the first step, run:
```
redis.info
```
You should see a different port after a few seconds delay
(the failover/reconnect time).
---
Read more on high-availability configuration:
1. [Configure the database](database.md)
1. [Configure NFS](nfs.md)
1. [Configure the GitLab application servers](gitlab.md)
1. [Configure the load balancers](load_balancer.md)
[ce-1877]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/1877
[restart]: ../restart_gitlab.md#installations-from-source
[reconfigure]: ../restart_gitlab.md#omnibus-gitlab-reconfigure
[gh-531]: https://github.com/redis/redis-rb/issues/531
[gh-534]: https://github.com/redis/redis-rb/issues/534
|