1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
|
---
stage: Enablement
group: Geo
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
type: howto
---
# Version specific update instructions
Check this document if it includes instructions for the version you are updating.
These steps go together with the [general steps](updating_the_geo_nodes.md#general-update-steps)
for updating Geo nodes.
## Updating to GitLab 13.3
In GitLab 13.3, Geo removed the PostgreSQL [Foreign Data Wrapper](https://www.postgresql.org/docs/11/postgres-fdw.html) dependency for the tracking database.
The FDW server, user, and the extension will be removed during the upgrade process on each **secondary** node. The GitLab settings related to the FDW in the `/etc/gitlab/gitlab.rb` have been deprecated and can be safely removed.
There are some scenarios like using an external PostgreSQL instance for the tracking database where the FDW settings must be removed manually. Enter the PostgreSQL console of that instance and remove them:
```shell
DROP SERVER gitlab_secondary CASCADE;
DROP EXTENSION IF EXISTS postgres_fdw;
```
DANGER: **Warning:**
In GitLab 13.3, promoting a secondary node to a primary while the secondary is paused fails. Do not pause replication before promoting a secondary. If the node is paused, please resume before promoting. To avoid this issue, upgrade to GitLab 13.4 or later.
## Updating to GitLab 13.2
In GitLab 13.2, promoting a secondary node to a primary while the secondary is paused fails. Do not pause replication before promoting a secondary. If the node is paused, please resume before promoting. To avoid this issue, upgrade to GitLab 13.4 or later.
## Updating to GitLab 13.0
Upgrading to GitLab 13.0 requires GitLab 12.10 to already be using PostgreSQL
version 11. Please see
[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance)
for the recommended procedure.
## Updating to GitLab 12.10
GitLab 12.10 does not attempt to automatically update the embedded PostgreSQL
server when using Geo, because the PostgreSQL upgrade requires downtime on
secondaries while reinitializing streaming replication. It must be upgraded
manually. Please see
[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance)
for the recommended procedure.
## Updating to GitLab 12.9
CAUTION: **Warning:**
GitLab 12.9.0 through GitLab 12.9.3 are affected by [a bug that stops
repository verification](https://gitlab.com/gitlab-org/gitlab/-/issues/213523).
The issue is fixed in GitLab 12.9.4. Please upgrade to GitLab 12.9.4 or later.
By default, GitLab 12.9 will attempt to automatically update the embedded
PostgreSQL server to 10.12 from 9.6, which requires downtime on secondaries
while reinitializing streaming replication. Please see
[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance)
for the recommended procedure.
This can be temporarily disabled by running the following before updating:
```shell
sudo touch /etc/gitlab/disable-postgresql-upgrade
```
## Updating to GitLab 12.8
By default, GitLab 12.8 will attempt to automatically update the embedded
PostgreSQL server to 10.12 from 9.6, which requires downtime on secondaries
while reinitializing streaming replication. Please see
[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance)
for the recommended procedure.
This can be temporarily disabled by running the following before updating:
```shell
sudo touch /etc/gitlab/disable-postgresql-upgrade
```
## Updating to GitLab 12.7
DANGER: **Warning:**
Only upgrade to GitLab 12.7.5 or later. Do not upgrade to versions 12.7.0
through 12.7.4 because there is [an initialization order
bug](https://gitlab.com/gitlab-org/gitlab/-/issues/199672) that causes Geo
**secondaries** to set the incorrect database connection pool size. [The
fix](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/24021) was
shipped in 12.7.5.
By default, GitLab 12.7 will attempt to automatically update the embedded
PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries while
reinitializing streaming replication. Please see
[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance)
for the recommended procedure.
This can be temporarily disabled by running the following before updating:
```shell
sudo touch /etc/gitlab/disable-postgresql-upgrade
```
## Updating to GitLab 12.6
By default, GitLab 12.6 will attempt to automatically update the embedded
PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries while
reinitializing streaming replication. Please see
[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance)
for the recommended procedure.
This can be temporarily disabled by running the following before updating:
```shell
sudo touch /etc/gitlab/disable-postgresql-upgrade
```
## Updating to GitLab 12.5
By default, GitLab 12.5 will attempt to automatically update the embedded
PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries while
reinitializing streaming replication. Please see
[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance)
for the recommended procedure.
This can be temporarily disabled by running the following before updating:
```shell
sudo touch /etc/gitlab/disable-postgresql-upgrade
```
## Updating to GitLab 12.4
By default, GitLab 12.4 will attempt to automatically update the embedded
PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries while
reinitializing streaming replication. Please see
[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance)
for the recommended procedure.
This can be temporarily disabled by running the following before updating:
```shell
sudo touch /etc/gitlab/disable-postgresql-upgrade
```
## Updating to GitLab 12.3
DANGER: **Warning:**
If the existing PostgreSQL server version is 9.6.x, it is recommended to
upgrade to GitLab 12.4 or newer. By default, GitLab 12.3 will attempt to
automatically update the embedded PostgreSQL server to 10.9 from 9.6. In
certain circumstances, it will fail. Please see
[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance)
for more information.
Additionally, if the PostgreSQL upgrade does not fail, a successful upgrade
requires downtime on secondaries while reinitializing streaming replication.
Please see
[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance)
for the recommended procedure.
## Updating to GitLab 12.2
DANGER: **Warning:**
If the existing PostgreSQL server version is 9.6.x, it is recommended to
upgrade to GitLab 12.4 or newer. By default, GitLab 12.2 will attempt to
automatically update the embedded PostgreSQL server to 10.9 from 9.6. In
certain circumstances, it will fail. Please see
[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance)
for more information.
Additionally, if the PostgreSQL upgrade does not fail, a successful upgrade
requires downtime on secondaries while reinitializing streaming replication.
Please see
[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance)
for the recommended procedure.
GitLab 12.2 includes the following minor PostgreSQL updates:
- To version `9.6.14` if you run PostgreSQL 9.6.
- To version `10.9` if you run PostgreSQL 10.
This update will occur even if major PostgreSQL updates are disabled.
Before [refreshing Foreign Data Wrapper during a Geo upgrade](https://docs.gitlab.com/omnibus/update/README.html#run-post-deployment-migrations-and-checks),
restart the Geo tracking database:
```shell
sudo gitlab-ctl restart geo-postgresql
```
The restart avoids a version mismatch when PostgreSQL tries to load the FDW extension.
## Updating to GitLab 12.1
DANGER: **Warning:**
If the existing PostgreSQL server version is 9.6.x, it is recommended to
upgrade to GitLab 12.4 or newer. By default, GitLab 12.1 will attempt to
automatically update the embedded PostgreSQL server to 10.9 from 9.6. In
certain circumstances, it will fail. Please see
[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance)
for more information.
Additionally, if the PostgreSQL upgrade does not fail, a successful upgrade
requires downtime on secondaries while reinitializing streaming replication.
Please see
[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance)
for the recommended procedure.
## Updating to GitLab 12.0
CAUTION: **Warning:**
This version is affected by [a bug that results in new LFS objects not being replicated to
Geo secondary nodes](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). The issue is fixed
in GitLab 12.1. Please upgrade to GitLab 12.1 or later.
## Updating to GitLab 11.11
CAUTION: **Warning:**
This version is affected by [a bug that results in new LFS objects not being replicated to
Geo secondary nodes](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). The issue is fixed
in GitLab 12.1. Please upgrade to GitLab 12.1 or later.
## Updating to GitLab 10.8
Before 10.8, broadcast messages would not propagate without flushing
the cache on the **secondary** nodes. This has been fixed in 10.8, but
requires one last cache flush on each **secondary** node:
```shell
sudo gitlab-rake cache:clear
```
## Updating to GitLab 10.6
In 10.4, we started to recommend that you define a password for database user (`gitlab`).
We now require this change as we use this password to enable the Foreign Data Wrapper, as a way to optimize
the Geo Tracking Database. We are also improving security by disabling the use of **trust**
authentication method.
1. **(primary)** Login to your **primary** node and run:
```shell
gitlab-ctl pg-password-md5 gitlab
# Enter password: <your_password_here>
# Confirm password: <your_password_here>
# fca0b89a972d69f00eb3ec98a5838484
```
Copy the generated hash and edit `/etc/gitlab/gitlab.rb`:
```ruby
# Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab`
postgresql['sql_user_password'] = '<md5_hash_of_your_password>'
# Every node that runs Unicorn or Sidekiq needs to have the database
# password specified as below.
# This must be present in all application nodes.
gitlab_rails['db_password'] = '<your_password_here>'
```
Still in the configuration file, locate and remove the `trust_auth_cidr_address`:
```ruby
postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','1.2.3.4/32'] # <- Remove this
```
1. **(primary)** Reconfigure and restart:
```shell
sudo gitlab-ctl reconfigure
sudo gitlab-ctl restart
```
1. **(secondary)** Login to all **secondary** nodes and edit `/etc/gitlab/gitlab.rb`:
```ruby
# Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab`
postgresql['sql_user_password'] = '<md5_hash_of_your_password>'
# Every node that runs Unicorn or Sidekiq needs to have the database
# password specified as below.
# This must be present in all application nodes.
gitlab_rails['db_password'] = '<your_password_here>'
# Enable Foreign Data Wrapper
geo_secondary['db_fdw'] = true
# Secondary address in CIDR format, for example '5.6.7.8/32'
postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32']
```
Still in the configuration file, locate and remove the `trust_auth_cidr_address`:
```ruby
postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','5.6.7.8/32'] # <- Remove this
```
1. **(secondary)** Reconfigure and restart:
```shell
sudo gitlab-ctl reconfigure
sudo gitlab-ctl restart
```
## Updating to GitLab 10.5
For Geo Disaster Recovery to work with minimum downtime, your **secondary** node
should use the same set of secrets as the **primary** node. However, setup instructions
prior to the 10.5 release only synchronized the `db_key_base` secret.
To rectify this error on existing installations, you should **overwrite** the
contents of `/etc/gitlab/gitlab-secrets.json` on each **secondary** node with the
contents of `/etc/gitlab/gitlab-secrets.json` on the **primary** node, then run the
following command on each **secondary** node:
```shell
sudo gitlab-ctl reconfigure
```
If you do not perform this step, you may find that two-factor authentication
[is broken following DR](troubleshooting.md#two-factor-authentication-is-broken-after-a-failover).
To prevent SSH requests to the newly promoted **primary** node from failing
due to SSH host key mismatch when updating the **primary** node domain's DNS record
you should perform the step to [Manually replicate **primary** SSH host keys](configuration.md#step-2-manually-replicate-the-primary-nodes-ssh-host-keys) in each
**secondary** node.
## Updating to GitLab 10.3
### Support for SSH repository synchronization removed
In GitLab 10.2, synchronizing secondaries over SSH was deprecated. In 10.3,
support is removed entirely. All installations will switch to the HTTP/HTTPS
cloning method instead. Before updating, ensure that all your Geo nodes are
configured to use this method and that it works for your installation. In
particular, ensure that [Git access over HTTP/HTTPS is enabled](configuration.md#step-6-enable-git-access-over-httphttps).
Synchronizing repositories over the public Internet using HTTP is insecure, so
you should ensure that you have HTTPS configured before updating. Note that
file synchronization is **also** insecure in these cases!
## Updating to GitLab 10.2
### Secure PostgreSQL replication
Support for TLS-secured PostgreSQL replication has been added. If you are
currently using PostgreSQL replication across the open internet without an
external means of securing the connection (e.g., a site-to-site VPN), then you
should immediately reconfigure your **primary** and **secondary** PostgreSQL instances
according to the [updated instructions](../setup/database.md).
If you *are* securing the connections externally and wish to continue doing so,
ensure you include the new option `--sslmode=prefer` in future invocations of
`gitlab-ctl replicate-geo-database`.
### HTTPS repository sync
Support for replicating repositories and wikis over HTTP/HTTPS has been added.
Replicating over SSH has been deprecated, and support for this option will be
removed in a future release.
To switch to HTTP/HTTPS replication, log into the **primary** node as an admin and visit
**Admin Area > Geo** (`/admin/geo/nodes`). For each **secondary** node listed,
press the "Edit" button, change the "Repository cloning" setting from
"SSH (deprecated)" to "HTTP/HTTPS", and press "Save changes". This should take
effect immediately.
Any new secondaries should be created using HTTP/HTTPS replication - this is the
default setting.
After you've verified that HTTP/HTTPS replication is working, you should remove
the now-unused SSH keys from your secondaries, as they may cause problems if the
**secondary** node if ever promoted to a **primary** node:
1. **(secondary)** Login to **all** your **secondary** nodes and run:
```ruby
sudo -u git -H rm ~git/.ssh/id_rsa ~git/.ssh/id_rsa.pub
```
### Hashed Storage
CAUTION: **Warning:**
Hashed storage is in **Alpha**. It is considered experimental and not
production-ready. See [Hashed Storage](../../repository_storage_types.md) for more detail.
If you previously enabled Hashed Storage and migrated all your existing
projects to Hashed Storage, disabling hashed storage will not migrate projects
to their previous project based storage path. As such, once enabled and
migrated we recommend leaving Hashed Storage enabled.
## Updating to GitLab 10.1
CAUTION: **Warning:**
Hashed storage is in **Alpha**. It is considered experimental and not
production-ready. See [Hashed Storage](../../repository_storage_types.md) for more detail.
[Hashed storage](../../repository_storage_types.md) was introduced in
GitLab 10.0, and a [migration path](../../raketasks/storage.md) for
existing repositories was added in GitLab 10.1.
## Updating to GitLab 10.0
In GitLab 10.0 and later, we require all **Geo** systems to [use SSH key lookups via
the database](../../operations/fast_ssh_key_lookup.md) to avoid having to maintain consistency of the
`authorized_keys` file for SSH access. Failing to do this will prevent users
from being able to clone via SSH.
Note that in older versions of Geo, attachments downloaded on the **secondary**
nodes would be saved to the wrong directory. We recommend that you do the
following to clean this up.
On the **secondary** Geo nodes, run as root:
```shell
mv /var/opt/gitlab/gitlab-rails/working /var/opt/gitlab/gitlab-rails/working.old
mkdir /var/opt/gitlab/gitlab-rails/working
chmod 700 /var/opt/gitlab/gitlab-rails/working
chown git:git /var/opt/gitlab/gitlab-rails/working
```
You may delete `/var/opt/gitlab/gitlab-rails/working.old` any time.
Once this is done, we advise restarting GitLab on the **secondary** nodes for the
new working directory to be used:
```shell
sudo gitlab-ctl restart
```
## Updating from GitLab 9.3 or older
If you started running Geo on GitLab 9.3 or older, we recommend that you
resync your **secondary** PostgreSQL databases to use replication slots. If you
started using Geo with GitLab 9.4 or 10.x, no further action should be
required because replication slots are used by default. However, if you
started with GitLab 9.3 and updated later, you should still follow the
instructions below.
When in doubt, it does not hurt to do a resync. The easiest way to do this in
Omnibus is the following:
1. Make sure you have Omnibus GitLab on the **primary** server.
1. Run `gitlab-ctl reconfigure` and `gitlab-ctl restart postgresql`. This will enable replication slots on the **primary** database.
1. Check the steps about defining `postgresql['sql_user_password']`, `gitlab_rails['db_password']`.
1. Make sure `postgresql['max_replication_slots']` matches the number of **secondary** Geo nodes locations.
1. Install GitLab on the **secondary** server.
1. Re-run the [database replication process](../setup/database.md#step-3-initiate-the-replication-process).
## Updating to GitLab 9.0
> **IMPORTANT**:
With GitLab 9.0, the PostgreSQL version is updated to 9.6 and manual steps are
required to update the **secondary** nodes and keep the Streaming Replication
working. Downtime is required, so plan ahead.
The following steps apply only if you update from a 8.17 GitLab version to
9.0+. For previous versions, update to GitLab 8.17 first before attempting to
update to 9.0+.
---
Make sure to follow the steps in the exact order as they appear below and pay
extra attention in what node (either **primary** or **secondary**) you execute them! Each step
is prepended with the relevant node for better clarity:
1. **(secondary)** Log in to **all** your **secondary** nodes and stop all services:
```ruby
sudo gitlab-ctl stop
```
1. **(secondary)** Make a backup of the `recovery.conf` file on **all**
**secondary** nodes to preserve PostgreSQL's credentials:
```shell
sudo cp /var/opt/gitlab/postgresql/data/recovery.conf /var/opt/gitlab/
```
1. **(primary)** Update the **primary** node to GitLab 9.0 following the
[regular update docs](../../../update/README.md). At the end of the
update, the **primary** node will be running with PostgreSQL 9.6.
1. **(primary)** To prevent a de-synchronization of the repository replication,
stop all services except `postgresql` as we will use it to re-initialize the
**secondary** node's database:
```shell
sudo gitlab-ctl stop
sudo gitlab-ctl start postgresql
```
1. **(secondary)** Run the following steps on each of the **secondary** nodes:
1. **(secondary)** Stop all services:
```shell
sudo gitlab-ctl stop
```
1. **(secondary)** Prevent running database migrations:
```shell
sudo touch /etc/gitlab/skip-auto-migrations
```
1. **(secondary)** Move the old database to another directory:
```shell
sudo mv /var/opt/gitlab/postgresql{,.bak}
```
1. **(secondary)** Update to GitLab 9.0 following the [regular update docs](../../../update/README.md).
At the end of the update, the node will be running with PostgreSQL 9.6.
1. **(secondary)** Make sure all services are up:
```shell
sudo gitlab-ctl start
```
1. **(secondary)** Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
1. **(secondary)** Run the PostgreSQL upgrade command:
```shell
sudo gitlab-ctl pg-upgrade
```
1. **(secondary)** See the stored credentials for the database that you will
need to re-initialize the replication:
```shell
sudo grep -s primary_conninfo /var/opt/gitlab/recovery.conf
```
1. **(secondary)** Save the snippet below in a file, let's say `/tmp/replica.sh`. Modify the
embedded paths if necessary:
```shell
#!/bin/bash
PORT="5432"
USER="gitlab_replicator"
echo ---------------------------------------------------------------
echo WARNING: Make sure this script is run from the secondary server
echo ---------------------------------------------------------------
echo
echo Enter the IP or FQDN of the primary PostgreSQL server
read HOST
echo Enter the password for $USER@$HOST
read -s PASSWORD
echo Enter the required sslmode
read SSLMODE
echo Stopping PostgreSQL and all GitLab services
sudo service gitlab stop
sudo service postgresql stop
echo Backing up postgresql.conf
sudo -u postgres mv /var/opt/gitlab/postgresql/data/postgresql.conf /var/opt/gitlab/postgresql/
echo Cleaning up old cluster directory
sudo -u postgres rm -rf /var/opt/gitlab/postgresql/data
echo Starting base backup as the replicator user
echo Enter the password for $USER@$HOST
sudo -u postgres /opt/gitlab/embedded/bin/pg_basebackup -h $HOST -D /var/opt/gitlab/postgresql/data -U gitlab_replicator -v -x -P
echo Writing recovery.conf file
sudo -u postgres bash -c "cat > /var/opt/gitlab/postgresql/data/recovery.conf <<- _EOF1_
standby_mode = 'on'
primary_conninfo = 'host=$HOST port=$PORT user=$USER password=$PASSWORD sslmode=$SSLMODE'
_EOF1_
"
echo Restoring postgresql.conf
sudo -u postgres mv /var/opt/gitlab/postgresql/postgresql.conf /var/opt/gitlab/postgresql/data/
echo Starting PostgreSQL
sudo service postgresql start
```
1. **(secondary)** Run the recovery script using the credentials from the
previous step:
```shell
sudo bash /tmp/replica.sh
```
1. **(secondary)** Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
1. **(secondary)** Start all services:
```shell
sudo gitlab-ctl start
```
1. **(secondary)** Repeat the steps for the remaining **secondary** nodes.
1. **(primary)** After all **secondary** nodes are updated, start all services in
**primary** node:
```shell
sudo gitlab-ctl start
```
### Update tracking database on **secondary** node
After updating a **secondary** node, you might need to run migrations on the
tracking database. The tracking database was added in GitLab 9.1, and is
required in GitLab 10.0 and later.
1. Run database migrations on tracking database:
```shell
sudo gitlab-rake geo:db:migrate
```
1. Repeat this step for each **secondary** node.
|