1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
|
---
stage: Enablement
group: Geo
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
type: howto
---
WARNING:
This runbook is in **alpha**. For complete, production-ready documentation, see the
[disaster recovery documentation](../index.md).
# Disaster Recovery (Geo) promotion runbooks **(PREMIUM ONLY)**
## Geo planned failover for a single-node configuration
| Component | Configuration |
|-------------|-----------------|
| PostgreSQL | Omnibus-managed |
| Geo site | Single-node |
| Secondaries | One |
This runbook will guide you through a planned failover of a single-node Geo site
with one secondary. The following general architecture is assumed:
```mermaid
graph TD
subgraph main[Geo deployment]
subgraph Primary[Primary site]
Node_1[(GitLab node)]
end
subgraph Secondary1[Secondary site]
Node_2[(GitLab node)]
end
end
```
This guide will result in the following:
1. An offline primary.
1. A promoted secondary that is now the new primary.
What is not covered:
1. Re-adding the old **primary** as a secondary.
1. Adding a new secondary.
### Preparation
NOTE:
Before following any of those steps, make sure you have `root` access to the
**secondary** to promote it, since there isn't provided an automated way to
promote a Geo replica and perform a failover.
On the **secondary** node, navigate to the **Admin Area > Geo** dashboard to
review its status. Replicated objects (shown in green) should be close to 100%,
and there should be no failures (shown in red). If a large proportion of
objects aren't yet replicated (shown in gray), consider giving the node more
time to complete.
![Replication status](../img/replication-status.png)
If any objects are failing to replicate, this should be investigated before
scheduling the maintenance window. After a planned failover, anything that
failed to replicate will be **lost**.
You can use the
[Geo status API](../../../../api/geo_nodes.md#retrieve-project-sync-or-verification-failures-that-occurred-on-the-current-node)
to review failed objects and the reasons for failure.
A common cause of replication failures is the data being missing on the
**primary** node - you can resolve these failures by restoring the data from backup,
or removing references to the missing data.
The maintenance window won't end until Geo replication and verification is
completely finished. To keep the window as short as possible, you should
ensure these processes are close to 100% as possible during active use.
If the **secondary** node is still replicating data from the **primary** node,
follow these steps to avoid unnecessary data loss:
1. Until a [read-only mode](https://gitlab.com/gitlab-org/gitlab/-/issues/14609)
is implemented, updates must be prevented from happening manually to the
**primary**. Note that your **secondary** node still needs read-only
access to the **primary** node during the maintenance window:
1. At the scheduled time, using your cloud provider or your node's firewall, block
all HTTP, HTTPS and SSH traffic to/from the **primary** node, **except** for your IP and
the **secondary** node's IP.
For instance, you can run the following commands on the **primary** node:
```shell
sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 22 -j ACCEPT
sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 22 -j ACCEPT
sudo iptables -A INPUT --destination-port 22 -j REJECT
sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 80 -j ACCEPT
sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 80 -j ACCEPT
sudo iptables -A INPUT --tcp-dport 80 -j REJECT
sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 443 -j ACCEPT
sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 443 -j ACCEPT
sudo iptables -A INPUT --tcp-dport 443 -j REJECT
```
From this point, users will be unable to view their data or make changes on the
**primary** node. They will also be unable to log in to the **secondary** node.
However, existing sessions will work for the remainder of the maintenance period, and
public data will be accessible throughout.
1. Verify the **primary** node is blocked to HTTP traffic by visiting it in browser via
another IP. The server should refuse connection.
1. Verify the **primary** node is blocked to Git over SSH traffic by attempting to pull an
existing Git repository with an SSH remote URL. The server should refuse
connection.
1. On the **primary** node, disable non-Geo periodic background jobs by navigating
to **Admin Area > Monitoring > Background Jobs > Cron**, clicking `Disable All`,
and then clicking `Enable` for the `geo_sidekiq_cron_config_worker` cron job.
This job will re-enable several other cron jobs that are essential for planned
failover to complete successfully.
1. Finish replicating and verifying all data:
WARNING:
Not all data is automatically replicated. Read more about
[what is excluded](../planned_failover.md#not-all-data-is-automatically-replicated).
1. If you are manually replicating any
[data not managed by Geo](../../replication/datatypes.md#limitations-on-replicationverification),
trigger the final replication process now.
1. On the **primary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues**
and wait for all queues except those with `geo` in the name to drop to 0.
These queues contain work that has been submitted by your users; failing over
before it is completed will cause the work to be lost.
1. On the **primary** node, navigate to **Admin Area > Geo** and wait for the
following conditions to be true of the **secondary** node you are failing over to:
- All replication meters to each 100% replicated, 0% failures.
- All verification meters reach 100% verified, 0% failures.
- Database replication lag is 0ms.
- The Geo log cursor is up to date (0 events behind).
1. On the **secondary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues**
and wait for all the `geo` queues to drop to 0 queued and 0 running jobs.
1. On the **secondary** node, use [these instructions](../../../raketasks/check.md)
to verify the integrity of CI artifacts, LFS objects, and uploads in file
storage.
At this point, your **secondary** node will contain an up-to-date copy of everything the
**primary** node has, meaning nothing will be lost when you fail over.
1. In this final step, you need to permanently disable the **primary** node.
WARNING:
When the **primary** node goes offline, there may be data saved on the **primary** node
that has not been replicated to the **secondary** node. This data should be treated
as lost if you proceed.
NOTE:
If you plan to [update the **primary** domain DNS record](../index.md#step-4-optional-updating-the-primary-domain-dns-record),
you may wish to lower the TTL now to speed up propagation.
When performing a failover, we want to avoid a split-brain situation where
writes can occur in two different GitLab instances. So to prepare for the
failover, you must disable the **primary** node:
- If you have SSH access to the **primary** node, stop and disable GitLab:
```shell
sudo gitlab-ctl stop
```
Prevent GitLab from starting up again if the server unexpectedly reboots:
```shell
sudo systemctl disable gitlab-runsvdir
```
NOTE:
(**CentOS only**) In CentOS 6 or older, there is no easy way to prevent GitLab from being
started if the machine reboots isn't available (see [Omnibus GitLab issue #3058](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3058)).
It may be safest to uninstall the GitLab package completely with `sudo yum remove gitlab-ee`.
NOTE:
(**Ubuntu 14.04 LTS**) If you are using an older version of Ubuntu
or any other distribution based on the Upstart init system, you can prevent GitLab
from starting if the machine reboots as `root` with
`initctl stop gitlab-runsvvdir && echo 'manual' > /etc/init/gitlab-runsvdir.override && initctl reload-configuration`.
- If you do not have SSH access to the **primary** node, take the machine offline and
prevent it from rebooting. Since there are many ways you may prefer to accomplish
this, we will avoid a single recommendation. You may need to:
- Reconfigure the load balancers.
- Change DNS records (for example, point the **primary** DNS record to the
**secondary** node to stop using the **primary** node).
- Stop the virtual servers.
- Block traffic through a firewall.
- Revoke object storage permissions from the **primary** node.
- Physically disconnect a machine.
### Promoting the **secondary** node
Note the following when promoting a secondary:
- A new **secondary** should not be added at this time. If you want to add a new
**secondary**, do this after you have completed the entire process of promoting
the **secondary** to the **primary**.
- If you encounter an `ActiveRecord::RecordInvalid: Validation failed: Name has already been taken`
error during this process, read
[the troubleshooting advice](../../replication/troubleshooting.md#fixing-errors-during-a-failover-or-when-promoting-a-secondary-to-a-primary-node).
To promote the secondary node:
1. SSH in to your **secondary** node and login as root:
```shell
sudo -i
```
1. Edit `/etc/gitlab/gitlab.rb` to reflect its new status as **primary** by
removing any lines that enabled the `geo_secondary_role`:
```ruby
## In pre-11.5 documentation, the role was enabled as follows. Remove this line.
geo_secondary_role['enable'] = true
## In 11.5+ documentation, the role was enabled as follows. Remove this line.
roles ['geo_secondary_role']
```
1. Run the following command to list out all preflight checks and automatically
check if replication and verification are complete before scheduling a planned
failover to ensure the process will go smoothly:
```shell
gitlab-ctl promotion-preflight-checks
```
1. Promote the **secondary**:
```shell
gitlab-ctl promote-to-primary-node
```
If you have already run the [preflight checks](../planned_failover.md#preflight-checks)
or don't want to run them, you can skip them:
```shell
gitlab-ctl promote-to-primary-node --skip-preflight-check
```
You can also promote the secondary node to primary **without any further confirmation**, even when preflight checks fail:
```shell
sudo gitlab-ctl promote-to-primary-node --force
```
1. Verify you can connect to the newly promoted **primary** node using the URL used
previously for the **secondary** node.
If successful, the **secondary** node has now been promoted to the **primary** node.
### Next steps
To regain geographic redundancy as quickly as possible, you should
[add a new **secondary** node](../../setup/index.md). To
do that, you can re-add the old **primary** as a new secondary and bring it back
online.
|