1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
|
# Gitaly High Availability (HA) Design
Gitaly HA is an active-active cluster configuration for resilient git operations. [Refer to our specific requirements](https://gitlab.com/gitlab-org/gitaly/issues/1332).
Refer to [epic &289][epic] for current issues and discussions revolving around
HA MVC development.
## Terminology
The following terminology may be used within the context of the Gitaly HA project:
- Shard - partition of the storage for all repos. Each shard will require redundancy in the form of multiple Gitaly nodes (at least 3 when optimal) to maintain HA.
- Praefect - a transparent front end to all Gitaly shards. This reverse proxy ensures that all gRPC calls are forwarded to the correct shard by consulting the coordinator. The reverse proxy also ensures that write actions are performed transactionally when needed.
- etymology: from Latin praefectus for _a person appointed to any of various positions of command, authority, or superintendence, as a chief magistrate in ancient Rome or the chief administrative official of a department of France or Italy._
- [pronounced _pree-fect_](https://www.youtube.com/watch?v=MHszCZjPmTQ)
- Node (TODO: we probably need a similar latin name here) - performs the actual git read/write operations to/from disk. Has no knowledge of shards/prafects/coordinators just as the Gitaly service existed prior to HA.
- RPC categories (#1496):
- Accessor - a side effect free (or read-only) RPC; does not modify the git repo (!228)
- Mutator - an RPC that modifies the data in the git repo (!228)
## Design
The high level design takes a reverse proxy approach to fanning out write requests to the appropriate nodes:
<img src="https://docs.google.com/drawings/d/e/2PACX-1vRl7WS-6RBOWxyLSBbBBAoV9MupmTh5vTqMOw_AX9axlboqkybTbFqGqExLyyYOilqEW7S9euXdBHzX/pub?w=960&h=720">
## Phases
An iterative low risk approach needs to be devised to add functionality and verify assumptions at a sustainable pace while not impeding the existing functionality.
### 1. Simple pass-through proxy - no added functionality
- allows us to set up telemetry for observability of new service
- allows us to evaluate a gRPC proxy library
### 2. Introduce State
The following details need to be persisted in Postgres:
- [x] Primary location for a project
- [ ] Redundant locations for a project
- [ ] Available storage locations (initially can be configuration file)
Initially, the state of the shard nodes will be static and loaded from a configuration file. Eventually, this will be made dynamic via a data store (Postgres).
### Resolving Location
The following existing interaction will remain intact for the first iteration of the HA feature:
```mermaid
sequenceDiagram
Client->>Rails: Modify repo X
Rails-->>Datastore: Where is Repo X?
Datastore-->> Rails: Repo X is at location A
Rails-->>Gitaly: Modify repo X at location A
Gitaly-->>Rails: Request succeeded/failed
```
Once the Rails app has resolved the primary location for the project, the request is made to the praefect. The praefect then resolves the redundant locations via the coordinator before applying the changes.
```mermaid
sequenceDiagram
Rails->>Praefect: Modify repo X at A
Praefect->>Coordinator: Which locations complement A for X?
Coordinator->>Praefect: Locations B and C complement A
Praefect->>Nodes ABC: Modify repo X
Nodes ABC->>Praefect: Modifications successful!
```
*Note: the above interaction between the praefect and nodes A-B-C is an all-or-nothing transaction. All nodes must complete in success, otherwise a single node failure will cause the entire transaction to fail. This will be improved when replication is introduced.*
### 3. Replication
The next phase is to enable replication of data between nodes. This makes transactions more efficient and fault tolerant. This could be done a few ways:
#### Node Orchestrated [👎]
Node orchestrated puts the intelligence of replication into one of the nodes being modified:
```mermaid
sequenceDiagram
Praefect->>Node A: Modify repo X
activate Node A
Node A->>Node B: Modify repo X
Node A->>Node C: Modify repo X
Node A->>Praefect: Modification successful!
```
Orchestration requires designating a leader node for the transaction. This leader node becomes a critical path for all nodes involved. Ideally, we want several simpler (less riskier) operations that can succeed/fail independently of each other. This way, failure and recovery can be handled externally of the nodes.
#### Praefect Orchestrated [👍]
With the praefect orchestrating replication, we are isolating the critical path to a stateless service. Stateless services are preferred for the critical path since another praefect can pick up the task after a praefect failure.
```mermaid
sequenceDiagram
Praefect->>Node A: Modify repo X
Node A->>Praefect: Success!
Praefect->>Node B: Replicate From A
Praefect->>Node C: Replicate From A
Node B->>Praefect: Success!
Node C->>Praefect: Success!
```
*Note: Once Node-A propagates changes to a peer, Node-A is no longer the critical path for subsequent propagations. If Node-A fails after a second peer is propagated, that second peer can become the new leader and resume replications.*
##### Replication Logic
Here are the steps during a Gitaly client GRPC call intercepted by Praefect:
```mermaid
graph TD
A[Gitaly Client]-->B{Does RPC Mutate a repository?}
B-->| yes | C[Peek into RPC Stream to determine Repository]
B-->| no | G[Forward request to Gitaly]
C-->D{Scoped for repository?}
D-->| yes | E[Get target repository from message]
D-->| no | G
E-->F[Schedule Replication]
F-->G
```
## Stages until v1.0
Rome wasn't built in a day, nor will Praefect be build in one. To iterate towards
a true HA system some properties will not be met until v1.0. Before that
milestone, an beta stage will worked towards.
The beta stage will consist of only a few required building blocks to iterate
on towards the envisioned HA system. The first of those building blocks is
creating and maintaining repository replica's. By maintaining a replica, there's
no requirement for the replica to be up to date right after each mutation on the
repository. Detecting that a repository is mutated, and bringing replicas up to
date in a consistent matter is the primary goal. Implicit in this goal is a way
to perform leader election.
When the beta nears completion further stages will be defined.
## Notes
* Existing discussions
* Requirements: https://gitlab.com/gitlab-org/gitaly/issues/1332
* Design: https://gitlab.com/gitlab-org/gitaly/issues/1335
* Prior art
* Stemma by Palantir
* [Announcement](https://medium.com/palantir/stemma-distributed-git-server-70afbca0fc29)
* Extends jgit (java git implementation)
* Spokes by GitHub
* Application layer approach: uses underlying git software to propagate changes to other locations.
* Bitbucket Data Center (BDC)
* [BDC FAQ](https://confluence.atlassian.com/enterprise/bitbucket-data-center-faq-776663707.html)
* Ketch by Google (no longer maintained)
* [Sid's comment on performance issue](https://news.ycombinator.com/item?id=13934698)
* Also jgit based
* gRPC proxy considerations
* [gRPC Proxy library](https://github.com/mwitkow/grpc-proxy)
* Pros
* Handles all gRPC requests generically
* Cons
* Lack of support
* [See current importers of project](https://godoc.org/github.com/mwitkow/grpc-proxy/proxy?importers)
* Low level implementation requires knowledge of gRPC internals
* Custom code generation
* Pros
* Simple and maintainable
* Allows us to handwrite proxy code and later automate with lessons learned via code generation
* Cons
* Process heavy; requires custom tooling
* Requires a way to tell which methods are read/write
* [See WIP for marking modifying RPCs](https://gitlab.com/gitlab-org/gitaly-proto/merge_requests/228)
* See also:
* [nRPC](https://github.com/nats-rpc/nrpc) - gRPC via NATS
* [grpclb](https://github.com/bsm/grpclb) - gRPC load balancer
* Complications
* Existing Rails app indicates the Gitaly instance that a request is destined for (e.g. request to modify repo X should be directed to gitaly #1).
* This means that rails app must be kept in the loop about any changes made to the location of a repo.
* This may be mitigated by changing the proxy implementation to intepret the destination address as a reference to a shard rather than a specific host. This might open the door to allowing for something like consistent hashing.
* While Git is distributed in nature, some write operations need to be serialized to avoid race conditions. This includes ref updates.
* How do we coordinate proxies when applying ref updates? Do we need to?
[epic]: https://gitlab.com/groups/gitlab-org/-/epics/289
|
