1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
|
---
stage: Verify
group: Pipeline Authoring
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
type: reference
---
# Pipeline architecture **(FREE ALL)**
Pipelines are the fundamental building blocks for CI/CD in GitLab. This page documents
some of the important concepts related to them.
You can structure your pipelines with different methods, each with their
own advantages. These methods can be mixed and matched if needed:
- [Basic](#basic-pipelines): Good for straightforward projects where all the configuration is in one place.
- [Directed Acyclic Graph](#directed-acyclic-graph-pipelines): Good for large, complex projects that need efficient execution.
- [Parent-child pipelines](#parent-child-pipelines): Good for monorepos and projects with lots of independently defined components.
<i class="fa fa-youtube-play youtube" aria-hidden="true"></i>
For an overview, see the [Parent-Child Pipelines feature demo](https://youtu.be/n8KpBSqZNbk).
- [Multi-project pipelines](downstream_pipelines.md#multi-project-pipelines): Good for larger products that require cross-project interdependencies,
like those with a [microservices architecture](https://about.gitlab.com/blog/2016/08/16/trends-in-version-control-land-microservices/).
For example, you might deploy your web application from three different GitLab projects.
With multi-project pipelines you can trigger a pipeline in each project, where each
has its own build, test, and deploy process. You can visualize the connected pipelines
in one place, including all cross-project interdependencies.
<i class="fa fa-youtube-play youtube" aria-hidden="true"></i>
For an overview, see the [Multi-project pipelines demo](https://www.youtube.com/watch?v=g_PIwBM1J84).
## Basic pipelines
Basic pipelines are the simplest pipelines in GitLab. It runs everything in the build stage concurrently,
and once all of those finish, it runs everything in the test and subsequent stages the same way.
It's not the most efficient, and if you have lots of steps it can grow quite complex, but it's
easier to maintain:
```mermaid
graph LR
subgraph deploy stage
deploy --> deploy_a
deploy --> deploy_b
end
subgraph test stage
test --> test_a
test --> test_b
end
subgraph build stage
build --> build_a
build --> build_b
end
build_a -.-> test
build_b -.-> test
test_a -.-> deploy
test_b -.-> deploy
```
Example basic `/.gitlab-ci.yml` pipeline configuration matching the diagram:
```yaml
stages:
- build
- test
- deploy
default:
image: alpine
build_a:
stage: build
script:
- echo "This job builds something."
build_b:
stage: build
script:
- echo "This job builds something else."
test_a:
stage: test
script:
- echo "This job tests something. It will only run when all jobs in the"
- echo "build stage are complete."
test_b:
stage: test
script:
- echo "This job tests something else. It will only run when all jobs in the"
- echo "build stage are complete too. It will start at about the same time as test_a."
deploy_a:
stage: deploy
script:
- echo "This job deploys something. It will only run when all jobs in the"
- echo "test stage complete."
environment: production
deploy_b:
stage: deploy
script:
- echo "This job deploys something else. It will only run when all jobs in the"
- echo "test stage complete. It will start at about the same time as deploy_a."
environment: production
```
## Directed Acyclic Graph Pipelines
If efficiency is important to you and you want everything to run as quickly as possible,
you can use [Directed Acyclic Graphs (DAG)](../directed_acyclic_graph/index.md). Use the
[`needs` keyword](../yaml/index.md#needs) to define dependency relationships between
your jobs. When GitLab knows the relationships between your jobs, it can run everything
as fast as possible, and even skips into subsequent stages when possible.
In the example below, if `build_a` and `test_a` are much faster than `build_b` and
`test_b`, GitLab starts `deploy_a` even if `build_b` is still running.
```mermaid
graph LR
subgraph Pipeline using DAG
build_a --> test_a --> deploy_a
build_b --> test_b --> deploy_b
end
```
Example DAG `/.gitlab-ci.yml` configuration matching the diagram:
```yaml
stages:
- build
- test
- deploy
default:
image: alpine
build_a:
stage: build
script:
- echo "This job builds something quickly."
build_b:
stage: build
script:
- echo "This job builds something else slowly."
test_a:
stage: test
needs: [build_a]
script:
- echo "This test job will start as soon as build_a finishes."
- echo "It will not wait for build_b, or other jobs in the build stage, to finish."
test_b:
stage: test
needs: [build_b]
script:
- echo "This test job will start as soon as build_b finishes."
- echo "It will not wait for other jobs in the build stage to finish."
deploy_a:
stage: deploy
needs: [test_a]
script:
- echo "Since build_a and test_a run quickly, this deploy job can run much earlier."
- echo "It does not need to wait for build_b or test_b."
environment: production
deploy_b:
stage: deploy
needs: [test_b]
script:
- echo "Since build_b and test_b run slowly, this deploy job will run much later."
environment: production
```
## Parent-child pipelines
As pipelines grow more complex, a few related problems start to emerge:
- The staged structure, where all steps in a stage must complete before the first
job in next stage begins, causes waits that slow things down.
- Configuration for the single global pipeline becomes
hard to manage.
- Imports with [`include`](../yaml/index.md#include) increase the complexity of the configuration, and can cause
namespace collisions where jobs are unintentionally duplicated.
- Pipeline UX has too many jobs and stages to work with.
Additionally, sometimes the behavior of a pipeline needs to be more dynamic. The ability
to choose to start sub-pipelines (or not) is a powerful ability, especially if the
YAML is dynamically generated.
![Parent pipeline graph expanded](img/parent_pipeline_graph_expanded_v14_3.png)
In the [basic pipeline](#basic-pipelines) and [directed acyclic graph](#directed-acyclic-graph-pipelines)
examples above, there are two packages that could be built independently.
These cases are ideal for using [parent-child pipelines](downstream_pipelines.md#parent-child-pipelines).
It separates out the configuration into multiple files, keeping things simpler.
You can combine parent-child pipelines with:
- The [`rules` keyword](../yaml/index.md#rules): For example, have the child pipelines triggered only
when there are changes to that area.
- The [`include` keyword](../yaml/index.md#include): Bring in common behaviors, ensuring
you are not repeating yourself.
- [DAG pipelines](#directed-acyclic-graph-pipelines) inside of child pipelines, achieving the benefits of both.
```mermaid
graph LR
subgraph Parent pipeline
trigger_a -.-> build_a
trigger_b -.-> build_b
subgraph child pipeline B
build_b --> test_b --> deploy_b
end
subgraph child pipeline A
build_a --> test_a --> deploy_a
end
end
```
Example `/.gitlab-ci.yml` configuration for the parent pipeline matching the diagram:
```yaml
stages:
- triggers
trigger_a:
stage: triggers
trigger:
include: a/.gitlab-ci.yml
rules:
- changes:
- a/*
trigger_b:
stage: triggers
trigger:
include: b/.gitlab-ci.yml
rules:
- changes:
- b/*
```
Example child `a` pipeline configuration, located in `/a/.gitlab-ci.yml`, making
use of the DAG `needs` keyword:
```yaml
stages:
- build
- test
- deploy
default:
image: alpine
build_a:
stage: build
script:
- echo "This job builds something."
test_a:
stage: test
needs: [build_a]
script:
- echo "This job tests something."
deploy_a:
stage: deploy
needs: [test_a]
script:
- echo "This job deploys something."
environment: production
```
Example child `b` pipeline configuration, located in `/b/.gitlab-ci.yml`, making
use of the DAG `needs` keyword:
```yaml
stages:
- build
- test
- deploy
default:
image: alpine
build_b:
stage: build
script:
- echo "This job builds something else."
test_b:
stage: test
needs: [build_b]
script:
- echo "This job tests something else."
deploy_b:
stage: deploy
needs: [test_b]
script:
- echo "This job deploys something else."
environment: production
```
Jobs can be set to run before or after triggering child pipelines in GitLab, allowing common setup steps or unified deployment.
|