1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
|
# `::Gitlab::Ci::Config` module overview
`::Gitlab::Ci::Config` is a concrete implementation of abstract
`::Gitlab::Config` module. It's being used to build, traverse and translate
hierarchical, user-provided, CI configuration, usually provided in
`.gitlab-ci.yml` and included files.
## High-level Overview
`::Gitlab::Ci::Config` is an indirection layer between user-provided data and
GitLab itself.
1. A user provides YAML configuration in `.gitlab-ci.yml` and all included files.
1. `::Gitlab::Ci::Config` loads the provided YAML using Ruby standard `Psych` library.
1. The resulting Hash is then passed to the module to build an Abstract Syntax Tree.
1. The module validates, transforms, translates and augments the data to build
a stable representation of user-provided configuration.
This additional layer helps us to validate the user-provided configuration and
surface any errors to a user if it is not valid. In case of a valid
configuration, it makes it possible to build a stable representation of
config that we can depend on.
For example, both following configurations using the
[environment](https://docs.gitlab.com/ee/ci/yaml/#environment)
keyword are correct:
```yaml
# First way to define an environment:
deploy:
environment: production
script: cap deploy
# Second way to define an environment:
deploy:
environment:
name: production
url: https://prod.example.com
kubernetes:
namespace: production
```
This demonstrates the concept of hidden / expanding complexity: if users need
more flexibility, they can opt-in into using a much more elaborate syntax to
configure their environments. **We use this technique to make it possible for
simplicity to coexist with flexibility without additional complexity**.
`::Gitlab::Ci::Config` allows us to achieve this, because it is an indirection
layer, that translates user-provided configuration into a known and expected
format when users can achieve the same thing in `.gitlab-ci.yml` in a few
different ways.
## Hierarchical configuration
`.gitlab-ci.yml` configuration is hierarchical but same keywords can often be
used on different levels in the hierarchy. `::Gitlab::Ci::Config` module makes
it easier to manage the complexity that stems from having same keyword
available in [many different places](https://docs.gitlab.com/ee/ci/yaml/#default):
```yaml
default:
image: ruby:3.0
rspec:
script: bundle exec rspec
rspec 2.7:
image: ruby:2.7
script: bundle exec rspec
```
We can achieve that, because in `::Gitlab::Ci::Config` most of the keywords are
implemented within separate Ruby classes, that then can be reused:
```ruby
# Simplified version of an entry class that describes a Docker image.
#
class Gitlab::Ci::Config::Entry
class Image < ::Gitlab::Config::Entry::Node
validates :config, allowed_keys: ALLOWED_IMAGE_CONFIG_KEYS
def value
if string?
{ name: @config }
elsif hash?
{
name: @config[:name],
entrypoint: @config[:entrypoint],
ports: (ports_value if ports_defined?),
pull_policy: pull_policy_value
}
else
{}
end
end
end
end
```
The config above is a simple demonstration of the translation layer, into a
stable configuration, depending on what simplification strategy has been used
by a user. There more complex examples, though:
```ruby
module Gitlab::Ci::Config::Entry
class Need < ::Gitlab::Config::Entry::Simplifiable
strategy :JobString, if: -> (config) { config.is_a?(String) }
strategy :JobHash,
if: -> (config) { config.is_a?(Hash) && same_pipeline_need?(config) }
strategy :CrossPipelineDependency,
if: -> (config) { config.is_a?(Hash) && cross_pipeline_need?(config) }
# [ ... ]
end
end
```
Every time we load config, an Abstract Syntax Tree is being built, because
nodes / entries know what the child nodes can be:
```ruby
# Simplified root entry code
#
module Gitlab::Ci::Config::Entry
class Root < ::Gitlab::Config::Entry::Node
include ::Gitlab::Config::Entry::Configurable
entry :default, Entry::Default,
description: 'Default configuration for all jobs.'
entry :include, Entry::Includes,
description: 'List of external YAML files to include.'
entry :before_script, Entry::Commands,
description: 'Script that will be executed before each job.'
entry :image, Entry::Image,
description: 'Docker image that will be used to execute jobs.'
entry :services, Entry::Services,
description: 'Docker images that will be linked to the container.'
entry :after_script, Entry::Commands,
description: 'Script that will be executed after each job.'
entry :variables, Entry::Variables,
description: 'Environment variables that will be used.'
# [ ... ]
end
end
```
Loading the configuration script mentioned at the beginning of this pargraph
will result in build a following AST:
```
Entry::Root
`-
|- Entry::Default
| `- Entry::Image('ruby:3.0')
|
|- Entry::Job('rspec')
| `- Entry::Script('bundle exec rspec')
|
|- Entry::Job('rspec 2.7')
| |- Entry::Image('ruby:2.7)
| `- Entry::Script('bundle exec rspec')
```
The AST will be validated, and eventually will generate a stable representation
of configuration that we can use to persist pipelines / stages / jobs in the
database, and start pipeline processing.
|
