1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
|
---
stage: Verify
group: Pipeline Execution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Test coverage visualization **(FREE)**
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3708) in GitLab 12.9.
> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/249811) in GitLab 13.5.
With the help of [GitLab CI/CD](../index.md), you can collect the test
coverage information of your favorite testing or coverage-analysis tool, and visualize
this information inside the file diff view of your merge requests (MRs). This allows you
to see which lines are covered by tests, and which lines still require coverage, before the
MR is merged.
![Test Coverage Visualization Diff View](img/test_coverage_visualization_v12_9.png)
## How test coverage visualization works
Collecting the coverage information is done via GitLab CI/CD's
[artifacts reports feature](../yaml/index.md#artifactsreports).
You can specify one or more coverage reports to collect, including wildcard paths.
GitLab then takes the coverage information in all the files and combines it
together. Coverage files are parsed in a background job so there can be a delay
between pipeline completion and the visualization loading on the page.
For the coverage analysis to work, you have to provide a properly formatted
[Cobertura XML](https://cobertura.github.io/cobertura/) report to
[`artifacts:reports:coverage_report`](../yaml/artifacts_reports.md#artifactsreportscoverage_report).
This format was originally developed for Java, but most coverage analysis frameworks
for other languages have plugins to add support for it, like:
- [simplecov-cobertura](https://rubygems.org/gems/simplecov-cobertura) (Ruby)
- [gocover-cobertura](https://github.com/boumenot/gocover-cobertura) (Go)
Other coverage analysis frameworks support the format out of the box, for example:
- [Istanbul](https://istanbul.js.org/docs/advanced/alternative-reporters/#cobertura) (JavaScript)
- [Coverage.py](https://coverage.readthedocs.io/en/coverage-5.0.4/cmd.html#xml-reporting) (Python)
- [PHPUnit](https://github.com/sebastianbergmann/phpunit-documentation-english/blob/master/src/textui.rst#command-line-options) (PHP)
Once configured, if you create a merge request that triggers a pipeline which collects
coverage reports, the coverage is shown in the diff view. This includes reports
from any job in any stage in the pipeline. The coverage displays for each line:
- `covered` (green): lines which have been checked at least once by tests
- `no test coverage` (orange): lines which are loaded but never executed
- no coverage information: lines which are non-instrumented or not loaded
Hovering over the coverage bar provides further information, such as the number
of times the line was checked by tests.
Uploading a test coverage report does not enable:
- [Test coverage results in merge requests](code_coverage.md#view-code-coverage-results-in-the-mr).
- [Code coverage history](code_coverage.md#view-history-of-project-code-coverage).
You must configure these separately.
### Limits
A limit of 100 `<source>` nodes for Cobertura format XML files applies. If your Cobertura report exceeds
100 nodes, there can be mismatches or no matches in the merge request diff view.
A single Cobertura XML file can be no more than 10 MiB. For large projects, split the Cobertura XML into
smaller files. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/328772) for more details.
When submitting many files, it can take a few minutes for coverage to show on a merge request.
The visualization only displays after the pipeline is complete. If the pipeline has
a [blocking manual job](../jobs/job_control.md#types-of-manual-jobs), the
pipeline waits for the manual job before continuing and is not considered complete.
The visualization cannot be displayed if the blocking manual job did not run.
### Data expiration
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/321323) in GitLab 13.12, the latest data is kept regardless of expiry time.
By default, the data used to draw the visualization on the merge request expires **one week** after creation.
### Coverage report from child pipeline
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/363301) in GitLab 15.1 [with a flag](../../administration/feature_flags.md) named `ci_child_pipeline_coverage_reports`. Disabled by default.
> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/363557) and feature flag `ci_child_pipeline_coverage_reports` removed in GitLab 15.2.
If a job in a child pipeline creates a coverage report, the report is included in
the parent pipeline's coverage report.
```yaml
child_test_pipeline:
trigger:
include:
- local: path/to/child_pipeline_with_coverage.yml
```
### Automatic class path correction
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217664) in GitLab 13.8.
> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/284822) in GitLab 13.9.
The coverage report properly matches changed files only if the `filename` of a `class` element
contains the full path relative to the project root. However, in some coverage analysis frameworks,
the generated Cobertura XML has the `filename` path relative to the class package directory instead.
To make an intelligent guess on the project root relative `class` path, the Cobertura XML parser
attempts to build the full path by:
- Extracting a portion of the `source` paths from the `sources` element and combining them with the
class `filename` path.
- Checking if the candidate path exists in the project.
- Using the first candidate that matches as the class full path.
#### Path correction example
As an example, a C# project with:
- A full path of `test-org/test-cs-project`.
- The following files relative to the project root:
```shell
Auth/User.cs
Lib/Utils/User.cs
```
- `sources` from Cobertura XML, the following paths in the format
`<CI_BUILDS_DIR>/<PROJECT_FULL_PATH>/...`:
```xml
<sources>
<source>/builds/test-org/test-cs-project/Auth</source>
<source>/builds/test-org/test-cs-project/Lib/Utils</source>
</sources>
```
The parser:
- Extracts `Auth` and `Lib/Utils` from the `sources` and uses these to determine the `class` path
relative to the project root.
- Combines these extracted `sources` and the class filename. For example, if there is a `class`
element with the `filename` value of `User.cs`, the parser takes the first candidate path that
matches, which is `Auth/User.cs`.
- For each `class` element, attempts to look for a match for each extracted `source` path up to
100 iterations. If it reaches this limit without finding a matching path in the file tree, the
class is not included in the final coverage report.
Automatic class path correction also works for a Java project with:
- A full path of `test-org/test-java-project`.
- The following files relative to the project root:
```shell
src/main/java/com/gitlab/security_products/tests/App.java
```
- `sources` from Cobertura XML:
```xml
<sources>
<source>/builds/test-org/test-java-project/src/main/java/</source>
</sources>
```
- `class` element with the `filename` value of `com/gitlab/security_products/tests/App.java`:
```xml
<class name="com.gitlab.security_products.tests.App" filename="com/gitlab/security_products/tests/App.java" line-rate="0.0" branch-rate="0.0" complexity="6.0">
```
NOTE:
Automatic class path correction only works on `source` paths in the format `<CI_BUILDS_DIR>/<PROJECT_FULL_PATH>/...`.
The `source` is ignored if the path does not follow this pattern. The parser assumes that the
`filename` of a `class` element contains the full path relative to the project root.
## Example test coverage configurations
This section provides test coverage configuration examples for different programming languages. You can also see a working example in
the [`coverage-report`](https://gitlab.com/gitlab-org/ci-sample-projects/coverage-report/) demonstration project.
### JavaScript example
The following [`.gitlab-ci.yml`](../yaml/index.md) example uses [Mocha](https://mochajs.org/)
JavaScript testing and [nyc](https://github.com/istanbuljs/nyc) coverage-tooling to
generate the coverage artifact:
```yaml
test:
script:
- npm install
- npx nyc --reporter cobertura mocha
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: coverage/cobertura-coverage.xml
```
### Java and Kotlin examples
#### Maven example
The following [`.gitlab-ci.yml`](../yaml/index.md) example for Java or Kotlin uses [Maven](https://maven.apache.org/)
to build the project and [JaCoCo](https://www.eclemma.org/jacoco/) coverage-tooling to
generate the coverage artifact.
You can check the [Docker image configuration and scripts](https://gitlab.com/haynes/jacoco2cobertura) if you want to build your own image.
GitLab expects the artifact in the Cobertura format, so you have to execute a few
scripts before uploading it. The `test-jdk11` job tests the code and generates an
XML artifact. The `coverage-jdk-11` job converts the artifact into a Cobertura report:
```yaml
test-jdk11:
stage: test
image: maven:3.6.3-jdk-11
script:
- mvn $MAVEN_CLI_OPTS clean org.jacoco:jacoco-maven-plugin:prepare-agent test jacoco:report
artifacts:
paths:
- target/site/jacoco/jacoco.xml
coverage-jdk11:
# Must be in a stage later than test-jdk11's stage.
# The `visualize` stage does not exist by default.
# Please define it first, or choose an existing stage like `deploy`.
stage: visualize
image: registry.gitlab.com/haynes/jacoco2cobertura:1.0.7
script:
# convert report from jacoco to cobertura, using relative project path
- python /opt/cover2cover.py target/site/jacoco/jacoco.xml $CI_PROJECT_DIR/src/main/java/ > target/site/cobertura.xml
needs: ["test-jdk11"]
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: target/site/cobertura.xml
```
#### Gradle example
The following [`.gitlab-ci.yml`](../yaml/index.md) example for Java or Kotlin uses [Gradle](https://gradle.org/)
to build the project and [JaCoCo](https://www.eclemma.org/jacoco/) coverage-tooling to
generate the coverage artifact.
You can check the [Docker image configuration and scripts](https://gitlab.com/haynes/jacoco2cobertura) if you want to build your own image.
GitLab expects the artifact in the Cobertura format, so you have to execute a few
scripts before uploading it. The `test-jdk11` job tests the code and generates an
XML artifact. The `coverage-jdk-11` job converts the artifact into a Cobertura report:
```yaml
test-jdk11:
stage: test
image: gradle:6.6.1-jdk11
script:
- 'gradle test jacocoTestReport' # jacoco must be configured to create an xml report
artifacts:
paths:
- build/jacoco/jacoco.xml
coverage-jdk11:
# Must be in a stage later than test-jdk11's stage.
# The `visualize` stage does not exist by default.
# Please define it first, or chose an existing stage like `deploy`.
stage: visualize
image: registry.gitlab.com/haynes/jacoco2cobertura:1.0.7
script:
# convert report from jacoco to cobertura, using relative project path
- python /opt/cover2cover.py build/jacoco/jacoco.xml $CI_PROJECT_DIR/src/main/java/ > build/cobertura.xml
needs: ["test-jdk11"]
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: build/cobertura.xml
```
### Python example
The following [`.gitlab-ci.yml`](../yaml/index.md) example uses [pytest-cov](https://pytest-cov.readthedocs.io/) to collect test coverage data:
```yaml
run tests:
stage: test
image: python:3
script:
- pip install pytest pytest-cov
- pytest --cov --cov-report term --cov-report xml:coverage.xml
coverage: '/(?i)total.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/'
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: coverage.xml
```
### PHP example
The following [`.gitlab-ci.yml`](../yaml/index.md) example for PHP uses [PHPUnit](https://phpunit.readthedocs.io/)
to collect test coverage data and generate the report.
With a minimal [`phpunit.xml`](https://docs.phpunit.de/en/10.2/configuration.html) file (you may reference
[this example repository](https://gitlab.com/yookoala/code-coverage-visualization-with-php/)), you can run the test and
generate the `coverage.xml`:
```yaml
run tests:
stage: test
image: php:latest
variables:
XDEBUG_MODE: coverage
before_script:
- apt-get update && apt-get -yq install git unzip zip libzip-dev zlib1g-dev
- docker-php-ext-install zip
- pecl install xdebug && docker-php-ext-enable xdebug
- php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"
- php composer-setup.php --install-dir=/usr/local/bin --filename=composer
- composer install
- composer require --dev phpunit/phpunit phpunit/php-code-coverage
script:
- php ./vendor/bin/phpunit --coverage-text --coverage-cobertura=coverage.cobertura.xml
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: coverage.cobertura.xml
```
[Codeception](https://codeception.com/), through PHPUnit, also supports generating Cobertura report with
[`run`](https://codeception.com/docs/reference/Commands#run). The path for the generated file
depends on the `--coverage-cobertura` option and [`paths`](https://codeception.com/docs/reference/Configuration#paths)
configuration for the [unit test suite](https://codeception.com/docs/05-UnitTests). Configure `.gitlab-ci.yml`
to find Cobertura in the appropriate path.
### C/C++ example
The following [`.gitlab-ci.yml`](../yaml/index.md) example for C/C++ with
`gcc` or `g++` as the compiler uses [`gcovr`](https://gcovr.com/en/stable/) to generate the coverage
output file in Cobertura XML format.
This example assumes:
- That the `Makefile` is created by `cmake` in the `build` directory,
within another job in a previous stage.
(If you use `automake` to generate the `Makefile`,
then you need to call `make check` instead of `make test`.)
- `cmake` (or `automake`) has set the compiler option `--coverage`.
```yaml
run tests:
stage: test
script:
- cd build
- make test
- gcovr --xml-pretty --exclude-unreachable-branches --print-summary -o coverage.xml --root ${CI_PROJECT_DIR}
coverage: /^\s*lines:\s*\d+.\d+\%/
artifacts:
name: ${CI_JOB_NAME}-${CI_COMMIT_REF_NAME}-${CI_COMMIT_SHA}
expire_in: 2 days
reports:
coverage_report:
coverage_format: cobertura
path: build/coverage.xml
```
### Go example
The following [`.gitlab-ci.yml`](../yaml/index.md) example for Go uses:
- [`go test`](https://go.dev/doc/tutorial/add-a-test) to run tests.
- [`gocover-cobertura`](https://github.com/boumenot/gocover-cobertura) to convert Go's coverage profile into the Cobertura XML format.
This example assumes that [Go modules](https://go.dev/ref/mod)
are being used. The `-covermode count` option does not work with the `-race` flag.
If you want to generate code coverage while also using the `-race` flag, you must switch to
`-covermode atomic` which is slower than `-covermode count`. See [this blog post](https://go.dev/blog/cover)
for more details.
```yaml
run tests:
stage: test
image: golang:1.17
script:
- go install
- go test ./... -coverprofile=coverage.txt -covermode count
- go get github.com/boumenot/gocover-cobertura
- go run github.com/boumenot/gocover-cobertura < coverage.txt > coverage.xml
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: coverage.xml
```
### Ruby example
The following [`.gitlab-ci.yml`](../yaml/index.md) example for Ruby uses
- [`rspec`](https://rspec.info/) to run tests.
- [`simplecov`](https://github.com/simplecov-ruby/simplecov) and [`simplecov-cobertura`](https://github.com/dashingrocket/simplecov-cobertura)
to record the coverage profile and create a report in the Cobertura XML format.
This example assumes:
- That [`bundler`](https://bundler.io/) is being used for dependency management.
The `rspec`, `simplecov` and `simplecov-cobertura` gems have been added to your `Gemfile`.
- The `CoberturaFormatter` has been added to your `SimpleCov.formatters`
configuration within the `spec_helper.rb` file.
```yaml
run tests:
stage: test
image: ruby:3.1
script:
- bundle install
- bundle exec rspec
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: coverage/coverage.xml
```
## Troubleshooting
### Test coverage visualization not displayed
If the test coverage visualization is not displayed in the diff view, you can check
the coverage report itself and verify that:
- The file you are viewing in the diff view is mentioned in the coverage report.
- The `source` and `filename` nodes in the report follows the [expected structure](#automatic-class-path-correction)
to match the files in your repository.
- The pipeline has completed. If the pipeline is [blocked on a manual job](../jobs/job_control.md#types-of-manual-jobs),
the pipeline is not considered complete.
- The coverage report file does not exceed the [limits](#limits).
Report artifacts are not downloadable by default. If you want the report to be downloadable
from the job details page, add your coverage report to the artifact `paths`:
```yaml
artifacts:
paths:
- coverage/cobertura-coverage.xml
reports:
coverage_report:
coverage_format: cobertura
path: coverage/cobertura-coverage.xml
```
|