diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2023-04-12 18:15:44 +0300 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2023-04-12 18:15:44 +0300 |
commit | c1a7bcdf1bfef9455bc58b1737f52530bf681a90 (patch) | |
tree | fb683b37e3ef58bb7bd7698629796ed9c5bfbeae /doc/ci | |
parent | e0d7577e29dcab90623e1f38cf11b351c665ee23 (diff) |
Add latest changes from gitlab-org/gitlab@master
Diffstat (limited to 'doc/ci')
-rw-r--r-- | doc/ci/pipelines/settings.md | 112 | ||||
-rw-r--r-- | doc/ci/testing/code_coverage.md | 129 | ||||
-rw-r--r-- | doc/ci/testing/img/code_coverage_graph_v13_1.png (renamed from doc/ci/pipelines/img/code_coverage_graph_v13_1.png) | bin | 29299 -> 29299 bytes | |||
-rw-r--r-- | doc/ci/testing/img/code_coverage_group_report.png | bin | 0 -> 308000 bytes | |||
-rw-r--r-- | doc/ci/testing/img/coverage_check_approval_rule_14_1.png (renamed from doc/ci/pipelines/img/coverage_check_approval_rule_14_1.png) | bin | 34393 -> 34393 bytes | |||
-rw-r--r-- | doc/ci/testing/img/pipelines_test_coverage_build.png (renamed from doc/ci/pipelines/img/pipelines_test_coverage_build.png) | bin | 4481 -> 4481 bytes | |||
-rw-r--r-- | doc/ci/testing/img/pipelines_test_coverage_mr_widget.png (renamed from doc/ci/pipelines/img/pipelines_test_coverage_mr_widget.png) | bin | 6375 -> 6375 bytes | |||
-rw-r--r-- | doc/ci/testing/index.md | 1 | ||||
-rw-r--r-- | doc/ci/testing/test_coverage_visualization.md | 4 | ||||
-rw-r--r-- | doc/ci/yaml/index.md | 1 |
10 files changed, 133 insertions, 114 deletions
diff --git a/doc/ci/pipelines/settings.md b/doc/ci/pipelines/settings.md index 1d7877a7b2f..3e9e6c50f64 100644 --- a/doc/ci/pipelines/settings.md +++ b/doc/ci/pipelines/settings.md @@ -208,119 +208,7 @@ Jobs that exceed the timeout are marked as failed. You can override this value [for individual runners](../runners/configure_runners.md#set-maximum-job-timeout-for-a-runner). -## Merge request test coverage results - -If you use test coverage in your code, you can use a regular expression to -find coverage results in the job log. You can then include these results -in the merge request in GitLab. - -If the pipeline succeeds, the coverage is shown in the merge request widget and -in the jobs table. If multiple jobs in the pipeline have coverage reports, they are -averaged. - -![MR widget coverage](img/pipelines_test_coverage_mr_widget.png) - -![Build status coverage](img/pipelines_test_coverage_build.png) - -### Add test coverage results using `coverage` keyword - -To add test coverage results to a merge request using the project's `.gitlab-ci.yml` file, provide a regular expression -using the [`coverage`](../yaml/index.md#coverage) keyword. - -### Test coverage examples - -Use this regex for commonly used test tools. - -<!-- vale gitlab.Spelling = NO --> - -- Simplecov (Ruby). Example: `/\(\d+.\d+\%\) covered/`. -- pytest-cov (Python). Example: `/(?i)total.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/`. -- Scoverage (Scala). Example: `/Statement coverage[A-Za-z\.*]\s*:\s*([^%]+)/`. -- `pest --coverage --colors=never` (PHP). Example: `/^\s*Cov:\s*\d+\.\d+?%$/`. -- `phpunit --coverage-text --colors=never` (PHP). Example: `/^\s*Lines:\s*\d+.\d+\%/`. -- gcovr (C/C++). Example: `/^TOTAL.*\s+(\d+\%)$/`. -- `tap --coverage-report=text-summary` (NodeJS). Example: `/^Statements\s*:\s*([^%]+)/`. -- `nyc npm test` (NodeJS). Example: `/All files[^|]*\|[^|]*\s+([\d\.]+)/`. -- `jest --ci --coverage` (NodeJS). Example: `/All files[^|]*\|[^|]*\s+([\d\.]+)/`. -- excoveralls (Elixir). Example: `/\[TOTAL\]\s+(\d+\.\d+)%/`. -- `mix test --cover` (Elixir). Example: `/\d+.\d+\%\s+\|\s+Total/`. -- JaCoCo (Java/Kotlin). Example: `/Total.*?([0-9]{1,3})%/`. -- `go test -cover` (Go). Example: `/coverage: \d+.\d+% of statements/`. -- .NET (OpenCover). Example: `/(Visited Points).*\((.*)\)/`. -- .NET (`dotnet test` line coverage). Example: `/Total\s*\|\s*(\d+(?:\.\d+)?)/`. -- tarpaulin (Rust). Example: `/^\d+.\d+% coverage/`. -- Pester (PowerShell). Example: `/Covered (\d+\.\d+%)/`. - -<!-- vale gitlab.Spelling = YES --> - -### View code coverage history - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/209121) the ability to download a `.csv` in GitLab 12.10. -> - Graph [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/33743) in GitLab 13.1. - -To see the evolution of your project code coverage over time, -you can view a graph or download a CSV file with this data. - -1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Analytics > Repository**. - -The historic data for each job is listed in the dropdown list above the graph. - -To view a CSV file of the data, select **Download raw data (`.csv`)**. - -![Code coverage graph of a project over time](img/code_coverage_graph_v13_1.png) - -Code coverage data is also [available at the group level](../../user/group/repositories_analytics/index.md). - -### Coverage check approval rule **(PREMIUM)** - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15765) in GitLab 14.0. -> - [Made configurable in Project Settings](https://gitlab.com/gitlab-org/gitlab/-/issues/331001) in GitLab 14.1. - -You can implement merge request approvals to require approval by selected users or a group -when merging a merge request would cause the project's test coverage to decline. - -Follow these steps to enable the `Coverage-Check` MR approval rule: - -1. Set up a [`coverage`](../yaml/index.md#coverage) regular expression for all jobs you want to include in the overall coverage value. -1. Go to your project and select **Settings > Merge requests**. -1. Under **Merge request approvals**, select **Enable** next to the `Coverage-Check` approval rule. -1. Select the **Target branch**. -1. Set the number of **Approvals required** to greater than zero. -1. Select the users or groups to provide approval. -1. Select **Add approval rule**. - -![Coverage-Check approval rule](img/coverage_check_approval_rule_14_1.png) - -### Remove color codes from code coverage - -Some test coverage tools output with ANSI color codes that aren't -parsed correctly by the regular expression. This causes coverage -parsing to fail. - -Some coverage tools don't provide an option to disable color -codes in the output. If so, pipe the output of the coverage tool through a -small one line script that strips the color codes off. - -For example: - -```shell -lein cloverage | perl -pe 's/\e\[?.*?[\@-~]//g' -``` - ## Pipeline badges You can use [pipeline badges](../../user/project/badges.md) to indicate the pipeline status and test coverage of your projects. These badges are determined by the latest successful pipeline. - -<!-- ## Troubleshooting - -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. - -Each scenario can be a third-level heading, for example `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> diff --git a/doc/ci/testing/code_coverage.md b/doc/ci/testing/code_coverage.md new file mode 100644 index 00000000000..bd1246d2f78 --- /dev/null +++ b/doc/ci/testing/code_coverage.md @@ -0,0 +1,129 @@ +--- +stage: Verify +group: Pipeline Execution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Code coverage **(FREE)** + +Use code coverage to provide insights on what source code is being validated by a test suite. Code coverage is one of many test metrics that can determine software performance and quality. + +## View Code Coverage results + +Code Coverage results are shown in: + +- Merge request widget +- Project repository analytics +- Group repository analytics +- Repository badge + +For more information on test coverage visualization in the file diff of the MR, see [Test Coverage Visualization](test_coverage_visualization.md). + +### View code coverage results in the MR + +If you use test coverage in your code, you can use a regular expression to +find coverage results in the job log. You can then include these results +in the merge request in GitLab. + +If the pipeline succeeds, the coverage is shown in the merge request widget and +in the jobs table. If multiple jobs in the pipeline have coverage reports, they are +averaged. + +![MR widget coverage](img/pipelines_test_coverage_mr_widget.png) + +![Build status coverage](img/pipelines_test_coverage_build.png) + +#### Add test coverage results using `coverage` keyword + +To add test coverage results to a merge request using the project's `.gitlab-ci.yml` file, provide a regular expression +using the [`coverage`](../yaml/index.md#coverage) keyword. + +#### Test coverage examples + +Use this regex for commonly used test tools. + +<!-- vale gitlab.Spelling = NO --> + +- Simplecov (Ruby). Example: `/\(\d+.\d+\%\) covered/`. +- pytest-cov (Python). Example: `/(?i)total.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/`. +- Scoverage (Scala). Example: `/Statement coverage[A-Za-z\.*]\s*:\s*([^%]+)/`. +- `pest --coverage --colors=never` (PHP). Example: `/^\s*Cov:\s*\d+\.\d+?%$/`. +- `phpunit --coverage-text --colors=never` (PHP). Example: `/^\s*Lines:\s*\d+.\d+\%/`. +- gcovr (C/C++). Example: `/^TOTAL.*\s+(\d+\%)$/`. +- `tap --coverage-report=text-summary` (NodeJS). Example: `/^Statements\s*:\s*([^%]+)/`. +- `nyc npm test` (NodeJS). Example: `/All files[^|]*\|[^|]*\s+([\d\.]+)/`. +- `jest --ci --coverage` (NodeJS). Example: `/All files[^|]*\|[^|]*\s+([\d\.]+)/`. +- excoveralls (Elixir). Example: `/\[TOTAL\]\s+(\d+\.\d+)%/`. +- `mix test --cover` (Elixir). Example: `/\d+.\d+\%\s+\|\s+Total/`. +- JaCoCo (Java/Kotlin). Example: `/Total.*?([0-9]{1,3})%/`. +- `go test -cover` (Go). Example: `/coverage: \d+.\d+% of statements/`. +- .NET (OpenCover). Example: `/(Visited Points).*\((.*)\)/`. +- .NET (`dotnet test` line coverage). Example: `/Total\s*\|\s*(\d+(?:\.\d+)?)/`. +- tarpaulin (Rust). Example: `/^\d+.\d+% coverage/`. +- Pester (PowerShell). Example: `/Covered (\d+\.\d+%)/`. + +<!-- vale gitlab.Spelling = YES --> + +### View history of project code coverage + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/209121) the ability to download a `.csv` in GitLab 12.10. +> - Graph [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/33743) in GitLab 13.1. + +To see the evolution of your project code coverage over time, +you can view a graph or download a CSV file with this data. + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Analytics > Repository**. + +The historic data for each job is listed in the dropdown list above the graph. + +To view a CSV file of the data, select **Download raw data (`.csv`)**. + +![Code coverage graph of a project over time](img/code_coverage_graph_v13_1.png) + +### View history of group code coverage **(PREMIUM)** + +To see the all the project's code coverage under a group over time, you can find view [group repository analytics](../../user/group/repositories_analytics/index.md). + +![Code coverage graph of a group over time](img/code_coverage_group_report.png) + +### Pipeline badges + +You can use [pipeline badges](../../user/project/badges.md#test-coverage-report-badges) to indicate the pipeline status and +test coverage of your projects. These badges are determined by the latest successful pipeline. + +## Coverage check approval rule **(PREMIUM)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15765) in GitLab 14.0. +> - [Made configurable in Project Settings](https://gitlab.com/gitlab-org/gitlab/-/issues/331001) in GitLab 14.1. + +When merging a request that would cause the project's test coverage to decline, you can stipulate that such merge requests require approval by selected users or a group. + +Follow these steps to enable the `Coverage-Check` MR approval rule: + +1. Set up a [`coverage`](../yaml/index.md#coverage) regular expression for all jobs you want to include in the overall coverage value. +1. Go to your project and select **Settings > Merge requests**. +1. Under **Merge request approvals**, select **Enable** next to the `Coverage-Check` approval rule. +1. Select the **Target branch**. +1. Set the number of **Approvals required** to greater than zero. +1. Select the users or groups to provide approval. +1. Select **Add approval rule**. + +![Coverage-Check approval rule](img/coverage_check_approval_rule_14_1.png) + +## Troubleshooting + +### Remove color codes from code coverage + +Some test coverage tools output with ANSI color codes that aren't +parsed correctly by the regular expression. This causes coverage +parsing to fail. + +Some coverage tools do not provide an option to disable color +codes in the output. If so, pipe the output of the coverage tool through a one-line script that strips the color codes. + +For example: + +```shell +lein cloverage | perl -pe 's/\e\[?.*?[\@-~]//g' +``` diff --git a/doc/ci/pipelines/img/code_coverage_graph_v13_1.png b/doc/ci/testing/img/code_coverage_graph_v13_1.png Binary files differindex da92a1b4a86..da92a1b4a86 100644 --- a/doc/ci/pipelines/img/code_coverage_graph_v13_1.png +++ b/doc/ci/testing/img/code_coverage_graph_v13_1.png diff --git a/doc/ci/testing/img/code_coverage_group_report.png b/doc/ci/testing/img/code_coverage_group_report.png Binary files differnew file mode 100644 index 00000000000..df8ca613f8c --- /dev/null +++ b/doc/ci/testing/img/code_coverage_group_report.png diff --git a/doc/ci/pipelines/img/coverage_check_approval_rule_14_1.png b/doc/ci/testing/img/coverage_check_approval_rule_14_1.png Binary files differindex 8c1d005e074..8c1d005e074 100644 --- a/doc/ci/pipelines/img/coverage_check_approval_rule_14_1.png +++ b/doc/ci/testing/img/coverage_check_approval_rule_14_1.png diff --git a/doc/ci/pipelines/img/pipelines_test_coverage_build.png b/doc/ci/testing/img/pipelines_test_coverage_build.png Binary files differindex 7eaba1a256f..7eaba1a256f 100644 --- a/doc/ci/pipelines/img/pipelines_test_coverage_build.png +++ b/doc/ci/testing/img/pipelines_test_coverage_build.png diff --git a/doc/ci/pipelines/img/pipelines_test_coverage_mr_widget.png b/doc/ci/testing/img/pipelines_test_coverage_mr_widget.png Binary files differindex fbcd612f3f2..fbcd612f3f2 100644 --- a/doc/ci/pipelines/img/pipelines_test_coverage_mr_widget.png +++ b/doc/ci/testing/img/pipelines_test_coverage_mr_widget.png diff --git a/doc/ci/testing/index.md b/doc/ci/testing/index.md index d97c58b21e8..a8fb6d688d7 100644 --- a/doc/ci/testing/index.md +++ b/doc/ci/testing/index.md @@ -14,6 +14,7 @@ display reports or link to important information directly from [merge requests]( | [Accessibility Testing](accessibility_testing.md) | Automatically report A11y violations for changed pages in merge requests. | | [Browser Performance Testing](browser_performance_testing.md) | Quickly determine the browser performance impact of pending code changes. | | [Load Performance Testing](load_performance_testing.md) | Quickly determine the server performance impact of pending code changes. | +| [Code Coverage](code_coverage.md) | See code coverage results in the MR, project or group. | | [Code Quality](code_quality.md) | Analyze your source code quality using the [Code Climate](https://codeclimate.com/) analyzer and show the Code Climate report right in the merge request widget area. | | [Display arbitrary job artifacts](../yaml/index.md#artifactsexpose_as) | Configure CI pipelines with the `artifacts:expose_as` parameter to directly link to selected [artifacts](../jobs/job_artifacts.md) in merge requests. | | [Unit test reports](unit_test_reports.md) | Configure your CI jobs to use Unit test reports, and let GitLab display a report on the merge request so that it's easier and faster to identify the failure without having to check the entire job log. | diff --git a/doc/ci/testing/test_coverage_visualization.md b/doc/ci/testing/test_coverage_visualization.md index 52cdde88538..94ee4d2dce2 100644 --- a/doc/ci/testing/test_coverage_visualization.md +++ b/doc/ci/testing/test_coverage_visualization.md @@ -54,8 +54,8 @@ of times the line was checked by tests. Uploading a test coverage report does not enable: -- [Test coverage results in merge requests](../pipelines/settings.md#merge-request-test-coverage-results). -- [Code coverage history](../pipelines/settings.md#view-code-coverage-history). +- [Test coverage results in merge requests](code_coverage.md#view-code-coverage-results-in-the-mr). +- [Code coverage history](code_coverage.md#view-history-of-project-code-coverage). You must configure these separately. diff --git a/doc/ci/yaml/index.md b/doc/ci/yaml/index.md index 06f574906c5..fc903144a87 100644 --- a/doc/ci/yaml/index.md +++ b/doc/ci/yaml/index.md @@ -1512,6 +1512,7 @@ In this example: **Additional details**: +- You can find parse examples in [Code Coverage](../testing/code_coverage.md#test-coverage-examples). - If there is more than one matched line in the job output, the last line is used (the first result of reverse search). - If there are multiple matches in a single line, the last match is searched |