diff options
Diffstat (limited to 'doc/ci/pipelines')
| -rw-r--r-- | doc/ci/pipelines/img/ci_efficiency_pipeline_health_grafana_dashboard.png | bin | 55200 -> 242875 bytes | |||
| -rw-r--r-- | doc/ci/pipelines/index.md | 20 | ||||
| -rw-r--r-- | doc/ci/pipelines/job_artifacts.md | 32 | ||||
| -rw-r--r-- | doc/ci/pipelines/pipeline_architectures.md | 10 | ||||
| -rw-r--r-- | doc/ci/pipelines/pipeline_artifacts.md | 4 | ||||
| -rw-r--r-- | doc/ci/pipelines/pipeline_efficiency.md | 14 | ||||
| -rw-r--r-- | doc/ci/pipelines/schedules.md | 16 | ||||
| -rw-r--r-- | doc/ci/pipelines/settings.md | 9 |
8 files changed, 72 insertions, 33 deletions
diff --git a/doc/ci/pipelines/img/ci_efficiency_pipeline_health_grafana_dashboard.png b/doc/ci/pipelines/img/ci_efficiency_pipeline_health_grafana_dashboard.png Binary files differindex 59276bda727..d9b8e71e6cb 100644 --- a/doc/ci/pipelines/img/ci_efficiency_pipeline_health_grafana_dashboard.png +++ b/doc/ci/pipelines/img/ci_efficiency_pipeline_health_grafana_dashboard.png diff --git a/doc/ci/pipelines/index.md b/doc/ci/pipelines/index.md index f22d2373e5f..22e331f2de0 100644 --- a/doc/ci/pipelines/index.md +++ b/doc/ci/pipelines/index.md @@ -1,7 +1,7 @@ --- stage: Verify group: Continuous Integration -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments disqus_identifier: 'https://docs.gitlab.com/ee/ci/pipelines.html' type: reference --- @@ -10,7 +10,7 @@ type: reference > Introduced in GitLab 8.8. -TIP: **Tip:** +NOTE: Watch the ["Mastering continuous software development"](https://about.gitlab.com/webcast/mastering-ci-cd/) webcast to see a comprehensive demo of a GitLab CI/CD pipeline. @@ -39,7 +39,7 @@ A typical pipeline might consist of four stages, executed in the following order - A `staging` stage, with a job called `deploy-to-stage`. - A `production` stage, with a job called `deploy-to-prod`. -NOTE: **Note:** +NOTE: If you have a [mirrored repository that GitLab pulls from](../../user/project/repository/repository_mirroring.md#pulling-from-a-remote-repository), you may need to enable pipeline triggering in your project's **Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**. @@ -213,7 +213,7 @@ page, then using the **Delete** button.  -CAUTION: **Warning:** +WARNING: Deleting a pipeline expires all pipeline caches, and deletes all related objects, such as builds, logs, artifacts, and triggers. **This action cannot be undone.** @@ -261,6 +261,18 @@ The union of A, B, and C is (1, 4) and (6, 7). Therefore, the total running time (4 - 1) + (7 - 6) => 4 ``` +#### How pipeline quota usage is calculated + +Pipeline quota usage is calculated as the sum of the duration of each individual job. This is slightly different to how pipeline _duration_ is [calculated](#how-pipeline-duration-is-calculated). Pipeline quota usage doesn't consider any overlap of jobs running in parallel. + +For example, a pipeline consists of the following jobs: + +- Job A takes 3 minutes. +- Job B takes 3 minutes. +- Job C takes 2 minutes. + +The pipeline quota usage is the sum of each job's duration. In this example, 8 runner minutes would be used, calculated as: 3 + 3 + 2. + ### Pipeline security on protected branches A strict security model is enforced when pipelines are executed on diff --git a/doc/ci/pipelines/job_artifacts.md b/doc/ci/pipelines/job_artifacts.md index 160399e4bc4..787ee8f8573 100644 --- a/doc/ci/pipelines/job_artifacts.md +++ b/doc/ci/pipelines/job_artifacts.md @@ -1,7 +1,7 @@ --- stage: Verify group: Continuous Integration -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments disqus_identifier: 'https://docs.gitlab.com/ee/user/project/pipelines/job_artifacts.html' type: reference, howto --- @@ -58,7 +58,7 @@ For more examples on artifacts, follow the [artifacts reference in > - Requires GitLab Runner 11.2 and above. The `artifacts:reports` keyword is used for collecting test reports, code quality -reports, and security reports from jobs. It also exposes these reports in GitLab's +reports, and security reports from jobs. It also exposes these reports in the GitLab UI (merge requests, pipeline views, and security dashboards). The test reports are collected regardless of the job results (success or failure). @@ -213,12 +213,34 @@ as artifacts. The collected DAST report uploads to GitLab as an artifact and is summarized in merge requests and the pipeline view. It's also used to provide data for security dashboards. +#### `artifacts:reports:api_fuzzing` **(ULTIMATE)** + +> - Introduced in GitLab 13.4. +> - Requires GitLab Runner 13.4 or later. + +The `api_fuzzing` report collects [API Fuzzing bugs](../../user/application_security/api_fuzzing/index.md) +as artifacts. + +The collected API Fuzzing report uploads to GitLab as an artifact and is summarized in merge +requests and the pipeline view. It's also used to provide data for security dashboards. + +#### `artifacts:reports:coverage_fuzzing` **(ULTIMATE)** + +> - Introduced in GitLab 13.4. +> - Requires GitLab Runner 13.4 or later. + +The `coverage_fuzzing` report collects [coverage fuzzing bugs](../../user/application_security/coverage_fuzzing/index.md) +as artifacts. + +The collected coverage fuzzing report uploads to GitLab as an artifact and is summarized in merge +requests and the pipeline view. It's also used to provide data for security dashboards. + #### `artifacts:reports:license_management` **(ULTIMATE)** > - Introduced in GitLab 11.5. > - Requires GitLab Runner 11.5 and above. -CAUTION: **Warning:** +WARNING: This artifact is still valid but is **deprecated** in favor of the [artifacts:reports:license_scanning](../pipelines/job_artifacts.md#artifactsreportslicense_scanning) introduced in GitLab 12.8. @@ -337,7 +359,7 @@ in the GitLab UI to do this: It's possible to download the latest artifacts of a job via a well known URL so you can use it for scripting purposes. -NOTE: **Note:** +NOTE: The latest artifacts are created by jobs in the **most recent** successful pipeline for the specific ref. If you run two types of pipelines for the same ref, timing determines the latest artifact. For example, if a merge request creates a branch pipeline at the same time as a scheduled pipeline, the pipeline that completed most recently creates the latest artifact. @@ -416,7 +438,7 @@ information in the UI. ## Erasing artifacts -DANGER: **Warning:** +WARNING: This is a destructive action that leads to data loss. Use with caution. You can erase a single job via the UI, which also removes the job's diff --git a/doc/ci/pipelines/pipeline_architectures.md b/doc/ci/pipelines/pipeline_architectures.md index 77614424b33..73677dd6986 100644 --- a/doc/ci/pipelines/pipeline_architectures.md +++ b/doc/ci/pipelines/pipeline_architectures.md @@ -1,7 +1,7 @@ --- stage: Verify group: Continuous Integration -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -22,8 +22,8 @@ any of the keywords used below, check out our [CI YAML reference](../yaml/README ## Basic Pipelines -This is the simplest pipeline in GitLab. It will run everything in the build stage concurrently, -and once all of those finish, it will run everything in the test stage the same way, and so on. +This is the simplest pipeline in GitLab. It runs everything in the build stage concurrently, +and once all of those finish, it runs everything in the test stage the same way, and so on. It's not the most efficient, and if you have lots of steps it can grow quite complex, but it's easier to maintain: @@ -101,7 +101,7 @@ your jobs. When GitLab knows the relationships between your jobs, it can run eve as fast as possible, and even skips into subsequent stages when possible. In the example below, if `build_a` and `test_a` are much faster than `build_b` and -`test_b`, GitLab will start `deploy_a` even if `build_b` is still running. +`test_b`, GitLab starts `deploy_a` even if `build_b` is still running. ```mermaid graph LR @@ -163,7 +163,7 @@ deploy_b: In the examples above, it's clear we've got two types of things that could be built independently. This is an ideal case for using [Child / Parent Pipelines](../parent_child_pipelines.md)) via -the [`trigger` keyword](../yaml/README.md#trigger). It will separate out the configuration +the [`trigger` keyword](../yaml/README.md#trigger). It separates out the configuration into multiple files, keeping things very simple. You can also combine this with: - The [`rules` keyword](../yaml/README.md#rules): For example, have the child pipelines triggered only diff --git a/doc/ci/pipelines/pipeline_artifacts.md b/doc/ci/pipelines/pipeline_artifacts.md index 8d70fff4e03..e405258092d 100644 --- a/doc/ci/pipelines/pipeline_artifacts.md +++ b/doc/ci/pipelines/pipeline_artifacts.md @@ -1,7 +1,7 @@ --- stage: Verify group: Continuous Integration -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference, howto --- @@ -13,6 +13,6 @@ Pipeline artifacts are used by the [test coverage visualization feature](../../u ## Storage -Pipeline artifacts are saved to disk or object storage. They count towards a project's [storage usage quota](../../user/group/index.md#storage-usage-quota). The **Artifacts** on the usage quote page is the sum of all job artifacts and pipeline artifacts. +Pipeline artifacts are saved to disk or object storage. They count towards a project's [storage usage quota](../../user/usage_quotas.md#storage-usage-quota). The **Artifacts** on the Usage Quotas page is the sum of all job artifacts and pipeline artifacts. Pipeline artifacts are erased after one week. diff --git a/doc/ci/pipelines/pipeline_efficiency.md b/doc/ci/pipelines/pipeline_efficiency.md index 149c596430c..de78b8558c4 100644 --- a/doc/ci/pipelines/pipeline_efficiency.md +++ b/doc/ci/pipelines/pipeline_efficiency.md @@ -1,7 +1,7 @@ --- stage: Verify group: Continuous Integration -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -105,10 +105,10 @@ External monitoring tools can poll the API and verify pipeline health or collect metrics for long term SLA analytics. For example, the [GitLab CI Pipelines Exporter](https://github.com/mvisonneau/gitlab-ci-pipelines-exporter) -for Prometheus fetches metrics from the API. It can check branches in projects automatically +for Prometheus fetches metrics from the API and pipeline events. It can check branches in projects automatically and get the pipeline status and duration. In combination with a Grafana dashboard, this helps build an actionable view for your operations team. Metric graphs can also -be embedded into incidents making problem resolving easier. +be embedded into incidents making problem resolving easier. Additionally, it can also export metrics about jobs and environments.  @@ -189,8 +189,12 @@ be more efficient, but can also make pipelines harder to understand and analyze. ### Caching -Another optimization method is to use [caching](../caching/index.md) between jobs and stages, -for example [`/node_modules` for NodeJS](../caching/index.md#caching-nodejs-dependencies). +Another optimization method is to [cache](../caching/index.md) dependencies. If your +dependencies change rarely, like [NodeJS `/node_modules`](../caching/index.md#cache-nodejs-dependencies), +caching can make pipeline execution much faster. + +You can use [`cache:when`](../yaml/README.md#cachewhen) to cache downloaded dependencies +even when a job fails. ### Docker Images diff --git a/doc/ci/pipelines/schedules.md b/doc/ci/pipelines/schedules.md index 9df73957293..35a8888381f 100644 --- a/doc/ci/pipelines/schedules.md +++ b/doc/ci/pipelines/schedules.md @@ -1,7 +1,7 @@ --- stage: Verify group: Continuous Integration -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments disqus_identifier: 'https://docs.gitlab.com/ee/user/project/pipelines/schedules.html' type: reference, howto --- @@ -43,7 +43,7 @@ To schedule a pipeline for project:  -NOTE: **Note:** +NOTE: Pipelines execution [timing is dependent](#advanced-configuration) on Sidekiq's own schedule. In the **Schedules** index page you can see a list of the pipelines that are @@ -56,15 +56,15 @@ is installed on. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12328) in GitLab 9.4. -You can pass any number of arbitrary variables and they will be available in +You can pass any number of arbitrary variables. They are available in GitLab CI/CD so that they can be used in your [`.gitlab-ci.yml` file](../../ci/yaml/README.md).  ### Using only and except -To configure that a job can be executed only when the pipeline has been -scheduled (or the opposite), you can use +To configure a job to be executed only when the pipeline has been +scheduled (or the opposite), use [only and except](../yaml/README.md#onlyexcept-basic) configuration keywords. For example: @@ -102,7 +102,7 @@ For GitLab.com, refer to the [dedicated settings page](../../user/gitlab_com/ind ## Working with scheduled pipelines -Once configured, GitLab supports many functions for working with scheduled pipelines. +After configuration, GitLab supports many functions for working with scheduled pipelines. ### Running manually @@ -115,7 +115,7 @@ To trigger a pipeline schedule manually, click the "Play" button: This schedules a background job to run the pipeline schedule. A flash message provides a link to the CI/CD Pipeline index page. -NOTE: **Note:** +NOTE: To help avoid abuse, users are rate limited to triggering a pipeline once per minute. @@ -128,7 +128,7 @@ The next time a pipeline is scheduled, your credentials are used.  -If the owner of a pipeline schedule does not have the ability to create +If the owner of a pipeline schedule cannot create pipelines on the target branch, the schedule stops creating new pipelines. diff --git a/doc/ci/pipelines/settings.md b/doc/ci/pipelines/settings.md index d2d2cb26256..5a758bccd62 100644 --- a/doc/ci/pipelines/settings.md +++ b/doc/ci/pipelines/settings.md @@ -1,7 +1,7 @@ --- stage: Verify group: Continuous Integration -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments disqus_identifier: 'https://docs.gitlab.com/ee/user/project/pipelines/settings.html' type: reference, howto --- @@ -26,7 +26,7 @@ There are two options. Using: - `git clone`, which is slower since it clones the repository from scratch for every job, ensuring that the local working copy is always pristine. -- `git fetch`, which is GitLab's default and faster as it re-uses the local working copy (falling +- `git fetch`, which is default in GitLab and faster as it re-uses the local working copy (falling back to clone if it doesn't exist). This is recommended, especially for [large repositories](../large_repositories/index.md#git-strategy). @@ -170,7 +170,7 @@ Pipeline visibility is determined by: - Your current [user access level](../../user/permissions.md). - The **Public pipelines** project setting under your project's **Settings > CI/CD > General pipelines**. -NOTE: **Note:** +NOTE: If the project visibility is set to **Private**, the [**Public pipelines** setting has no effect](../enable_or_disable_ci.md#per-project-user-setting). This also determines the visibility of these related features: @@ -209,7 +209,8 @@ You can set pending or running pipelines to cancel automatically when a new pipe 1. Check the **Auto-cancel redundant, pending pipelines** checkbox. 1. Click **Save changes**. -Note that only jobs with [interruptible](../yaml/README.md#interruptible) set to `true` are cancelled. +Use the [`interruptible`](../yaml/README.md#interruptible) keyword to indicate if a +running job can be cancelled before it completes. ## Skip outdated deployment jobs |
