diff options
Diffstat (limited to 'doc/ci/jobs/job_control.md')
| -rw-r--r-- | doc/ci/jobs/job_control.md | 391 |
1 files changed, 388 insertions, 3 deletions
diff --git a/doc/ci/jobs/job_control.md b/doc/ci/jobs/job_control.md index 6e9197c223b..d7e192bbfda 100644 --- a/doc/ci/jobs/job_control.md +++ b/doc/ci/jobs/job_control.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -19,6 +19,277 @@ To configure a job to be included or excluded from certain pipelines, you can us Use [`needs`](../yaml/README.md#needs) to configure a job to run as soon as the earlier jobs it depends on finish running. +## Specify when jobs run with `rules` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27863) in GitLab 12.3. + +Use [`rules`](../yaml/README.md#rules) to include or exclude jobs in pipelines. + +Rules are evaluated in order until the first match. When a match is found, the job +is either included or excluded from the pipeline, depending on the configuration. +See the [`rules`](../yaml/README.md#rules) reference for more details. + +Future keyword improvements are being discussed in our [epic for improving `rules`](https://gitlab.com/groups/gitlab-org/-/epics/2783), +where anyone can add suggestions or requests. + +### `rules` examples + +The following example uses `if` to define that the job runs in only two specific cases: + +```yaml +job: + script: echo "Hello, Rules!" + rules: + - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' + when: manual + allow_failure: true + - if: '$CI_PIPELINE_SOURCE == "schedule"' +``` + +- If the pipeline is for a merge request, the first rule matches, and the job + is added to the [merge request pipeline](../merge_request_pipelines/index.md) + with attributes of: + - `when: manual` (manual job) + - `allow_failure: true` (the pipeline continues running even if the manual job is not run) +- If the pipeline is **not** for a merge request, the first rule doesn't match, and the + second rule is evaluated. +- If the pipeline is a scheduled pipeline, the second rule matches, and the job + is added to the scheduled pipeline. No attributes were defined, so it is added + with: + - `when: on_success` (default) + - `allow_failure: false` (default) +- In **all other cases**, no rules match, so the job is **not** added to any other pipeline. + +Alternatively, you can define a set of rules to exclude jobs in a few cases, but +run them in all other cases: + +```yaml +job: + script: echo "Hello, Rules!" + rules: + - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' + when: never + - if: '$CI_PIPELINE_SOURCE == "schedule"' + when: never + - when: on_success +``` + +- If the pipeline is for a merge request, the job is **not** added to the pipeline. +- If the pipeline is a scheduled pipeline, the job is **not** added to the pipeline. +- In **all other cases**, the job is added to the pipeline, with `when: on_success`. + +WARNING: +If you use a `when:` clause as the final rule (not including `when: never`), two +simultaneous pipelines may start. Both push pipelines and merge request pipelines can +be triggered by the same event (a push to the source branch for an open merge request). +See how to [prevent duplicate pipelines](#avoid-duplicate-pipelines) +for more details. + +### Complex rules + +You can use all `rules` keywords, like `if`, `changes`, and `exists`, in the same +rule. The rule evaluates to true only when all included keywords evaluate to true. + +For example: + +```yaml +docker build: + script: docker build -t my-image:$CI_COMMIT_REF_SLUG . + rules: + - if: '$VAR == "string value"' + changes: # Include the job and set to when:manual if any of the follow paths match a modified file. + - Dockerfile + - docker/scripts/* + when: manual + allow_failure: true +``` + +If the `Dockerfile` file or any file in `/docker/scripts` has changed **and** `$VAR` == "string value", +then the job runs manually and is allowed to fail. + +You can use [parentheses](#group-variable-expressions-together-with-parentheses) with `&&` and `||` to build more complicated variable expressions. +[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230938) in GitLab 13.3: + +```yaml +job1: + script: + - echo This rule uses parentheses. + rules: + if: ($CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH || $CI_COMMIT_BRANCH == "develop") && $MY_VARIABLE +``` + +WARNING: +[Before GitLab 13.3](https://gitlab.com/gitlab-org/gitlab/-/issues/230938), +rules that use both `||` and `&&` may evaluate with an unexpected order of operations. + +### Avoid duplicate pipelines + +If a job uses `rules`, a single action, like pushing a commit to a branch, can trigger +multiple pipelines. You don't have to explicitly configure rules for multiple types +of pipeline to trigger them accidentally. + +Some configurations that have the potential to cause duplicate pipelines cause a +[pipeline warning](../troubleshooting.md#pipeline-warnings) to be displayed. +[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/219431) in GitLab 13.3. + +For example: + +```yaml +job: + script: echo "This job creates double pipelines!" + rules: + - if: '$CUSTOM_VARIABLE == "false"' + when: never + - when: always +``` + +This job does not run when `$CUSTOM_VARIABLE` is false, but it *does* run in **all** +other pipelines, including **both** push (branch) and merge request pipelines. With +this configuration, every push to an open merge request's source branch +causes duplicated pipelines. + +To avoid duplicate pipelines, you can: + +- Use [`workflow`](../yaml/README.md#workflow) to specify which types of pipelines + can run. +- Rewrite the rules to run the job only in very specific cases, + and avoid a final `when:` rule: + + ```yaml + job: + script: echo "This job does NOT create double pipelines!" + rules: + - if: '$CUSTOM_VARIABLE == "true" && $CI_PIPELINE_SOURCE == "merge_request_event"' + ``` + +You can also avoid duplicate pipelines by changing the job rules to avoid either push (branch) +pipelines or merge request pipelines. However, if you use a `- when: always` rule without +`workflow: rules`, GitLab still displays a [pipeline warning](../troubleshooting.md#pipeline-warnings). + +For example, the following does not trigger double pipelines, but is not recommended +without `workflow: rules`: + +```yaml +job: + script: echo "This job does NOT create double pipelines!" + rules: + - if: '$CI_PIPELINE_SOURCE == "push"' + when: never + - when: always +``` + +You should not include both push and merge request pipelines in the same job without +[`workflow:rules` that prevent duplicate pipelines](../yaml/README.md#switch-between-branch-pipelines-and-merge-request-pipelines): + +```yaml +job: + script: echo "This job creates double pipelines!" + rules: + - if: '$CI_PIPELINE_SOURCE == "push"' + - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' +``` + +Also, do not mix `only/except` jobs with `rules` jobs in the same pipeline. +It may not cause YAML errors, but the different default behaviors of `only/except` +and `rules` can cause issues that are difficult to troubleshoot: + +```yaml +job-with-no-rules: + script: echo "This job runs in branch pipelines." + +job-with-rules: + script: echo "This job runs in merge request pipelines." + rules: + - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' +``` + +For every change pushed to the branch, duplicate pipelines run. One +branch pipeline runs a single job (`job-with-no-rules`), and one merge request pipeline +runs the other job (`job-with-rules`). Jobs with no rules default +to [`except: merge_requests`](../yaml/README.md#only--except), so `job-with-no-rules` +runs in all cases except merge requests. + +### Common `if` clauses for `rules` + +For behavior similar to the [`only`/`except` keywords](../yaml/README.md#only--except), you can +check the value of the `$CI_PIPELINE_SOURCE` variable: + +| Value | Description | +|-------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `api` | For pipelines triggered by the [pipelines API](../../api/pipelines.md#create-a-new-pipeline). | +| `chat` | For pipelines created by using a [GitLab ChatOps](../chatops/index.md) command. | +| `external` | When you use CI services other than GitLab. | +| `external_pull_request_event` | When an external pull request on GitHub is created or updated. See [Pipelines for external pull requests](../ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests). | +| `merge_request_event` | For pipelines created when a merge request is created or updated. Required to enable [merge request pipelines](../merge_request_pipelines/index.md), [merged results pipelines](../merge_request_pipelines/pipelines_for_merged_results/index.md), and [merge trains](../merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md). | +| `parent_pipeline` | For pipelines triggered by a [parent/child pipeline](../parent_child_pipelines.md) with `rules`. Use this pipeline source in the child pipeline configuration so that it can be triggered by the parent pipeline. | +| `pipeline` | For [multi-project pipelines](../multi_project_pipelines.md) created by [using the API with `CI_JOB_TOKEN`](../multi_project_pipelines.md#triggering-multi-project-pipelines-through-api), or the [`trigger`](../yaml/README.md#trigger) keyword. | +| `push` | For pipelines triggered by a `git push` event, including for branches and tags. | +| `schedule` | For [scheduled pipelines](../pipelines/schedules.md). | +| `trigger` | For pipelines created by using a [trigger token](../triggers/README.md#trigger-token). | +| `web` | For pipelines created by using **Run pipeline** button in the GitLab UI, from the project's **CI/CD > Pipelines** section. | +| `webide` | For pipelines created by using the [WebIDE](../../user/project/web_ide/index.md). | + +The following example runs the job as a manual job in scheduled pipelines or in push +pipelines (to branches or tags), with `when: on_success` (default). It does not +add the job to any other pipeline type. + +```yaml +job: + script: echo "Hello, Rules!" + rules: + - if: '$CI_PIPELINE_SOURCE == "schedule"' + when: manual + allow_failure: true + - if: '$CI_PIPELINE_SOURCE == "push"' +``` + +The following example runs the job as a `when: on_success` job in [merge request pipelines](../merge_request_pipelines/index.md) +and scheduled pipelines. It does not run in any other pipeline type. + +```yaml +job: + script: echo "Hello, Rules!" + rules: + - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' + - if: '$CI_PIPELINE_SOURCE == "schedule"' +``` + +Other commonly used variables for `if` clauses: + +- `if: $CI_COMMIT_TAG`: If changes are pushed for a tag. +- `if: $CI_COMMIT_BRANCH`: If changes are pushed to any branch. +- `if: '$CI_COMMIT_BRANCH == "main"'`: If changes are pushed to `main`. +- `if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'`: If changes are pushed to the default + branch. Use when you want to have the same configuration in multiple + projects with different default branches. +- `if: '$CI_COMMIT_BRANCH =~ /regex-expression/'`: If the commit branch matches a regular expression. +- `if: '$CUSTOM_VARIABLE !~ /regex-expression/'`: If the [custom variable](../variables/README.md#custom-cicd-variables) + `CUSTOM_VARIABLE` does **not** match a regular expression. +- `if: '$CUSTOM_VARIABLE == "value1"'`: If the custom variable `CUSTOM_VARIABLE` is + exactly `value1`. + +### Variables in `rules:changes` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34272) in GitLab 13.6. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/267192) in GitLab 13.7. + +You can use CI/CD variables in `rules:changes` expressions to determine when +to add jobs to a pipeline: + +```yaml +docker build: + variables: + DOCKERFILES_DIR: 'path/to/files/' + script: docker build -t my-image:$CI_COMMIT_REF_SLUG . + rules: + - changes: + - $DOCKERFILES_DIR/* +``` + +You can use the `$` character for both variables and paths. For example, if the +`$DOCKERFILES_DIR` variable exists, its value is used. If it does not exist, the +`$` is interpreted as being part of a path. + ## Specify when jobs run with `only` and `except` You can use [`only`](../yaml/README.md#only--except) and [`except`](../yaml/README.md#only--except) @@ -73,7 +344,7 @@ end-to-end: - $CI_COMMIT_MESSAGE =~ /skip-end-to-end-tests/ ``` -You can use [parentheses](../variables/README.md#parentheses) with `&&` and `||` +You can use [parentheses](#group-variable-expressions-together-with-parentheses) with `&&` and `||` to build more complicated variable expressions: ```yaml @@ -82,9 +353,12 @@ job1: - echo This rule uses parentheses. only: variables: - - ($CI_COMMIT_BRANCH == "master" || $CI_COMMIT_BRANCH == "develop") && $MY_VARIABLE + - ($CI_COMMIT_BRANCH == "main" || $CI_COMMIT_BRANCH == "develop") && $MY_VARIABLE ``` +When multiple entries are specified in `only:variables`, the job runs when at least one of them evaluates to `true`. +You can use `&&` in a single entry when multiple conditions must be satisfied at the same time. + ### `only:changes` / `except:changes` examples You can skip a job if a change is detected in any file with a @@ -315,3 +589,114 @@ this feature flag again: ```ruby Feature.enable(:allow_unsafe_ruby_regexp) ``` + +## CI/CD variable expressions + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/37397) in GitLab 10.7 for [the `only` and `except` CI keywords](../yaml/README.md#onlyvariables--exceptvariables) +> - [Expanded](https://gitlab.com/gitlab-org/gitlab/-/issues/27863) in GitLab 12.3 with [the `rules` keyword](../yaml/README.md#rules) + +Use variable expressions to control which jobs are created in a pipeline after changes +are pushed to GitLab. You can use variable expressions with: + +- [`rules:if`](../yaml/README.md#rules). +- [`only:variables` and `except:variables`](../yaml/README.md#onlyvariables--exceptvariables). + +For example, with `rules:if`: + +```yaml +job1: + variables: + VAR1: "variable1" + script: + - echo "Test variable comparison + rules: + - if: $VAR1 == "variable1" +``` + +### Compare a variable to a string + +You can use the equality operators `==` and `!=` to compare a variable with a +string. Both single quotes and double quotes are valid. The order doesn't matter, +so the variable can be first, or the string can be first. For example: + +- `if: $VARIABLE == "some value"` +- `if: $VARIABLE != "some value"` +- `if: "some value" == $VARIABLE` + +### Compare two variables + +You can compare the values of two variables. For example: + +- `if: $VARIABLE_1 == $VARIABLE_2` +- `if: $VARIABLE_1 != $VARIABLE_2` + +### Check if a variable is undefined + +You can compare a variable to the `null` keyword to see if it is defined. For example: + +- `if: $VARIABLE == null` +- `if: $VARIABLE != null` + +### Check if a variable is empty + +You can check if a variable is defined but empty. For example: + +- `if: $VARIABLE == ""` +- `if: $VARIABLE != ""` + +### Check if a variable exists + +You can check for the existence of a variable by using just the variable name in +the expression. The variable must not be empty. For example: + +- `if: $VARIABLE` + +### Compare a variable to a regex pattern + +You can do regex pattern matching on variable values with the `=~` and `!~` operators. +Variable pattern matching with regular expressions uses the +[RE2 regular expression syntax](https://github.com/google/re2/wiki/Syntax). + +Expressions evaluate as `true` if: + +- Matches are found when using `=~`. +- Matches are *not* found when using `!~`. + +For example: + +- `$VARIABLE =~ /^content.*/` +- `$VARIABLE_1 !~ /^content.*/` + +Pattern matching is case-sensitive by default. Use the `i` flag modifier to make a +pattern case-insensitive. For example: `/pattern/i`. + +### Join variable expressions together with `&&` or `||` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/62867) in GitLab 12.0 + +You can join multiple expressions using `&&` (and) or `||` (or), for example: + +- `$VARIABLE1 =~ /^content.*/ && $VARIABLE2 == "something"` +- `$VARIABLE1 =~ /^content.*/ && $VARIABLE2 =~ /thing$/ && $VARIABLE3` +- `$VARIABLE1 =~ /^content.*/ || $VARIABLE2 =~ /thing$/ && $VARIABLE3` + +The precedence of operators follows the [Ruby 2.5 standard](https://ruby-doc.org/core-2.5.0/doc/syntax/precedence_rdoc.html), +so `&&` is evaluated before `||`. + +#### Group variable expressions together with parentheses + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230938) in GitLab 13.3. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/238174) in GitLab 13.5. + +You can use parentheses to group expressions together. Parentheses take precedence over +`&&` and `||`, so expressions enclosed in parentheses are evaluated first, and the +result is used for the rest of the expression. + +You can nest parentheses to create complex conditions, and the inner-most expressions +in parentheses are evaluated first. + +For example: + +- `($VARIABLE1 =~ /^content.*/ || $VARIABLE2) && ($VARIABLE3 =~ /thing$/ || $VARIABLE4)` +- `($VARIABLE1 =~ /^content.*/ || $VARIABLE2 =~ /thing$/) && $VARIABLE3` +- `$CI_COMMIT_BRANCH == "my-branch" || (($VARIABLE1 == "thing" || $VARIABLE2 == "thing") && $VARIABLE3)` |