diff options
author | Tomasz Maczukin <tomasz@gitlab.com> | 2018-05-29 20:33:48 +0300 |
---|---|---|
committer | Achilleas Pipinellis <axil@gitlab.com> | 2018-05-29 20:33:48 +0300 |
commit | 770504baeed6333115cb3672b9369f7f738f4d28 (patch) | |
tree | e7e6877b3900d1a5698df348ee976859382ae727 /doc/ci | |
parent | a30e6b819bcf87c5297fe68bb54dd1751f7e0bfb (diff) |
Add documentation about variables usage in GitLab CI
Diffstat (limited to 'doc/ci')
-rw-r--r-- | doc/ci/README.md | 6 | ||||
-rw-r--r-- | doc/ci/environments.md | 25 | ||||
-rw-r--r-- | doc/ci/variables/README.md | 43 | ||||
-rw-r--r-- | doc/ci/variables/where_variables_can_be_used.md | 113 |
4 files changed, 135 insertions, 52 deletions
diff --git a/doc/ci/README.md b/doc/ci/README.md index 8d1d72c2a2b..7666219acb0 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -19,7 +19,7 @@ Here's some info we've gathered to get you started. The first steps towards your GitLab CI/CD journey. - [Getting started with GitLab CI/CD](quick_start/README.md): understand how GitLab CI/CD works. -- GitLab CI/CD configuration file: [`.gitlab-ci.yml`](yaml/README.md) - Learn all about the ins and outs of `.gitlab-ci.yml`. +- [GitLab CI/CD configuration file: `.gitlab-ci.yml`](yaml/README.md) - Learn all about the ins and outs of `.gitlab-ci.yml`. - [Pipelines and jobs](pipelines.md): configure your GitLab CI/CD pipelines to build, test, and deploy your application. - Runners: The [GitLab Runner](https://docs.gitlab.com/runner/) is responsible by running the jobs in your CI/CD pipeline. On GitLab.com, Shared Runners are enabled by default, so you don't need to set up anything to start to use them with GitLab CI/CD. @@ -46,7 +46,9 @@ you don't need to set up anything to start to use them with GitLab CI/CD. ## Exploring GitLab CI/CD - [CI/CD Variables](variables/README.md) - Learn how to use variables defined in - your `.gitlab-ci.yml` or secured ones defined in your project's settings + your `.gitlab-ci.yml` or the ones defined in your project's settings + - [Where variables can be used](variables/where_variables_can_be_used.md) - A + deeper look on where and how the CI/CD variables can be used - **The permissions model** - Learn about the access levels a user can have for performing certain CI actions - [User permissions](../user/permissions.md#gitlab-ci) diff --git a/doc/ci/environments.md b/doc/ci/environments.md index 0d54f375c93..7f034409580 100644 --- a/doc/ci/environments.md +++ b/doc/ci/environments.md @@ -246,23 +246,14 @@ As the name suggests, it is possible to create environments on the fly by just declaring their names dynamically in `.gitlab-ci.yml`. Dynamic environments is the basis of [Review apps](review_apps/index.md). ->**Note:** -The `name` and `url` parameters can use most of the defined CI variables, -including predefined, secure variables and `.gitlab-ci.yml` -[`variables`](yaml/README.md#variables). You however cannot use variables -defined under `script` or on the Runner's side. There are other variables that -are unsupported in environment name context: -- `CI_PIPELINE_ID` -- `CI_JOB_ID` -- `CI_JOB_TOKEN` -- `CI_BUILD_ID` -- `CI_BUILD_TOKEN` -- `CI_REGISTRY_USER` -- `CI_REGISTRY_PASSWORD` -- `CI_REPOSITORY_URL` -- `CI_ENVIRONMENT_URL` -- `CI_DEPLOY_USER` -- `CI_DEPLOY_PASSWORD` +NOTE: **Note:** +The `name` and `url` parameters can use most of the CI/CD variables, +including [predefined](variables/README.md#predefined-variables-environment-variables), +[secret](variables/README.md#secret-variables) and +[`.gitlab-ci.yml` variables](yaml/README.md#variables). You however cannot use variables +defined under `script` or on the Runner's side. There are also other variables that +are unsupported in the context of `environment:name`. You can read more about +[where variables can be used](variables/where_variables_can_be_used.md). GitLab Runner exposes various [environment variables][variables] when a job runs, and as such, you can use them as environment names. Let's add another job in diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index 683846a536b..f10423b92cf 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -22,6 +22,12 @@ For example, if you define `API_TOKEN=secure` as a secret variable and `API_TOKEN=yaml` in your `.gitlab-ci.yml`, the `API_TOKEN` will take the value `secure` as the secret variables are higher in the chain. +## Unsupported variables + +There are cases where some variables cannot be used in the context of a +`.gitlab-ci.yml` definition (for example under `script`). Read more +about which variables are [not supported](where_variables_can_be_used.md). + ## Predefined variables (Environment variables) Some of the predefined environment variables are available only if a minimum @@ -36,6 +42,7 @@ future GitLab releases.** | Variable | GitLab | Runner | Description | |-------------------------------- |--------|--------|-------------| +| **ARTIFACT_DOWNLOAD_ATTEMPTS** | 8.15 | 1.9 | Number of attempts to download artifacts running a job | | **CI** | all | 0.4 | Mark that job is executed in CI environment | | **CI_COMMIT_REF_NAME** | 9.0 | all | The branch or tag name for which project is built | | **CI_COMMIT_REF_SLUG** | 9.0 | all | `$CI_COMMIT_REF_NAME` lowercased, shortened to 63 bytes, and with everything except `0-9` and `a-z` replaced with `-`. No leading / trailing `-`. Use in URLs, host names and domain names. | @@ -46,6 +53,8 @@ future GitLab releases.** | **CI_COMMIT_DESCRIPTION** | 10.8 | all | The description of the commit: the message without first line, if the title is shorter than 100 characters; full message in other case. | | **CI_CONFIG_PATH** | 9.4 | 0.5 | The path to CI config file. Defaults to `.gitlab-ci.yml` | | **CI_DEBUG_TRACE** | all | 1.7 | Whether [debug tracing](#debug-tracing) is enabled | +| **CI_DEPLOY_USER** | 10.8 | all | Authentication username of the [GitLab Deploy Token][gitlab-deploy-token], only present if the Project has one related.| +| **CI_DEPLOY_PASSWORD** | 10.8 | all | Authentication password of the [GitLab Deploy Token][gitlab-deploy-token], only present if the Project has one related.| | **CI_DISPOSABLE_ENVIRONMENT** | all | 10.1 | Marks that the job is executed in a disposable environment (something that is created only for this job and disposed of/destroyed after the execution - all executors except `shell` and `ssh`). If the environment is disposable, it is set to true, otherwise it is not defined at all. | | **CI_ENVIRONMENT_NAME** | 8.15 | all | The name of the environment for this job | | **CI_ENVIRONMENT_SLUG** | 8.15 | all | A simplified version of the environment name, suitable for inclusion in DNS, URLs, Kubernetes labels, etc. | @@ -82,16 +91,13 @@ future GitLab releases.** | **CI_SERVER_REVISION** | all | all | GitLab revision that is used to schedule jobs | | **CI_SERVER_VERSION** | all | all | GitLab version that is used to schedule jobs | | **CI_SHARED_ENVIRONMENT** | all | 10.1 | Marks that the job is executed in a shared environment (something that is persisted across CI invocations like `shell` or `ssh` executor). If the environment is shared, it is set to true, otherwise it is not defined at all. | -| **ARTIFACT_DOWNLOAD_ATTEMPTS** | 8.15 | 1.9 | Number of attempts to download artifacts running a job | | **GET_SOURCES_ATTEMPTS** | 8.15 | 1.9 | Number of attempts to fetch sources running a job | | **GITLAB_CI** | all | all | Mark that job is executed in GitLab CI environment | -| **GITLAB_USER_ID** | 8.12 | all | The id of the user who started the job | | **GITLAB_USER_EMAIL** | 8.12 | all | The email of the user who started the job | +| **GITLAB_USER_ID** | 8.12 | all | The id of the user who started the job | | **GITLAB_USER_LOGIN** | 10.0 | all | The login username of the user who started the job | | **GITLAB_USER_NAME** | 10.0 | all | The real name of the user who started the job | | **RESTORE_CACHE_ATTEMPTS** | 8.15 | 1.9 | Number of attempts to restore the cache running a job | -| **CI_DEPLOY_USER** | 10.8 | all | Authentication username of the [GitLab Deploy Token][gitlab-deploy-token], only present if the Project has one related.| -| **CI_DEPLOY_PASSWORD** | 10.8 | all | Authentication password of the [GitLab Deploy Token][gitlab-deploy-token], only present if the Project has one related.| ## 9.0 Renaming @@ -540,34 +546,6 @@ Below you can find supported syntax reference: Pattern matching is case-sensitive by default. Use `i` flag modifier, like `/pattern/i` to make a pattern case-insensitive. -### Unsupported predefined variables - -Because GitLab evaluates variables before creating jobs, we do not support a -few variables that depend on persistence layer, like `$CI_JOB_ID`. - -Environments (like `production` or `staging`) are also being created based on -what jobs pipeline consists of, thus some environment-specific variables are -not supported as well. - -We do not support variables containing tokens because of security reasons. - -You can find a full list of unsupported variables below: - -- `CI_PIPELINE_ID` -- `CI_JOB_ID` -- `CI_JOB_TOKEN` -- `CI_BUILD_ID` -- `CI_BUILD_TOKEN` -- `CI_REGISTRY_USER` -- `CI_REGISTRY_PASSWORD` -- `CI_REPOSITORY_URL` -- `CI_ENVIRONMENT_URL` -- `CI_DEPLOY_USER` -- `CI_DEPLOY_PASSWORD` - -These variables are also not supported in a context of a -[dynamic environment name][dynamic-environments]. - [ce-13784]: https://gitlab.com/gitlab-org/gitlab-ce/issues/13784 "Simple protection of CI secret variables" [eep]: https://about.gitlab.com/products/ "Available only in GitLab Premium" [envs]: ../environments.md @@ -579,5 +557,4 @@ These variables are also not supported in a context of a [triggers]: ../triggers/README.md#pass-job-variables-to-a-trigger [subgroups]: ../../user/group/subgroups/index.md [builds-policies]: ../yaml/README.md#only-and-except-complex -[dynamic-environments]: ../environments.md#dynamic-environments [gitlab-deploy-token]: ../../user/project/deploy_tokens/index.md#gitlab-deploy-token diff --git a/doc/ci/variables/where_variables_can_be_used.md b/doc/ci/variables/where_variables_can_be_used.md new file mode 100644 index 00000000000..9800784d918 --- /dev/null +++ b/doc/ci/variables/where_variables_can_be_used.md @@ -0,0 +1,113 @@ +# Where variables can be used + +As it's described in the [CI/CD variables](README.md) docs, you can +define many different variables. Some of them can be used for all GitLab CI/CD +features, but some of them are more or less limited. + +This document describes where and how the different types of variables can be used. + +## Variables usage + +There are basically two places where you can use any defined variables: + +1. On GitLab's side there's `.gitlab-ci.yml` +1. On the Runner's side there's `config.toml` + +### `.gitlab-ci.yml` file + +| Definition | Can be expanded? | Expansion place | Description | +|--------------------------------------|-------------------|-----------------|--------------| +| `environment:url` | yes | GitLab | The variable expansion is made by GitLab's [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism).<ul><li>**Supported:** all variables defined for a job (secret variables, variables from `.gitlab-ci.yml`, variables from triggers, variables from pipeline schedules)</li><li>**Not suported:** variables defined in Runner's `config.toml` and variables created in job's `script`</li></ul> | +| `environment:name` | yes | GitLab | Similar to `environment:url`, but the variables expansion **doesn't support**: <ul><li>variables that are based on the environment's name (`CI_ENVIRONMENT_NAME`, `CI_ENVIRONMENT_SLUG`)</li><li>any other variables related to environment (currently only `CI_ENVIRONMENT_URL`)</li><li>[persisted variables](#persisted-variables)</li></ul> | +| `variables` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | +| `image` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | +| `services:[]` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | +| `services:[]:name` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | +| `cache:key` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | +| `artifacts:name` | yes | Runner | The variable expansion is made by GitLab Runner's shell environment | +| `script`, `before_script`, `after_script` | yes | Script execution shell | The variable expansion is made by the [execution shell environment](#execution-shell-environment) | +| `only:variables:[]`, `except:variables:[]` | no | n/a | The variable must be in the form of `$variable`.<br/>**Not supported:**<ul><li>variables that are based on the environment's name (`CI_ENVIRONMENT_NAME`, `CI_ENVIRONMENT_SLUG`)</li><li>any other variables related to environment (currently only `CI_ENVIRONMENT_URL`)</li><li>[persisted variables](#persisted-variables)</li></ul> | + +### `config.toml` file + +NOTE: **Note:** +You can read more about `config.toml` in the [Runner's docs](https://docs.gitlab.com/runner/configuration/advanced-configuration.html). + +| Definition | Can be expanded? | Description | +|--------------------------------------|------------------|-------------| +| `runners.environment` | yes | The variable expansion is made by the Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | +| `runners.kubernetes.pod_labels` | yes | The Variable expansion is made by the Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | +| `runners.kubernetes.pod_annotations` | yes | The Variable expansion is made by the Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | + +## Expansion mechanisms + +There are three expansion mechanisms: + +- GitLab +- GitLab Runner +- Execution shell environment + +### GitLab internal variable expansion mechanism + +The expanded part needs to be in a form of `$variable`, or `${variable}` or `%variable%`. +Each form is handled in the same way, no matter which OS/shell will finally handle the job, +since the expansion is done in GitLab before any Runner will get the job. + +### GitLab Runner internal variable expansion mechanism + +- **Supported:** secret variables, `.gitlab-ci.yml` variables, `config.toml` variables, and + variables from triggers and pipeline schedules +- **Not supported:** variables defined inside of scripts (e.g., `export MY_VARIABLE="test"`) + +The Runner uses Go's `os.Expand()` method for variable expansion. It means that it will handle +only variables defined as `$variable` and `${variable}`. What's also important, is that +the expansion is done only once, so nested variables may or may not work, depending on the +ordering of variables definitions. + +### Execution shell environment + +This is an expansion that takes place during the `script` execution. +How it works depends on the used shell (bash/sh/cmd/PowerShell). For example, if the job's +`script` contains a line `echo $MY_VARIABLE-${MY_VARIABLE_2}`, it should be properly handled +by bash/sh (leaving empty strings or some values depending whether the variables were +defined or not), but will not work with Windows' cmd/PowerShell, since these shells +are using a different variables syntax. + +**Supported:** + +- The `script` may use all available variables that are default for the shell (e.g., `$PATH` which + should be present in all bash/sh shells) and all variables defined by GitLab CI/CD (secret variables, + `.gitlab-ci.yml` variables, `config.toml` variables, and variables from triggers and pipeline schedules). +- The `script` may also use all variables defined in the lines before. So, for example, if you define + a variable `export MY_VARIABLE="test"`: + + - in `before_script`, it will work in the following lines of `before_script` and + all lines of the related `script` + - in `script`, it will work in the following lines of `script` + - in `after_script`, it will work in following lines of `after_script` + +## Persisted variables + +NOTE: **Note:** +Some of the persisted variables contain tokens and cannot be used by some definitions +due to security reasons. + +The following variables are known as "persisted": + +- `CI_PIPELINE_ID` +- `CI_JOB_ID` +- `CI_JOB_TOKEN` +- `CI_BUILD_ID` +- `CI_BUILD_TOKEN` +- `CI_REGISTRY_USER` +- `CI_REGISTRY_PASSWORD` +- `CI_REPOSITORY_URL` +- `CI_DEPLOY_USER` +- `CI_DEPLOY_PASSWORD` + +They are: + +- **supported** for all definitions as [described in the table](#gitlab-ci-yml-file) where the "Expansion place" is "Runner" +- **not supported:** + - by the definitions [described in the table](#gitlab-ci-yml-file) where the "Expansion place" is "GitLab" + - in the `only` and `except` [variables expressions](README.md#variables-expressions) |