diff options
Diffstat (limited to 'doc/ci')
43 files changed, 482 insertions, 239 deletions
diff --git a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md index a0665e1c054..ceb56b01dcd 100644 --- a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md +++ b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md @@ -122,7 +122,7 @@ To use GitLab CI/CD with a Bitbucket Cloud repository: 1. In Bitbucket, create a `.gitlab-ci.yml` file to use the script to push pipeline success and failures to Bitbucket. Similar to the script added above, - this file is copied to the GitLab repo as part of the mirroring process. + this file is copied to the GitLab repository as part of the mirroring process. ```yaml stages: diff --git a/doc/ci/cloud_services/azure/index.md b/doc/ci/cloud_services/azure/index.md index b2f78648be9..b846ee4b792 100644 --- a/doc/ci/cloud_services/azure/index.md +++ b/doc/ci/cloud_services/azure/index.md @@ -17,7 +17,7 @@ Prerequisites: - Access to an existing Azure Subscription with `Owner` access level. - Access to the corresponding Azure Active Directory Tenant with at least the `Application Developer` access level. - A local installation of the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli). - Alternatively, you can follow all the steps below with the [Azure Cloud Shell](https://shell.azure.com/). + Alternatively, you can follow all the steps below with the [Azure Cloud Shell](https://portal.azure.com/#cloudshell/). - A GitLab project. To complete this tutorial: diff --git a/doc/ci/directed_acyclic_graph/index.md b/doc/ci/directed_acyclic_graph/index.md index 39f45471021..49bce75e183 100644 --- a/doc/ci/directed_acyclic_graph/index.md +++ b/doc/ci/directed_acyclic_graph/index.md @@ -10,7 +10,7 @@ type: reference > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/47063) in GitLab 12.2. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/206902) in GitLab 12.10. -A [directed acyclic graph](https://www.techopedia.com/definition/5739/directed-acyclic-graph-dag) can be +A [directed acyclic graph](https://en.wikipedia.org/wiki/Directed_acyclic_graph) can be used in the context of a CI/CD pipeline to build relationships between jobs such that execution is performed in the quickest possible manner, regardless how stages may be set up. @@ -38,10 +38,10 @@ It has a pipeline that looks like the following: | build | test | deploy | | ----- | ---- | ------ | -| build_a | test_a | deploy_a | -| build_b | test_b | deploy_b | -| build_c | test_c | deploy_c | -| build_d | test_d | deploy_d | +| `build_a` | `test_a` | `deploy_a` | +| `build_b` | `test_b` | `deploy_b` | +| `build_c` | `test_c` | `deploy_c` | +| `build_d` | `test_d` | `deploy_d` | Using a DAG, you can relate the `_a` jobs to each other separately from the `_b` jobs, and even if service `a` takes a very long time to build, service `b` doesn't diff --git a/doc/ci/environments/deployment_approvals.md b/doc/ci/environments/deployment_approvals.md index d7fa31b583b..a4815a85bc1 100644 --- a/doc/ci/environments/deployment_approvals.md +++ b/doc/ci/environments/deployment_approvals.md @@ -134,6 +134,8 @@ To approve or reject a deployment to a protected environment using the UI: 1. On the left sidebar, select **Deployments > Environments**. 1. Select the environment's name. 1. In the deployment's row, select **Approval options** (**{thumb-up}**). + Before approving or rejecting the deployment, you can view the number of approvals granted and + remaining, also who has approved or rejected it. 1. Optional. Add a comment which describes your reason for approving or rejecting the deployment. 1. Select **Approve** or **Reject**. @@ -154,6 +156,29 @@ curl --data "status=approved&comment=Looks good to me" \ --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/deployments/1/approval" ``` +### View the approval details of a deployment + +Prerequisites: + +- Permission to deploy to the protected environment. + +A deployment to a protected environment can only proceed after all required approvals have been +granted. + +To view the approval details of a deployment: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Deployments > Environments**. +1. Select the environment's name. +1. In the deployment's row, select **Approval options** (**{thumb-up}**). + +The approval status details are shown: + +- Eligible approvers +- Number of approvals granted, and number of approvals required +- Users who have granted approval +- History of approvals or rejections + ## How to see blocked deployments ### Using the UI diff --git a/doc/ci/environments/external_deployment_tools.md b/doc/ci/environments/external_deployment_tools.md new file mode 100644 index 00000000000..ff3172f0e02 --- /dev/null +++ b/doc/ci/environments/external_deployment_tools.md @@ -0,0 +1,88 @@ +--- +stage: Release +group: Release +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: reference +--- + +# Track deployments of an external deployment tool **(FREE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22513) in GitLab 12.5. + +While GitLab offers a [built-in deployment solution](index.md), you might prefer to use an external deployment tool, such as Heroku or ArgoCD. +GitLab can receive deployment events from these external tools and allows you to track the deployments within GitLab. +For example, the following features are available by setting up tracking: + +- [See when an merge request has been deployed, and to which environment](../../user/project/merge_requests/widgets.md#post-merge-pipeline-status). +- [Filter merge requests by environment or deployment date](../../user/project/merge_requests/index.md#filter-merge-requests-by-environment-or-deployment-date). +- [DevOps Research and Assessment (DORA) metrics](../../user/analytics/dora_metrics.md). +- [View environments and deployments](index.md#view-environments-and-deployments). +- [Track newly included merge requests per deployment](index.md#track-newly-included-merge-requests-per-deployment). + +NOTE: +Some of the features are not available because GitLab can't authorize and leverage those external deployments, including +[Protected Environments](protected_environments.md), [Deployment Approvals](deployment_approvals.md), [Deployment safety](deployment_safety.md), and [Environment rollback](index.md#environment-rollback). + +## How to set up deployment tracking + +External deployment tools usually offer a [webhook](https://en.wikipedia.org/wiki/Webhook) to execute an additional API request when deployment state is changed. +You can configure your tool to make a request to the GitLab [Deployment API](../../api/deployments.md). Here is an overview of the event and API request flow: + +- When a deployment starts running, [create a deployment with `running` status](../../api/deployments.md#create-a-deployment). +- When a deployment succeeds, [update the deployment status to `success`](../../api/deployments.md#update-a-deployment). +- When a deployment fails, [update the deployment status to `failed`](../../api/deployments.md#update-a-deployment). + +NOTE: +You can create a [project access token](../../user/project/settings/project_access_tokens.md) for the GitLab API authentication. + +### Example: Track deployments of ArgoCD + +You can use [ArgoCD webhook](https://argocd-notifications.readthedocs.io/en/stable/services/webhook/) to send deployment events to GitLab Deployment API. +Here is an example setup that creates a `success` deployment record in GitLab when ArgoCD successfully deploys a new revision: + +1. Create a new webhook. You can save the following manifest file and apply it by `kubectl apply -n argocd -f <manifiest-file-path>`: + + ```yaml + apiVersion: v1 + kind: ConfigMap + metadata: + name: argocd-notifications-cm + data: + trigger.on-deployed: | + - description: Application is synced and healthy. Triggered once per commit. + oncePer: app.status.sync.revision + send: + - gitlab-deployment-status + when: app.status.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy' + template.gitlab-deployment-status: | + webhook: + gitlab: + method: POST + path: /projects/<your-project-id>/deployments + body: | + { + "status": "success", + "environment": "production", + "sha": "{{.app.status.operationState.operation.sync.revision}}", + "ref": "main", + "tag": "false" + } + service.webhook.gitlab: | + url: https://gitlab.com/api/v4 + headers: + - name: PRIVATE-TOKEN + value: <your-access-token> + - name: Content-type + value: application/json + ``` + +1. Create a new subscription in your application: + + ```shell + kubectl patch app <your-app-name> -n argocd -p '{"metadata": {"annotations": {"notifications.argoproj.io/subscribe.on-deployed.gitlab":""}}}' --type merge + ``` + +NOTE: +If a deployment wasn't created as expected, you can troubleshoot with [`argocd-notifications` tool](https://argocd-notifications.readthedocs.io/en/stable/troubleshooting/). +For example, `argocd-notifications template notify gitlab-deployment-status <your-app-name> --recipient gitlab:argocd-notifications` +triggers API request immediately and renders an error message from GitLab API server if any. diff --git a/doc/ci/environments/index.md b/doc/ci/environments/index.md index c4672b9dc7e..0c412c85e47 100644 --- a/doc/ci/environments/index.md +++ b/doc/ci/environments/index.md @@ -45,15 +45,20 @@ Deployments show up in this list only after a deployment job has created them. ## Search environments -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10754) in GitLab 15.5. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10754) in GitLab 15.5. +> - [Searching environments within a folder](https://gitlab.com/gitlab-org/gitlab/-/issues/373850) was introduced in GitLab 15.7 with [Feature flag `enable_environments_search_within_folder`](https://gitlab.com/gitlab-org/gitlab/-/issues/382108). Enabled by default. To search environments by name: 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left sidebar, select **Deployments > Environments**. -1. In the search bar, enter your search term. Matching applies from the - beginning of the environment name. For example, `devel` matches the - environment name `development`, but `elop` does not. +1. In the search bar, enter your search term. + - The length of your **search term should be 3 or more characters**. + - Matching applies from the beginning of the environment name. + - For example, `devel` matches the environment name `development`, but `elop` does not. + - For environments with a folder name format, matching applies after the base folder name. + - For example when the name is `review/test-app`, search term `test` matches `review/test-app`. + - Also searching with the folder name prefixed like `review/test` matches `review/test-app`. ## Types of environments @@ -340,7 +345,7 @@ and displayed at a post-merge pipeline in [merge request pages](../../user/proje To activate this tracking, your environment must be configured in the following: -- [Environment name](../yaml/index.md#environmentname) is not foldered with `/` (that is, top-level/long-lived environments), _OR_ +- [Environment name](../yaml/index.md#environmentname) is not using folders with `/` (that is, top-level/long-lived environments), _OR_ - [Environment tier](#deployment-tier-of-environments) is either `production` or `staging`. Here are the example setups of [`environment` keyword](../yaml/index.md#environment) in `.gitlab-ci.yml`: @@ -609,6 +614,20 @@ Because `stop_review_app` is set to `auto_stop_in: 1 week`, if a merge request is inactive for more than a week, GitLab automatically triggers the `stop_review_app` job to stop the environment. +#### Stop an environment through the UI + +NOTE: +To trigger an `on_stop` action and manually stop an environment from the +Environments view, the stop and deploy jobs must be in the same +[`resource_group`](../yaml/index.md#resource_group). + +To stop an environment in the GitLab UI: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Deployments > Environments**. +1. Next to the environment you want to stop, select **Stop**. +1. On the confirmation dialog box, select **Stop environment**. + #### Multiple stop actions for an environment > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22456) in GitLab 14.10 [with a flag](../../administration/feature_flags.md) named `environment_multiple_stop_actions`. Disabled by default. @@ -977,6 +996,7 @@ Instead, you need to delete the old environment and create a new one: - [Environments Dashboard](../environments/environments_dashboard.md): View a summary of each environment's operational health. **(PREMIUM)** - [Deployment safety](deployment_safety.md#restrict-write-access-to-a-critical-environment): Secure your deployments. +- [Track deployments of an external deployment tool](external_deployment_tools.md): Use an external deployment tool instead of built-in deployment solution. ## Troubleshooting diff --git a/doc/ci/environments/protected_environments.md b/doc/ci/environments/protected_environments.md index 5c120da32a0..638bcf77967 100644 --- a/doc/ci/environments/protected_environments.md +++ b/doc/ci/environments/protected_environments.md @@ -132,13 +132,13 @@ they have the following privileges: Users granted access to a protected environment, but not push or merge access to the branch deployed to it, are only granted access to deploy the environment. -[Invited groups](../../user/project/members/share_project_with_groups.md#share-a-project-with-a-group-of-users) added +[Invited groups](../../user/project/members/share_project_with_groups.md#share-a-project-with-a-group) added to the project with [Reporter role](../../user/permissions.md#project-members-permissions), appear in the dropdown list for deployment-only access. To add deployment-only access: 1. Create a group with members who are granted to access to the protected environment, if it doesn't exist yet. -1. [Invite the group](../../user/project/members/share_project_with_groups.md#share-a-project-with-a-group-of-users) to the project with the Reporter role. +1. [Invite the group](../../user/project/members/share_project_with_groups.md#share-a-project-with-a-group) to the project with the Reporter role. 1. Follow the steps in [Protecting Environments](#protecting-environments). ## Modifying and unprotecting environments @@ -200,7 +200,7 @@ To maximize the effectiveness of group-level protected environments, [group-level memberships](../../user/group/index.md) must be correctly configured: -- Operators should be given at least the Owner role +- Operators should be given the Owner role for the top-level group. They can maintain CI/CD configurations for the higher environments (such as production) in the group-level settings page, which includes group-level protected environments, diff --git a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md index 7208caaccae..125ae3650c9 100644 --- a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md +++ b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md @@ -90,7 +90,7 @@ The JWT is encoded by using RS256 and signed with a dedicated private key. The e You can use this JWT and your instance's JWKS endpoint (`https://gitlab.example.com/-/jwks`) to authenticate with a Vault server that is configured to allow the JWT Authentication method for authentication. -When configuring roles in Vault, you can use [bound_claims](https://developer.hashicorp.com/vault/docs/auth/jwt#bound-claims) to match against the JWT's claims and restrict which secrets each CI job has access to. +When configuring roles in Vault, you can use [bound claims](https://developer.hashicorp.com/vault/docs/auth/jwt#bound-claims) to match against the JWT claims and restrict which secrets each CI/CD job has access to. To communicate with Vault, you can use either its CLI client or perform API requests (using `curl` or another client). @@ -180,10 +180,35 @@ $ vault write auth/jwt/role/myproject-production - <<EOF EOF ``` -This example uses [bound_claims](https://developer.hashicorp.com/vault/api-docs/auth/jwt#bound_claims) to specify that only a JWT with matching values for the specified claims is allowed to authenticate. +This example uses [bound claims](https://developer.hashicorp.com/vault/api-docs/auth/jwt#bound_claims) to specify that only a JWT with matching values for the specified claims is allowed to authenticate. Combined with [protected branches](../../../user/project/protected_branches.md), you can restrict who is able to authenticate and read the secrets. +To use the same policy for a list of projects, use `namespace_id`: + +```json +"bound_claims": { + "namespace_id": ["12", "22", "37"] +} +``` + +Any of the claims [included in the JWT](#how-it-works) can be matched against a list of values +in the bound claims. For example: + +```json +"bound_claims": { + "user_login": ["alice", "bob", "mallory"] +} + +"bound_claims": { + "ref": ["main", "develop", "test"] +} + +"bound_claims": { + "project_id": ["12", "22", "37"] +} +``` + [`token_explicit_max_ttl`](https://developer.hashicorp.com/vault/api-docs/auth/jwt#token_explicit_max_ttl) specifies that the token issued by Vault, upon successful authentication, has a hard lifetime limit of 60 seconds. [`user_claim`](https://developer.hashicorp.com/vault/api-docs/auth/jwt#user_claim) specifies the name for the Identity alias created by Vault upon a successful login. @@ -225,7 +250,7 @@ $ vault write auth/jwt/config \ bound_issuer="gitlab.example.com" ``` -[bound_issuer](https://developer.hashicorp.com/vault/api-docs/auth/jwt#bound_issuer) specifies that only a JWT with the issuer (that is, the `iss` claim) set to `gitlab.example.com` can use this method to authenticate, and that the JWKS endpoint (`https://gitlab.example.com/-/jwks`) should be used to validate the token. +[`bound_issuer`](https://developer.hashicorp.com/vault/api-docs/auth/jwt#bound_issuer) specifies that only a JWT with the issuer (that is, the `iss` claim) set to `gitlab.example.com` can use this method to authenticate, and that the JWKS endpoint (`https://gitlab.example.com/-/jwks`) should be used to validate the token. For the full list of available configuration options, see Vault's [API documentation](https://developer.hashicorp.com/vault/api-docs/auth/jwt#configure). @@ -256,7 +281,7 @@ NOTE: If you're using a Vault instance provided by HashiCorp Cloud Platform, you need to export the `VAULT_NAMESPACE` variable. Its default value is `admin`. - + The following job is able to authenticate using the `myproject-production` role and read secrets under `/secret/myproject/production/`: @@ -279,14 +304,14 @@ read_secrets: - echo $PASSWORD ``` - + ### Limit token access to Vault secrets You can control `CI_JOB_JWT` access to Vault secrets by using Vault protections and GitLab features. For example, restrict the token by: -- Using Vault [bound_claims](https://developer.hashicorp.com/vault/docs/auth/jwt#bound-claims) +- Using Vault [bound claims](https://developer.hashicorp.com/vault/docs/auth/jwt#bound-claims) for specific groups using `group_claim`. - Hard coding values for Vault bound claims based on the `user_login` and `user_email` of specific users. diff --git a/doc/ci/examples/deployment/composer-npm-deploy.md b/doc/ci/examples/deployment/composer-npm-deploy.md index a603207aef7..533b6519d9a 100644 --- a/doc/ci/examples/deployment/composer-npm-deploy.md +++ b/doc/ci/examples/deployment/composer-npm-deploy.md @@ -120,7 +120,7 @@ Therefore, for a production environment we use additional steps to ensure that a ## Where to go next -Since this was a WordPress project, I gave real life code snippets. Some further ideas you can pursue: +Since this was a WordPress project, it includes real code snippets. Some further ideas you can pursue: - Having a slightly different script for the default branch allows you to deploy to a production server from that branch and to a stage server from any other branches. - Instead of pushing it live, you can push it to WordPress official repository. diff --git a/doc/ci/examples/index.md b/doc/ci/examples/index.md index 361061d0d75..c8ad653e41f 100644 --- a/doc/ci/examples/index.md +++ b/doc/ci/examples/index.md @@ -33,7 +33,7 @@ The following table lists examples with step-by-step tutorials that are containe | npm with semantic-release | [Publish npm packages to the GitLab Package Registry using semantic-release](semantic-release.md). | | PHP with Laravel, Envoy | [Test and deploy Laravel applications with GitLab CI/CD and Envoy](laravel_with_gitlab_and_envoy/index.md). | | PHP with npm, SCP | [Running Composer and npm scripts with deployment via SCP in GitLab CI/CD](deployment/composer-npm-deploy.md). | -| PHP with PHPunit, `atoum` | [Testing PHP projects](php.md). | +| PHP with PHPUnit, `atoum` | [Testing PHP projects](php.md). | | Secrets management with Vault | [Authenticating and Reading Secrets With HashiCorp Vault](authenticating-with-hashicorp-vault/index.md). | ### Contributed examples diff --git a/doc/ci/examples/php.md b/doc/ci/examples/php.md index 2d0c6382dd8..a83bcf69491 100644 --- a/doc/ci/examples/php.md +++ b/doc/ci/examples/php.md @@ -246,7 +246,7 @@ the [SSH keys](../ssh_keys/index.md) to be able to clone it. ## Use databases or other services Most of the time, you need a running database for your tests to be able to -run. If you're using the Docker executor, you can leverage Docker's ability to +run. If you're using the Docker executor, you can leverage Docker to link to other containers. With GitLab Runner, this can be achieved by defining a `service`. diff --git a/doc/ci/examples/semantic-release.md b/doc/ci/examples/semantic-release.md index 8f0321517ab..1fa526e787a 100644 --- a/doc/ci/examples/semantic-release.md +++ b/doc/ci/examples/semantic-release.md @@ -13,7 +13,7 @@ You can also view or fork the complete [example source](https://gitlab.com/gitla ## Initialize the module 1. Open a terminal and navigate to the project's repository. -1. Run `npm init`. Name the module according to [the Package Registry's naming conventions](../../user/packages/npm_registry/index.md#package-naming-convention). For example, if the project's path is `gitlab-examples/semantic-release-npm`, name the module `@gitlab-examples/semantic-release-npm`. +1. Run `npm init`. Name the module according to [the Package Registry's naming conventions](../../user/packages/npm_registry/index.md#naming-convention). For example, if the project's path is `gitlab-examples/semantic-release-npm`, name the module `@gitlab-examples/semantic-release-npm`. 1. Install the following npm packages: @@ -89,6 +89,7 @@ The default `before_script` generates a temporary `.npmrc` that is used to authe As part of publishing a package, semantic-release increases the version number in `package.json`. For semantic-release to commit this change and push it back to GitLab, the pipeline requires a custom CI/CD variable named `GITLAB_TOKEN`. To create this variable: <!-- markdownlint-disable MD044 --> + 1. On the top bar, on the top right, select your avatar. 1. On the left sidebar, select **Access Tokens**. 1. In the **Token name** box, enter a token name. diff --git a/doc/ci/git_submodules.md b/doc/ci/git_submodules.md index 68cbff21fae..0f206b3fceb 100644 --- a/doc/ci/git_submodules.md +++ b/doc/ci/git_submodules.md @@ -68,7 +68,7 @@ To make submodules work correctly in CI/CD jobs: ``` 1. You can filter or exclude specific submodules to control which submodules will be synced using - [`GIT_SUBMODULE_PATHS`](runners/configure_runners.md#git-submodule-paths). + [`GIT_SUBMODULE_PATHS`](runners/configure_runners.md#sync-or-exclude-specific-submodules-from-ci-jobs). ```yaml variables: @@ -83,14 +83,6 @@ To make submodules work correctly in CI/CD jobs: GIT_SUBMODULE_STRATEGY: recursive GIT_SUBMODULE_UPDATE_FLAGS: --jobs 4 ``` - -1. You can set the [GIT_SUBMODULE_PATHS](runners/configure_runners.md#sync-or-exclude-specific-submodules-from-ci-jobs) to explicitly ignore submodules during cloning: - - ```yaml - variables: - GIT_SUBMODULE_STRATEGY: recursive - GIT_SUBMODULE_PATHS: ':(exclude)submodule' - ``` If you use the [`CI_JOB_TOKEN`](jobs/ci_job_token.md) to clone a submodule in a pipeline job, the user executing the job must be assigned to a role that has diff --git a/doc/ci/jobs/ci_job_token.md b/doc/ci/jobs/ci_job_token.md index 1e7b389c84a..d95451a67dc 100644 --- a/doc/ci/jobs/ci_job_token.md +++ b/doc/ci/jobs/ci_job_token.md @@ -54,7 +54,7 @@ To make sure that this token doesn't leak, GitLab: To make sure that this token doesn't leak, you should also configure your [runners](../runners/index.md) to be secure. Avoid: -- Using Docker's `privileged` mode if the machines are re-used. +- Using Docker `privileged` mode if the machines are re-used. - Using the [`shell` executor](https://docs.gitlab.com/runner/executors/shell.html) when jobs run on the same machine. diff --git a/doc/ci/jobs/index.md b/doc/ci/jobs/index.md index 4eb8952dd73..15ec92a896e 100644 --- a/doc/ci/jobs/index.md +++ b/doc/ci/jobs/index.md @@ -110,6 +110,7 @@ You can't use these keywords as job names: - `true` - `false` - `nil` +- `pages:deploy` configured for a `deploy` stage Job names must be 255 characters or fewer. diff --git a/doc/ci/large_repositories/index.md b/doc/ci/large_repositories/index.md index c7ecc25dc44..5a553228ba1 100644 --- a/doc/ci/large_repositories/index.md +++ b/doc/ci/large_repositories/index.md @@ -196,18 +196,14 @@ Our pipeline is most performant if we use the following `.gitlab-ci.yml`: ```yaml variables: - GIT_DEPTH: 10 GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/$CI_PROJECT_NAME build: script: ls -al ``` -The above configures a: - -- Shallow clone of 10, to speed up subsequent `git fetch` commands. -- Custom clone path to make it possible to re-use worktrees between parent project and all forks - because we use the same clone path for all forks. +This YAML setting configures a custom clone path. This path makes it possible to re-use worktrees +between the parent project and forks because we use the same clone path for all forks. Why use `$CI_CONCURRENT_ID`? The main reason is to ensure that worktrees used are not conflicting between projects. The `$CI_CONCURRENT_ID` represents a unique identifier within the given executor. @@ -264,4 +260,4 @@ For very active repositories with a large number of references and files, you ca seeding the repository data also helps avoid `429 Too many requests` errors from Cloudflare. This error can occur if you have many runners behind a single, - NAT'ed IP address that pulls from GitLab.com. + IP address using NAT, that pulls from GitLab.com. diff --git a/doc/ci/migration/jenkins.md b/doc/ci/migration/jenkins.md index 4ba59e14811..235dd0e80ca 100644 --- a/doc/ci/migration/jenkins.md +++ b/doc/ci/migration/jenkins.md @@ -8,9 +8,9 @@ type: index, howto # Migrating from Jenkins **(FREE)** -A lot of GitLab users have successfully migrated to GitLab CI/CD from Jenkins. To make this -easier if you're just getting started, we've collected several resources here that you might find useful -before diving in. Think of this page as a "GitLab CI/CD for Jenkins Users" guide. +A lot of GitLab users have successfully migrated to GitLab CI/CD from Jenkins. +We've collected several resources here that you might find informative if you're just getting started. +Think of this page as a "GitLab CI/CD for Jenkins Users" guide. The following list of recommended steps was created after observing organizations that were able to quickly complete this migration: @@ -29,9 +29,10 @@ that were able to quickly complete this migration: 1. Check the [pipeline efficiency documentation](../pipelines/pipeline_efficiency.md) to learn how to make your GitLab CI/CD pipelines faster and more efficient. -For an example of how to convert a Jenkins pipeline into a GitLab CI/CD pipeline, -or how to use Auto DevOps to test your code automatically, watch the -[Migrating from Jenkins to GitLab](https://www.youtube.com/watch?v=RlEVGOpYF5Y) video. +Watch the [Migrating from Jenkins to GitLab](https://www.youtube.com/watch?v=RlEVGOpYF5Y) video for examples of how to: + +- Convert a Jenkins pipeline into a GitLab CI/CD pipeline. +- Use Auto DevOps to test your code automatically. Otherwise, read on for important information that helps you get the ball rolling. Welcome to GitLab! @@ -42,8 +43,8 @@ can be a great resource. ## Manage organizational transition An important part of transitioning from Jenkins to GitLab is the cultural and organizational -changes that come with the move, and successfully managing them. There are a few -things we have found that help this: +changes that come with the move, and successfully managing them. A few +things we have found that help this are: - Setting and communicating a clear vision of what your migration goals are helps your users understand why the effort is worth it. The value is clear when @@ -67,11 +68,11 @@ of transition, by letting you delay the migration of less urgent pipelines for a If you are interested in helping GitLab test the wrapper, join our [public testing issue](https://gitlab.com/gitlab-org/gitlab/-/issues/215675) for instructions and to provide your feedback. NOTE: -If you have a paid GitLab subscription, note that the JenkinsFile Wrapper is not packaged as part of GitLab, and falls outside of the scope of support. For more information, see the [Statement of Support](https://about.gitlab.com/support/statement-of-support/). +If you have a paid GitLab subscription, the JenkinsFile Wrapper is not packaged with GitLab and falls outside of the scope of support. For more information, see the [Statement of Support](https://about.gitlab.com/support/statement-of-support/). ## Important product differences -There are some high level differences between the products worth mentioning: +Some high level differences between the products worth mentioning are: - With GitLab you don't need a root `pipeline` keyword to wrap everything. - The way pipelines are triggered and [trigger other pipelines](../yaml/index.md#trigger) @@ -93,14 +94,13 @@ There are some high level differences between the products worth mentioning: contain scripts or other reusable code. - You can also use the [`extends` keyword](../yaml/index.md#extends) to reuse configuration in a single pipeline configuration. -- All jobs in a single stage always run in parallel, and all stages run in sequence. We are planning - to allow certain jobs to break this sequencing as needed with our [directed acyclic graph](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/47063) +- All jobs in a single stage always run in parallel, and all stages run in sequence. + Certain jobs might break this sequencing as needed with our [directed acyclic graph](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/47063) feature. - The [`parallel`](../yaml/index.md#parallel) keyword can automatically parallelize tasks, like tests that support parallelization. - Normally all jobs in a single stage run in parallel, and all stages run in sequence. - There are different [pipeline architectures](../pipelines/pipeline_architectures.md) - that allow you to change this behavior. + Different [pipeline architectures](../pipelines/pipeline_architectures.md) allow you to change this behavior. - The new [`rules` syntax](../yaml/index.md#rules) is the recommended method of controlling when different jobs run. It is more powerful than the `only/except` syntax. - One important difference is that jobs run independently of each other and have a @@ -114,10 +114,10 @@ There are some high level differences between the products worth mentioning: - Manual approvals or gates can be set up as [`when:manual` jobs](../jobs/job_control.md#create-a-job-that-must-be-run-manually). These can also leverage [`protected environments`](../jobs/job_control.md#run-a-job-after-a-delay) to control who is able to approve them. -- GitLab comes with a [container registry](../../user/packages/container_registry/index.md), and we recommend using +- GitLab comes with a [container registry](../../user/packages/container_registry/index.md), so you can use container images to set up your build environment. For example, set up one pipeline that builds your build environment itself and publish that to the container registry. Then, have your pipelines use this instead of each building their - own environment, which is slower and may be less consistent. We have extensive docs on [how to use the Container Registry](../../user/packages/container_registry/index.md). + own environment, which is slower and may be less consistent. We have extensive documentation on [how to use the Container Registry](../../user/packages/container_registry/index.md). - A central utilities repository can be a great place to put assorted scheduled jobs or other manual jobs that function like utilities. Jenkins installations tend to have a few of these. @@ -129,7 +129,7 @@ Jenkins agent, uninstall it and then [install and register the runner](../runner Runners do not require much overhead, so you can size them similarly to the Jenkins agents you were using. -There are some important differences in the way runners work in comparison to agents: +Some important differences in the way runners work in comparison to agents are: - Runners can be set up as [shared across an instance, be added at the group level, or set up at the project level](../runners/runners_scope.md). They self-select jobs from the scopes you've defined automatically. @@ -137,8 +137,8 @@ There are some important differences in the way runners work in comparison to ag associate runners with specific jobs. For example, you can use a tag for jobs that require dedicated, more powerful, or specific hardware. - GitLab has [autoscaling for runners](https://docs.gitlab.com/runner/configuration/autoscale.html). - Use autoscaling to provision runners only when needed, and scale down when not needed. - This is similar to ephemeral agents in Jenkins. + Use autoscaling to provision runners only when needed and scale down when not needed, + similar to ephemeral agents in Jenkins. If you are using `gitlab.com`, you can take advantage of our [shared runner fleet](../runners/index.md) to run jobs without provisioning your own runners. We are investigating making them @@ -148,14 +148,14 @@ as well. ## Groovy vs. YAML Jenkins Pipelines are based on [Groovy](https://groovy-lang.org/), so the pipeline specification is written as code. -GitLab works a bit differently, we use the more highly structured [YAML](https://yaml.org/) format, which -places scripting elements inside of `script` blocks separate from the pipeline specification itself. +GitLab works a bit differently, using the more highly structured [YAML](https://yaml.org/) format. +The scripting elements are in `script` blocks separate from the pipeline specification itself. -This is a strength of GitLab, in that it helps keep the learning curve much simpler to get up and running -and avoids some of the problem of unconstrained complexity which can make your Jenkinsfile hard to understand +Using YAML is a strength of GitLab, in that it helps keep the learning curve much simpler to get up and running. +It also avoids some of the problem of unconstrained complexity which can make your Jenkinsfile hard to understand and manage. -That said, we do of course still value DRY (don't repeat yourself) principles and want to ensure that +We do of course still value DRY (don't repeat yourself) principles. We want to ensure that behaviors of your jobs can be codified once and applied as needed. You can use the `extends` syntax to [reuse configuration in your jobs](../yaml/index.md#extends), and `include` can be used to [reuse pipeline configurations](../yaml/index.md#include) in pipelines @@ -197,13 +197,13 @@ can leverage. You can see the complete list of packaging features in the ## Integrated features -Where you may have used plugins to get things like code quality, unit tests, and security scanning working in Jenkins, +You may have used plugins to get things like code quality, unit tests, and security scanning working in Jenkins. GitLab takes advantage of our connected ecosystem to automatically pull these kinds of results into your merge requests, pipeline details pages, and other locations. You may find that you actually don't need to configure anything to have these appear. -If they aren't working as expected, or if you'd like to see what's available, our [CI/CD feature index](../index.md#features) has the full list -of bundled features and links to the documentation for each. +Our [CI/CD feature index](../index.md#features) has the full list of bundled features and links to the documentation for each. +Refer to this index if these features aren't working as expected, or if you'd like to see what's available. ### Templates @@ -213,7 +213,7 @@ as well as encourage inner sourcing. In self-managed GitLab instances, you can build an [Instance Template Repository](../../user/admin_area/settings/instance_template_repository.md). Development teams across the whole organization can select templates from a dropdown list. A group maintainer or a group owner is able to set a group to use as the source for the -[custom project templates](../../user/admin_area/custom_project_templates.md), which can +[custom project templates](../../user/admin_area/custom_project_templates.md). This can be used by all projects in the group. An instance administrator can set a group as the source for [instance project templates](../../user/group/custom_project_templates.md), which can be used by projects in that instance. @@ -221,7 +221,7 @@ which can be used by projects in that instance. ## Convert a declarative Jenkinsfile A declarative Jenkinsfile contains "Sections" and "Directives" which are used to control the behavior of your -pipelines. There are equivalents for all of these in GitLab, which we've documented below. +pipelines. Equivalents for all of these exist in GitLab, which we've documented below. This section is based on the [Jenkinsfile syntax documentation](https://www.jenkins.io/doc/book/pipeline/syntax/) and is meant to be a mapping of concepts there to concepts in GitLab. @@ -231,8 +231,8 @@ and is meant to be a mapping of concepts there to concepts in GitLab. #### `agent` The agent section is used to define how a pipeline executes. For GitLab, we use [runners](../runners/index.md) -to provide this capability. You can configure your own runners in Kubernetes or on any host, or take advantage -of our shared runner fleet (note that the shared runner fleet is only available for GitLab.com users). +to provide this capability. You can configure your own runners in Kubernetes or on any host. You can also take advantage +of our shared runner fleet (the shared runner fleet is only available for GitLab.com users). We also support using [tags](../runners/configure_runners.md#use-tags-to-control-which-jobs-a-runner-can-run) to direct different jobs to different runners (execution agents). @@ -272,11 +272,11 @@ default: #### `stages` GitLab CI/CD also lets you define stages, but is a little bit more free-form to configure. The GitLab [`stages` keyword](../yaml/index.md#stages) -is a top level setting that enumerates the list of stages, but you are not required to nest individual jobs underneath +is a top level setting that enumerates the list of stages. You are not required to nest individual jobs underneath the `stages` section. Any job defined in the `.gitlab-ci.yml` can be made a part of any stage through use of the [`stage` keyword](../yaml/index.md#stage). -Note that, unless otherwise specified, every pipeline is instantiated with a `build`, `test`, and `deploy` stage +Unless otherwise specified, every pipeline is instantiated with a `build`, `test`, and `deploy` stage which are run in that order. Jobs that have no `stage` defined are placed by default in the `test` stage. Of course, each job that refers to a stage must refer to a stage that exists in the pipeline configuration. @@ -292,8 +292,8 @@ my_job: #### `steps` -The `steps` section is equivalent to the [`script` section](../yaml/index.md#script) of an individual job. This is -a simple YAML array with each line representing an individual command to be run: +The `steps` section is equivalent to the [`script` section](../yaml/index.md#script) of an individual job. The `steps` section is a YAML array +with each line representing an individual command to be run: ```yaml my_job: @@ -308,7 +308,7 @@ my_job: In GitLab, we use the [`variables` keyword](../yaml/index.md#variables) to define different variables at runtime. These can also be set up through the GitLab UI, under CI/CD settings. See also our [general documentation on variables](../variables/index.md), -including the section on [protected variables](../variables/index.md#protected-cicd-variables) which can be used +including the section on [protected variables](../variables/index.md#protected-cicd-variables). This can be used to limit access to certain variables to certain environments or runners: ```yaml @@ -330,18 +330,18 @@ can provide any variables they like. #### `triggers` / `cron` -Because GitLab is integrated tightly with Git, SCM polling options for triggers are not needed. We support an easy to use +Because GitLab is integrated tightly with Git, SCM polling options for triggers are not needed. We support a [syntax for scheduling pipelines](../pipelines/schedules.md). #### `tools` GitLab does not support a separate `tools` directive. Our best-practice recommendation is to use pre-built -container images, which can be cached, and can be built to already contain the tools you need for your pipelines. Pipelines can +container images. These images can be cached and can be built to already contain the tools you need for your pipelines. Pipelines can be set up to automatically build these images as needed and deploy them to the [container registry](../../user/packages/container_registry/index.md). -If you're not using container images with Docker/Kubernetes, for example on Mac or FreeBSD, then the `shell` executor does require you to -set up your environment either in advance or as part of the jobs. You could create a `before_script` -action that handles this for you. +If you don't use container images with Docker or Kubernetes, but use the `shell` executor on your own system, +you must set up your environment. You can set up the environment in advance, or as part of the jobs +with a `before_script` action that handles this for you. #### `input` @@ -351,7 +351,7 @@ variable entry. #### `when` GitLab does support a [`when` keyword](../yaml/index.md#when) which is used to indicate when a job should be -run in case of (or despite) failure, but most of the logic for controlling pipelines can be found in +run in case of (or despite) failure. Most of the logic for controlling pipelines can be found in our very powerful [`rules` system](../yaml/index.md#rules): ```yaml diff --git a/doc/ci/pipelines/cicd_minutes.md b/doc/ci/pipelines/cicd_minutes.md index 772a06980af..d9ad224ab95 100644 --- a/doc/ci/pipelines/cicd_minutes.md +++ b/doc/ci/pipelines/cicd_minutes.md @@ -126,7 +126,7 @@ These additional CI/CD minutes: - Are used only after the monthly quota included in your subscription runs out. - Are carried over to the next month, if any remain at the end of the month. -- Are valid for 12 months from date of purchase or until all minutes are consumed, whichever comes first. Expiry of minutes is not currently enforced. +- Are valid for 12 months from date of purchase or until all minutes are consumed, whichever comes first. Expiry of minutes is not enforced. For example, with a GitLab SaaS Premium license: @@ -137,7 +137,7 @@ For example, with a GitLab SaaS Premium license: If you use `13,000` minutes during the month, the next month your additional minutes become `2,000`. If you use `9,000` minutes during the month, your additional minutes remain the same. -If you bought additional CI/CD minutes while on a trial subscription those minutes will be available after the trial ends or you upgrade to a paid plan. +If you bought additional CI/CD minutes while on a trial subscription, those minutes are available after the trial ends or you upgrade to a paid plan. You can find pricing for additional CI/CD minutes on the [GitLab Pricing page](https://about.gitlab.com/pricing/). diff --git a/doc/ci/pipelines/downstream_pipelines.md b/doc/ci/pipelines/downstream_pipelines.md index c8fcd06da07..6c17c23552d 100644 --- a/doc/ci/pipelines/downstream_pipelines.md +++ b/doc/ci/pipelines/downstream_pipelines.md @@ -183,7 +183,7 @@ You can trigger a child pipeline from a YAML file generated in a job, instead of static file saved in your project. This technique can be very powerful for generating pipelines targeting content that changed or to build a matrix of targets and architectures. -The artifact containing the generated YAML file must not be [larger than 5MB](https://gitlab.com/gitlab-org/gitlab/-/issues/249140). +The artifact containing the generated YAML file must not be [larger than 5 MB](https://gitlab.com/gitlab-org/gitlab/-/issues/249140). <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For an overview, see [Create child pipelines using dynamically generated configurations](https://youtu.be/nMdfus2JWHM). diff --git a/doc/ci/pipelines/index.md b/doc/ci/pipelines/index.md index f1ca8afa62c..ab98bab022e 100644 --- a/doc/ci/pipelines/index.md +++ b/doc/ci/pipelines/index.md @@ -188,6 +188,30 @@ In this example: - `DEPLOY_ENVIRONMENT` is listed in the **Run pipeline** page, but with no value set. The user is expected to define the value each time the pipeline is run manually. +##### Configure a list of selectable values for a prefilled variable + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/363660) in GitLab 15.5 [with a flag](../../administration/feature_flags.md) named `run_pipeline_graphql`. Disabled by default. +> - The `options` keyword was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/105502) in GitLab 15.7. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/106038) in GitLab 15.7. Feature flag `run_pipeline_graphql` removed. + +You can define an array of CI/CD variable values the user can select from when running a pipeline manually. +These values are in a dropdown list in the **Run pipeline** page. Add the list of +value options to `options` and set the default value with `value`. The string in `value` +must also be included in the `options` list. + +For example: + +```yaml +variables: + DEPLOY_ENVIRONMENT: + value: "staging" + options: + - "production" + - "staging" + - "canary" + description: "The deployment target. Set to 'staging' by default." +``` + ### Run a pipeline by using a URL query string > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24146) in GitLab 12.5. diff --git a/doc/ci/pipelines/multi_project_pipelines.md b/doc/ci/pipelines/multi_project_pipelines.md deleted file mode 100644 index 25ac9e13185..00000000000 --- a/doc/ci/pipelines/multi_project_pipelines.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'downstream_pipelines.md' -remove_date: '2022-11-30' ---- - -This document was moved to [another location](downstream_pipelines.md). - -<!-- This redirect file can be deleted after <2022-11-30>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/ci/pipelines/parent_child_pipelines.md b/doc/ci/pipelines/parent_child_pipelines.md deleted file mode 100644 index be8ed8ba6d7..00000000000 --- a/doc/ci/pipelines/parent_child_pipelines.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'downstream_pipelines.md' -remove_date: '2022-12-05' ---- - -This document was moved to [another location](downstream_pipelines.md). - -<!-- This redirect file can be deleted after <2022-12-05>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/ci/pipelines/pipeline_efficiency.md b/doc/ci/pipelines/pipeline_efficiency.md index 276a03cb480..0952fdf1f8f 100644 --- a/doc/ci/pipelines/pipeline_efficiency.md +++ b/doc/ci/pipelines/pipeline_efficiency.md @@ -219,7 +219,7 @@ that download and run faster. Try to use custom Docker images with the software pre-installed. It's usually much faster to download a larger pre-configured image than to use a common image and install -software on it each time. Docker's [Best practices for writing Dockerfiles](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) +software on it each time. The Docker [Best practices for writing Dockerfiles article](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) has more information about building efficient Docker images. Methods to reduce Docker image size: diff --git a/doc/ci/pipelines/settings.md b/doc/ci/pipelines/settings.md index a7f3cfb59d3..423ee31dec4 100644 --- a/doc/ci/pipelines/settings.md +++ b/doc/ci/pipelines/settings.md @@ -32,7 +32,7 @@ To change the visibility of your pipelines and related features: When it is selected, pipelines and related features are visible: - For [**Public**](../../user/public_access.md) projects, to everyone. - - For **Internal** projects, to all logged-in users except [external users](../../user/permissions.md#external-users). + - For **Internal** projects, to all logged-in users except [external users](../../user/admin_area/external_users.md). - For **Private** projects, to all project members (Guest or higher). When it is cleared: @@ -41,7 +41,7 @@ To change the visibility of your pipelines and related features: and the **CI/CD** menu items are visible only to project members (Reporter or higher). Other users, including guest users, can only view the status of pipelines and jobs, and only when viewing merge requests or commits. - - For **Internal** projects, pipelines are visible to all logged in users except [external users](../../user/permissions.md#external-users). + - For **Internal** projects, pipelines are visible to all logged in users except [external users](../../user/admin_area/external_users.md). Related features are visible only to project members (Reporter or higher). - For **Private** projects, pipelines and related features are visible to project members (Reporter or higher) only. @@ -284,9 +284,8 @@ when merging a merge request would cause the project's test coverage to decline. Follow these steps to enable the `Coverage-Check` MR approval rule: 1. Set up a [`coverage`](../yaml/index.md#coverage) regular expression for all jobs you want to include in the overall coverage value. -1. Go to your project and select **Settings > General**. -1. Expand **Merge request approvals**. -1. Select **Enable** next to the `Coverage-Check` approval rule. +1. Go to your project and select **Settings > Merge requests**. +1. Under **Merge request approvals**, select **Enable** next to the `Coverage-Check` approval rule. 1. Select the **Target branch**. 1. Set the number of **Approvals required** to greater than zero. 1. Select the users or groups to provide approval. diff --git a/doc/ci/runners/configure_runners.md b/doc/ci/runners/configure_runners.md index c675f7204ec..b61e11dd8bc 100644 --- a/doc/ci/runners/configure_runners.md +++ b/doc/ci/runners/configure_runners.md @@ -62,7 +62,7 @@ How this feature works: With some [runner executors](https://docs.gitlab.com/runner/executors/), if you can run a job on the runner, you can get full access to the file system, and thus any code it runs as well as the token of the runner. With shared runners, this means that anyone -that runs jobs on the runner, can access anyone else's code that runs on the +that runs jobs on the runner, can access another user's code that runs on the runner. In addition, because you can get access to the runner token, it is possible @@ -292,12 +292,12 @@ variables: A runner can have one of the following statuses. -| Status | Description | -|--------|-------------| -| **online** | The runner has contacted GitLab within the last 2 hours and is available to run jobs. | -| **offline** | The runner has not contacted GitLab in more than 2 hours and is not available to run jobs. Check the runner to see if you can bring it online. | -| **stale** | The runner has not contacted GitLab in more than 3 months. If the runner was created more than 3 months ago, but it never contacted the instance, it is also considered **stale**. | -| **never_contacted** | The runner has never contacted GitLab. To make the runner contact GitLab, run `gitlab-runner run`. | +| Status | Description | +|---------|-------------| +| `online` | The runner has contacted GitLab within the last 2 hours and is available to run jobs. | +| `offline` | The runner has not contacted GitLab in more than 2 hours and is not available to run jobs. Check the runner to see if you can bring it online. | +| `stale` | The runner has not contacted GitLab in more than 3 months. If the runner was created more than 3 months ago, but it never contacted the instance, it is also considered **stale**. | +| `never_contacted` | The runner has never contacted GitLab. To make the runner contact GitLab, run `gitlab-runner run`. | ## Configure runner behavior with variables @@ -306,11 +306,9 @@ globally or for individual jobs: - [`GIT_STRATEGY`](#git-strategy) - [`GIT_SUBMODULE_STRATEGY`](#git-submodule-strategy) -- [`GIT_SUBMODULE_PATHS`](#sync-or-exclude-specific-submodules-from-ci-jobs) - [`GIT_CHECKOUT`](#git-checkout) - [`GIT_CLEAN_FLAGS`](#git-clean-flags) - [`GIT_FETCH_EXTRA_FLAGS`](#git-fetch-extra-flags) -- [`GIT_SUBMODULE_PATHS`](#git-submodule-paths) - [`GIT_SUBMODULE_UPDATE_FLAGS`](#git-submodule-update-flags) - [`GIT_DEPTH`](#shallow-cloning) (shallow cloning) - [`GIT_SUBMODULE_DEPTH`](#git-submodule-depth) @@ -498,16 +496,13 @@ git fetch origin $REFSPECS --depth 50 --prune Where `$REFSPECS` is a value provided to the runner internally by GitLab. -### Git submodule paths +### Sync or exclude specific submodules from CI jobs > [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/2249) in GitLab Runner 14.0. Use the `GIT_SUBMODULE_PATHS` variable to control which submodules have to be synced or updated. You can set it globally or per-job in the [`variables`](../yaml/index.md#variables) section. -This variable can be very useful for projects which have a large number of submodules which not all of them -need to be synced or updated in all CI jobs. - The path syntax is the same as [`git submodule`](https://git-scm.com/docs/git-submodule#Documentation/git-submodule.txt-ltpathgt82308203): - To sync and update specific paths: @@ -524,6 +519,12 @@ The path syntax is the same as [`git submodule`](https://git-scm.com/docs/git-su GIT_SUBMODULE_PATHS: :(exclude)submoduleA :(exclude)submoduleB ``` +WARNING: +Git ignores nested paths. To ignore a nested submodule, exclude +the parent submodule and then manually clone it in the job's scripts. For example, + `git clone <repo> --recurse-submodules=':(exclude)nested-submodule'`. Make sure +to wrap the string in single quotes so the YAML can be parsed successfully. + ### Git submodule update flags > [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/3192) in GitLab Runner 14.8. @@ -565,34 +566,6 @@ You should be aware of the implications for the security, stability, and reprodu your builds when using the `--remote` flag. In most cases, it is better to explicitly track submodule commits as designed, and update them using an auto-remediation/dependency bot. -### Sync or exclude specific submodules from CI jobs - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/26495) in GitLab Runner 14.0. - -Some projects have a large number of submodules, and not all of them need to be -synced or updated in all CI jobs. Use the `GIT_SUBMODULE_PATHS` variable to control this behavior. -The path syntax is the same as [`git submodule`](https://git-scm.com/docs/git-submodule#Documentation/git-submodule.txt-ltpathgt82308203): - -- To sync and update specific paths: - - ```yaml - variables: - GIT_SUBMODULE_PATHS: 'submoduleA' - ``` - -- To exclude specific paths: - - ```yaml - variables: - GIT_SUBMODULE_PATHS: ':(exclude)submoduleA' - ``` - -WARNING: -Git ignores nested and multiple submodule paths. To ignore a nested submodule, exclude -the parent submodule and then manually clone it in the job's scripts. For example, - `git clone <repo> --recurse-submodules=':(exclude)nested-submodule'`. Make sure -to wrap the string in single quotes so the YAML can be parsed successfully. - ### Shallow cloning > Introduced in GitLab 8.9 as an experimental feature. @@ -930,7 +903,7 @@ The default is the number of CPUs available, but given the memory ramifications, setting. `FASTZIP_EXTRACTOR_CONCURRENCY` controls how many files are decompressed at once. Files from a zip archive can natively -be read from concurrency, so no additional memory is allocated in additional to what the decompressor requires. This +be read from concurrency, so no additional memory is allocated in addition to what the decompressor requires. This defaults to the number of CPUs available. ## Clean up stale runners **(ULTIMATE)** diff --git a/doc/ci/runners/saas/linux_saas_runner.md b/doc/ci/runners/saas/linux_saas_runner.md index df7d5570953..61421f86ff5 100644 --- a/doc/ci/runners/saas/linux_saas_runner.md +++ b/doc/ci/runners/saas/linux_saas_runner.md @@ -17,7 +17,7 @@ For Free, Premium, and Ultimate plan customers, jobs on these instances consume | | Small | Medium | Large | |-------------------|---------------------------|---------------------------|--------------------------| -| Specs | 1 vCPU, 3.75GB RAM | 2 vCPUs, 8GB RAM | 4 vCPUs, 16GB RAM | +| Specs | 1 vCPU, 3.75 GB RAM | 2 vCPUs, 8 GB RAM | 4 vCPUs, 16 GB RAM | | GitLab CI/CD tags | `saas-linux-small-amd64` | `saas-linux-medium-amd64` | `saas-linux-large-amd64` | | Subscription | Free, Premium, Ultimate | Free, Premium, Ultimate | Premium, Ultimate | @@ -73,12 +73,12 @@ Below are the settings for SaaS runners on Linux. | Setting | GitLab.com | Default | |-------------------------------------------------------------------------|------------------|---------| | Executor | `docker+machine` | - | -| Default Docker image | `ruby:2.5` | - | +| Default Docker image | `ruby:3.1` | - | | `privileged` (run [Docker in Docker](https://hub.docker.com/_/docker/)) | `true` | `false` | - **Cache**: These runners share a [distributed cache](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) - that's stored in a Google Cloud Storage (GCS) bucket. Cache contents not updated within + that's stored in a Google Cloud Storage (GCS) bucket. Cache contents not updated in the last 14 days are automatically removed, based on the [object lifecycle management policy](https://cloud.google.com/storage/docs/lifecycle). @@ -88,8 +88,8 @@ Below are the settings for SaaS runners on Linux. and [#4070](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/4070). NOTE: -The final disk space your jobs can use will be less than 25GB. Some disk space -allocated to the instance will be occupied by the operating system, the Docker image, +The final disk space your jobs can use is less than 25 GB. Some disk space +allocated to the instance is occupied by the operating system, the Docker image, and a copy of your cloned repository. ## Pre-clone script @@ -198,7 +198,7 @@ sentry_dsn = "X" ] limit = X [runners.docker] - image = "ruby:2.5" + image = "ruby:3.1" privileged = true volumes = [ "/certs/client", diff --git a/doc/ci/runners/saas/macos_saas_runner.md b/doc/ci/runners/saas/macos_saas_runner.md index cd40aac25bc..270f85e65e2 100644 --- a/doc/ci/runners/saas/macos_saas_runner.md +++ b/doc/ci/runners/saas/macos_saas_runner.md @@ -18,9 +18,11 @@ Jobs handled by macOS shared runners on GitLab.com **time out after 3 hours**, r ## Access request process -While in beta, to run CI jobs on the macOS runners, GitLab SaaS customer namespaces must be explicitly added to the macOS `allow-list`. +While in beta, to run CI jobs on the macOS runners, you must specify the GitLab SaaS customer personal or group [namespaces](../../../user/namespace/index.md) in the macOS `allow-list`. These are the namespaces that use the macOS runners. -After you have been added, you can use the macOS runners for any projects in your namespace. +When you specify a personal or group namespace, the top level group is not added unless you specify it. + +After you add your namespace, you can use the macOS runners for any projects under the namespace you included. To request access, open an [access request](https://gitlab.com/gitlab-com/runner-saas-macos-limited-availability/-/issues/new). The expected turnaround for activation is two business days. diff --git a/doc/ci/secrets/index.md b/doc/ci/secrets/index.md index a5082af89bc..bfc53210348 100644 --- a/doc/ci/secrets/index.md +++ b/doc/ci/secrets/index.md @@ -143,7 +143,7 @@ different policies together. If authentication is successful, these policies are attached to the resulting Vault token. [Bound claims](https://developer.hashicorp.com/vault/docs/auth/jwt#bound-claims) are predefined -values that are matched to the JWT's claims. With bounded claims, you can restrict access +values that are matched to the JWT claims. With bounded claims, you can restrict access to specific GitLab users, specific projects, or even jobs running for specific Git references. You can have as many bounded claims you need, but they must *all* match for authentication to be successful. diff --git a/doc/ci/secure_files/index.md b/doc/ci/secure_files/index.md index bba8a3e4c27..e63b671ad2b 100644 --- a/doc/ci/secure_files/index.md +++ b/doc/ci/secure_files/index.md @@ -29,7 +29,7 @@ plain text and binary file types. You can manage secure files in the project settings, or with the [secure files API](../../api/secure_files.md). Secure files can be [downloaded and used by CI/CD jobs](#use-secure-files-in-cicd-jobs) -by using the [load-secure-files](https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/load-secure-files) +by using the [download-secure-files](https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/download-secure-files) tool. NOTE: @@ -49,11 +49,11 @@ To add a secure file to a project: ## Use secure files in CI/CD jobs -To use your secure files in a CI/CD job, you must use the [`load-secure-files`](https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/load-secure-files) +To use your secure files in a CI/CD job, you must use the [`download-secure-files`](https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/download-secure-files) tool to download the files in the job. After they are downloaded, you can use them with your other script commands. -Add a command in the `script` section of your job to download the `load-secure-files` tool +Add a command in the `script` section of your job to download the `download-secure-files` tool and execute it. The files download into a `.secure_files` directory in the root of the project. To change the download location for the secure files, set the path in the `SECURE_FILES_DOWNLOAD_PATH` [CI/CD variable](../variables/index.md). @@ -65,5 +65,5 @@ test: variables: SECURE_FILES_DOWNLOAD_PATH: './where/files/should/go/' script: - - curl --silent "https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/load-secure-files/-/raw/main/installer" | bash + - curl --silent "https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/download-secure-files/-/raw/main/installer" | bash ``` diff --git a/doc/ci/services/index.md b/doc/ci/services/index.md index 830b9406c6e..beebaa6d03f 100644 --- a/doc/ci/services/index.md +++ b/doc/ci/services/index.md @@ -260,7 +260,7 @@ test: | Setting | Required | GitLab version | Description | |------------|----------|----------------| ----------- | | `name` | yes, when used with any other option | 9.4 | Full name of the image to use. If the full image name includes a registry hostname, use the `alias` option to define a shorter service access name. For more information, see [Accessing the services](#accessing-the-services). | -| `entrypoint` | no | 9.4 |Command or script to execute as the container's entrypoint. It's translated to Docker's `--entrypoint` option while creating the container. The syntax is similar to [`Dockerfile`'s `ENTRYPOINT`](https://docs.docker.com/engine/reference/builder/#entrypoint) directive, where each shell token is a separate string in the array. | +| `entrypoint` | no | 9.4 |Command or script to execute as the container's entrypoint. It's translated to the Docker `--entrypoint` option while creating the container. The syntax is similar to [`Dockerfile`'s `ENTRYPOINT`](https://docs.docker.com/engine/reference/builder/#entrypoint) directive, where each shell token is a separate string in the array. | | `command` | no | 9.4 |Command or script that should be used as the container's command. It's translated to arguments passed to Docker after the image's name. The syntax is similar to [`Dockerfile`'s `CMD`](https://docs.docker.com/engine/reference/builder/#cmd) directive, where each shell token is a separate string in the array. | | `alias` (1) | no | 9.4 | Additional alias that can be used to access the service from the job's container. Read [Accessing the services](#accessing-the-services) for more information. | | `variables` (2) | no | 14.5 | Additional environment variables that are passed exclusively to the service. The syntax is the same as [Job Variables](../variables/index.md). | @@ -394,6 +394,8 @@ time. ## Capturing service container logs +> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/3680) in GitLab Runner 15.6. + Logs generated by applications running in service containers can be captured for subsequent examination and debugging. You might want to look at service container's logs when the service container has started successfully, but is not behaving es expected, leading to job failures. The logs can indicate missing or incorrect configuration of the service diff --git a/doc/ci/testing/code_quality.md b/doc/ci/testing/code_quality.md index d1ed28b79c0..8e1c3d72d3d 100644 --- a/doc/ci/testing/code_quality.md +++ b/doc/ci/testing/code_quality.md @@ -71,11 +71,12 @@ See also the Code Climate list of [Supported Languages for Maintainability](http > - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/284140) in GitLab 13.12. > - [Disabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/2526) in GitLab 14.0 due to [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/334116). > - [Inline annotation added](https://gitlab.com/gitlab-org/gitlab/-/issues/2526) and [feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/284140) in GitLab 14.1. +> - [Updated](https://gitlab.com/groups/gitlab-org/-/epics/8071) to handle multiple findings better in GitLab 15.7, disabled by default behind the `refactor_code_quality_inline_findings` [feature flag](../../administration/feature_flags.md). Changes to files in merge requests can cause Code Quality to fall if merged. In these cases, the merge request's diff view displays an indicator next to lines with new Code Quality violations. For example: - + ## Example configuration diff --git a/doc/ci/testing/fail_fast_testing.md b/doc/ci/testing/fail_fast_testing.md index 58471a626da..b1bc44a0a37 100644 --- a/doc/ci/testing/fail_fast_testing.md +++ b/doc/ci/testing/fail_fast_testing.md @@ -12,7 +12,7 @@ For applications that use RSpec for running tests, we've introduced the `Verify/ [template to run subsets of your test suite](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates/Verify/FailFast.gitlab-ci.yml), based on the changes in your merge request. -The template uses the [test_file_finder (`tff`) gem](https://gitlab.com/gitlab-org/ci-cd/test_file_finder/) +The template uses the [`test_file_finder` (`tff`) gem](https://gitlab.com/gitlab-org/ci-cd/test_file_finder/) that accepts a list of files as input, and returns a list of spec (test) files that it believes to be relevant to the input files. diff --git a/doc/ci/testing/img/code_quality_mr_diff_report_v14_2.png b/doc/ci/testing/img/code_quality_mr_diff_report_v14_2.png Binary files differdeleted file mode 100644 index c1e9aad24ac..00000000000 --- a/doc/ci/testing/img/code_quality_mr_diff_report_v14_2.png +++ /dev/null diff --git a/doc/ci/testing/img/code_quality_mr_diff_report_v15_7.png b/doc/ci/testing/img/code_quality_mr_diff_report_v15_7.png Binary files differnew file mode 100644 index 00000000000..b45124e0e5d --- /dev/null +++ b/doc/ci/testing/img/code_quality_mr_diff_report_v15_7.png diff --git a/doc/ci/testing/unit_test_report_examples.md b/doc/ci/testing/unit_test_report_examples.md index 5d4cfa88d88..87426fc8496 100644 --- a/doc/ci/testing/unit_test_report_examples.md +++ b/doc/ci/testing/unit_test_report_examples.md @@ -42,7 +42,7 @@ Use the following job in `.gitlab-ci.yml`: golang: stage: test script: - - go get gotest.tools/gotestsum + - go install gotest.tools/gotestsum@latest - gotestsum --junitfile report.xml --format testname artifacts: when: always diff --git a/doc/ci/troubleshooting.md b/doc/ci/troubleshooting.md index 87ebff74600..81e13192cef 100644 --- a/doc/ci/troubleshooting.md +++ b/doc/ci/troubleshooting.md @@ -40,29 +40,8 @@ is at: https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/editor/schema/ci.json. ``` -The schema rules for custom YAML tags like `!reference` are treated as invalid by your editor until the custom tags are -set in your editor settings. For example: - -- In VS Code, you can set `vscode-yaml` to parse `customTags` in your `settings.json` file: - - ```json - "yaml.customTags": [ - "!reference sequence" - ] - ``` - -- In Sublime Text, if you are using the `LSP-yaml` package, you can set `customTags` in your `LSP-yaml` user settings: - - ```json - { - "settings": { - "yaml.customTags": ["!reference sequence"] - } - } - ``` - To see the full list of custom tags covered by the CI/CD schema, check the -latest version of the schema, linked above. +latest version of the schema. ### Verify syntax with CI Lint tool @@ -116,7 +95,7 @@ if you are using that type: Troubleshooting guides are available for some CI/CD features and related topics: -- [Container Registry](../user/packages/container_registry/index.md#troubleshooting-the-gitlab-container-registry) +- [Container Registry](../user/packages/container_registry/troubleshoot_container_registry.md) - [GitLab Runner](https://docs.gitlab.com/runner/faq/) - [Merge Trains](pipelines/merge_trains.md#troubleshooting) - [Docker Build](docker/using_docker_build.md#troubleshooting) @@ -362,7 +341,7 @@ To [prevent duplicate pipelines](jobs/job_control.md#avoid-duplicate-pipelines), [`workflow: rules`](yaml/index.md#workflow) or rewrite your rules to control which pipelines can run. -### Console workaround if job using resource_group gets stuck **(FREE SELF)** +### Console workaround if job using `resource_group` gets stuck **(FREE SELF)** ```ruby # find resource group by name diff --git a/doc/ci/variables/index.md b/doc/ci/variables/index.md index 97707c603bd..10dfa0174a0 100644 --- a/doc/ci/variables/index.md +++ b/doc/ci/variables/index.md @@ -153,6 +153,8 @@ job: ### Add a CI/CD variable to a project +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/362227) in GitLab 15.7, projects can define a maximum of 200 CI/CD variables. + You can add CI/CD variables to a project's settings. Only project members with the Maintainer role can add or update project CI/CD variables. To keep a CI/CD variable secret, put it @@ -191,7 +193,8 @@ The output is: ### Add a CI/CD variable to a group -> Support for environment scopes [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2874) in GitLab Premium 13.11 +> - Support for environment scopes [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2874) in GitLab Premium 13.11 +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/362227) in GitLab 15.7, groups can define a maximum of 200 CI/CD variables. To make a CI/CD variable available to all projects in a group, define a group CI/CD variable. Only group owners can add or update group-level CI/CD variables. @@ -391,6 +394,21 @@ To mark a variable as protected: The variable is available for all subsequent pipelines. +### Expand CI/CD variables + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217309) in GitLab 13.7. + +Expanded variables treat values with the `$` character as a reference to another variable. CI/CD variables are expanded +by default. + +To treat variables with a `$` character as raw strings, disable variable expansion for the variable: + +1. In the project or group, go to **Settings > CI/CD**. +1. Expand the **Variables** section. +1. Next to the variable you want to do not want expanded, select **Edit**. +1. Clear the **Expand variable** checkbox. +1. Select **Update variable**. + ### CI/CD variable security Malicious code pushed to your `.gitlab-ci.yml` file could compromise your variables @@ -715,7 +733,7 @@ You can override the value of a CI/CD variable when you > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/295234) in GitLab 13.8. -You can grant permission to override variables to [maintainers](../../user/permissions.md#project-features) only. When other users try to run a pipeline +You can grant permission to override variables to users with the Maintainer role only. When other users try to run a pipeline with overridden variables, they receive the `Insufficient permissions to set pipeline variables` error message. @@ -757,7 +775,7 @@ You can configure [Auto DevOps](../../topics/autodevops/index.md) to pass CI/CD to a running application. To make a CI/CD variable available as an environment variable in the running application's container, -[prefix the variable key](../../topics/autodevops/customize.md#application-secret-variables) +[prefix the variable key](../../topics/autodevops/cicd_variables.md#configure-application-secret-variables) with `K8S_SECRET_`. CI/CD variables with multi-line values are not supported. @@ -776,7 +794,7 @@ or job scripts. Debug logging exposes job execution details that are usually hid by the runner and makes job logs more verbose. It also exposes all variables and secrets available to the job. -Before you enable debug logging, make sure only [team members](../../user/permissions.md#project-features) +Before you enable debug logging, make sure only team members can view job logs. You should also [delete job logs](../jobs/index.md#view-jobs-in-a-pipeline) with debug output before you make logs public again. diff --git a/doc/ci/variables/predefined_variables.md b/doc/ci/variables/predefined_variables.md index 852110a1415..d7f67b79416 100644 --- a/doc/ci/variables/predefined_variables.md +++ b/doc/ci/variables/predefined_variables.md @@ -40,7 +40,7 @@ as it can cause the pipeline to behave unexpectedly. | `CI_COMMIT_SHORT_SHA` | 11.7 | all | The first eight characters of `CI_COMMIT_SHA`. | | `CI_COMMIT_TAG` | 9.0 | 0.5 | The commit tag name. Available only in pipelines for tags. | | `CI_COMMIT_TAG_MESSAGE` | 15.5 | all | The commit tag message. Available only in pipelines for tags. | -| `CI_COMMIT_TIMESTAMP` | 13.4 | all | The timestamp of the commit in the ISO 8601 format. | +| `CI_COMMIT_TIMESTAMP` | 13.4 | all | The timestamp of the commit in the [ISO 8601](https://www.rfc-editor.org/rfc/rfc3339#appendix-A) format. | | `CI_COMMIT_TITLE` | 10.8 | all | The title of the commit. The full first line of the message. | | `CI_CONCURRENT_ID` | all | 11.10 | The unique ID of build execution in a single executor. | | `CI_CONCURRENT_PROJECT_ID` | all | 11.10 | The unique ID of build execution in a single executor and project. | @@ -75,6 +75,7 @@ as it can cause the pipeline to behave unexpectedly. | `CI_JOB_NAME_SLUG` | 15.4 | all | `CI_JOB_NAME_SLUG` in lowercase, shortened to 63 bytes, and with everything except `0-9` and `a-z` replaced with `-`. No leading / trailing `-`. Use in paths. | | `CI_JOB_STAGE` | 9.0 | 0.5 | The name of the job's stage. | | `CI_JOB_STATUS` | all | 13.5 | The status of the job as each runner stage is executed. Use with [`after_script`](../yaml/index.md#after_script). Can be `success`, `failed`, or `canceled`. | +| `CI_JOB_TIMEOUT` | 15.7 | 15.7 | The job timeout value. | | `CI_JOB_TOKEN` | 9.0 | 1.2 | A token to authenticate with [certain API endpoints](../jobs/ci_job_token.md). The token is valid as long as the job is running. | | `CI_JOB_URL` | 11.1 | 0.5 | The job details URL. | | `CI_JOB_STARTED_AT` | 13.10 | all | The UTC datetime when a job started, in [ISO 8601](https://www.rfc-editor.org/rfc/rfc3339#appendix-A) format. | @@ -94,6 +95,7 @@ as it can cause the pipeline to behave unexpectedly. | `CI_PROJECT_ID` | all | all | The ID of the current project. This ID is unique across all projects on the GitLab instance. | | `CI_PROJECT_NAME` | 8.10 | 0.5 | The name of the directory for the project. For example if the project URL is `gitlab.example.com/group-name/project-1`, `CI_PROJECT_NAME` is `project-1`. | | `CI_PROJECT_NAMESPACE` | 8.10 | 0.5 | The project namespace (username or group name) of the job. | +| `CI_PROJECT_NAMESPACE_ID` | 15.7 | 0.5 | The project namespace ID of the job. | | `CI_PROJECT_PATH_SLUG` | 9.3 | all | `$CI_PROJECT_PATH` in lowercase with characters that are not `a-z` or `0-9` replaced with `-` and shortened to 63 bytes. Use in URLs and domain names. | | `CI_PROJECT_PATH` | 8.10 | 0.5 | The project namespace with the project name included. | | `CI_PROJECT_REPOSITORY_LANGUAGES` | 12.3 | all | A comma-separated, lowercase list of the languages used in the repository. For example `ruby,javascript,html,css`. The maximum number of languages is limited to 5. An issue [proposes to increase the limit](https://gitlab.com/gitlab-org/gitlab/-/issues/368925). | diff --git a/doc/ci/variables/where_variables_can_be_used.md b/doc/ci/variables/where_variables_can_be_used.md index c8436fc044d..88d0c9a2454 100644 --- a/doc/ci/variables/where_variables_can_be_used.md +++ b/doc/ci/variables/where_variables_can_be_used.md @@ -28,16 +28,16 @@ There are two places defined variables can be used. On the: | [`artifacts:name`](../yaml/index.md#artifactsname) | yes | Runner | The variable expansion is made by GitLab Runner's shell environment. | | [`before_script`](../yaml/index.md#before_script) | yes | Script execution shell | The variable expansion is made by the [execution shell environment](#execution-shell-environment) | | [`cache:key`](../yaml/index.md#cachekey) | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism). | -| [`environment:name`](../yaml/index.md#environmentname) | yes | GitLab | Similar to `environment:url`, but the variables expansion doesn't support the following:<br/><br/>- Variables that are based on the environment's name (`CI_ENVIRONMENT_NAME`, `CI_ENVIRONMENT_SLUG`).<br/>- Any other variables related to environment (currently only `CI_ENVIRONMENT_URL`).<br/>- [Persisted variables](#persisted-variables). | +| [`environment:name`](../yaml/index.md#environmentname) | yes | GitLab | Similar to `environment:url`, but the variables expansion doesn't support the following:<br/><br/>- `CI_ENVIRONMENT_*` variables.<br/>- [Persisted variables](#persisted-variables). | | [`environment:url`](../yaml/index.md#environmenturl) | yes | GitLab | The variable expansion is made by the [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism) in GitLab.<br/><br/>Supported are all variables defined for a job (project/group variables, variables from `.gitlab-ci.yml`, variables from triggers, variables from pipeline schedules).<br/><br/>Not supported are variables defined in the GitLab Runner `config.toml` and variables created in the job's `script`. | | [`environment:auto_stop_in`](../yaml/index.md#environmentauto_stop_in)| yes | GitLab | The variable expansion is made by the [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism) in GitLab.<br/><br/> The value of the variable being substituted should be a period of time in a human readable natural language form. See [possible inputs](../yaml/index.md#environmentauto_stop_in) for more information.| -| [`except:variables`](../yaml/index.md#onlyvariables--exceptvariables) | no | Not applicable | The variable must be in the form of `$variable`. Not supported are the following:<br/><br/>- Variables that are based on the environment's name (`CI_ENVIRONMENT_NAME`, `CI_ENVIRONMENT_SLUG`).<br/>- Any other variables related to environment (currently only `CI_ENVIRONMENT_URL`).<br/>- [Persisted variables](#persisted-variables). | +| [`except:variables`](../yaml/index.md#onlyvariables--exceptvariables) | no | Not applicable | The variable must be in the form of `$variable`. Not supported are the following:<br/><br/>- `CI_ENVIRONMENT_*` variables, except `CI_ENVIRONMENT_NAME` which is supported.<br/>- [Persisted variables](#persisted-variables). | | [`image`](../yaml/index.md#image) | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism). | | [`include`](../yaml/index.md#include) | yes | GitLab | The variable expansion is made by the [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism) in GitLab. <br/><br/>See [Use variables with include](../yaml/includes.md#use-variables-with-include) for more information on supported variables. | -| [`only:variables`](../yaml/index.md#onlyvariables--exceptvariables) | no | Not applicable | The variable must be in the form of `$variable`. Not supported are the following:<br/><br/>- Variables that are based on the environment's name (`CI_ENVIRONMENT_NAME`, `CI_ENVIRONMENT_SLUG`).<br/>- Any other variables related to environment (currently only `CI_ENVIRONMENT_URL`).<br/>- [Persisted variables](#persisted-variables). | +| [`only:variables`](../yaml/index.md#onlyvariables--exceptvariables) | no | Not applicable | The variable must be in the form of `$variable`. Not supported are the following:<br/><br/>- `CI_ENVIRONMENT_*` variables, except `CI_ENVIRONMENT_NAME` which is supported.<br/>- [Persisted variables](#persisted-variables). | | [`resource_group`](../yaml/index.md#resource_group) | yes | GitLab | Similar to `environment:url`, but the variables expansion doesn't support the following:<br/>- `CI_ENVIRONMENT_URL`<br/>- [Persisted variables](#persisted-variables). | | [`rules:exists`](../yaml/index.md#rulesexists) | yes | GitLab | The variable expansion is made by the [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism) in GitLab. | -| [`rules:if`](../yaml/index.md#rulesif) | no | Not applicable | The variable must be in the form of `$variable`. Not supported are the following:<br/><br/>- Variables that are based on the environment's name (`CI_ENVIRONMENT_NAME`, `CI_ENVIRONMENT_SLUG`).<br/>- Any other variables related to environment (currently only `CI_ENVIRONMENT_URL`).<br/>- [Persisted variables](#persisted-variables). | +| [`rules:if`](../yaml/index.md#rulesif) | no | Not applicable | The variable must be in the form of `$variable`. Not supported are the following:<br/><br/>- `CI_ENVIRONMENT_*` variables, except `CI_ENVIRONMENT_NAME` which is supported.<br/>- [Persisted variables](#persisted-variables). | | [`script`](../yaml/index.md#script) | yes | Script execution shell | The variable expansion is made by the [execution shell environment](#execution-shell-environment). | | [`services:name`](../yaml/index.md#services) | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism). | | [`services`](../yaml/index.md#services) | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism). | diff --git a/doc/ci/yaml/artifacts_reports.md b/doc/ci/yaml/artifacts_reports.md index 9f5e4e919b0..e12786f06ce 100644 --- a/doc/ci/yaml/artifacts_reports.md +++ b/doc/ci/yaml/artifacts_reports.md @@ -107,19 +107,17 @@ GitLab can display the results of coverage report in the merge request ## `artifacts:reports:codequality` -> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212499) to GitLab Free in 13.2. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212499) to GitLab Free in 13.2. +> - [Added support for multiple reports in diff annotations and full pipeline report](https://gitlab.com/gitlab-org/gitlab/-/issues/9014) in 15.7. The `codequality` report collects [code quality issues](../testing/code_quality.md). The collected code quality report uploads to GitLab as an artifact. -GitLab can display the results of: +GitLab can display the results of one or more reports in: -- One or more reports in the merge request [code quality widget](../testing/code_quality.md#code-quality-widget). -- Only one report in: - - The merge request [diff annotations](../testing/code_quality.md#code-quality-in-diff-view). - Track progress on adding support for multiple reports in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/328257). - - The [full report](../testing/metrics_reports.md). Track progress on adding support for multiple reports in - [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/9014). +- The merge request [code quality widget](../testing/code_quality.md#code-quality-widget). +- The merge request [diff annotations](../testing/code_quality.md#code-quality-in-diff-view). +- The [full report](../testing/metrics_reports.md). ## `artifacts:reports:container_scanning` **(ULTIMATE)** @@ -152,16 +150,16 @@ GitLab can display the results of one or more reports in: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/360766) in GitLab 15.3 This report is a Software Bill of Materials describing the components of a project -following the [cyclonedx](https://cyclonedx.org/docs/1.4) protocol format. +following the [CycloneDX](https://cyclonedx.org/docs/1.4) protocol format. -You can specify multiple cyclonedx reports per job. These can be either supplied +You can specify multiple CycloneDX reports per job. These can be either supplied as a list of filenames, a filename pattern, or both: - List of filenames: `cyclonedx: [gl-sbom-npm-npm.cdx.json, gl-sbom-bundler-gem.cdx.json]`. - A filename pattern: `cyclonedx: gl-sbom-*.json`. - Combination of both of the above: `cyclonedx: [gl-sbom-*.json, my-cyclonedx.json]`. -Below is an example of a job exposing cyclonedx artifacts: +Below is an example of a job exposing CycloneDX artifacts: ```yaml artifacts: diff --git a/doc/ci/yaml/includes.md b/doc/ci/yaml/includes.md index 2b9f42e1c75..daf2e653250 100644 --- a/doc/ci/yaml/includes.md +++ b/doc/ci/yaml/includes.md @@ -13,8 +13,8 @@ You can use [`include`](index.md#include) to include external YAML files in your To include a single configuration file, use either of these syntax options: -- `include` by itself with a single file, which is the same as - [`include:local`](index.md#includelocal): +- `include` by itself with a single file. If this is a local file, it is the same as [`include:local`](index.md#includelocal). + If this is a remote file, it is the same as [`include:remote`](index.md#includeremote). ```yaml include: '/templates/.after-script-template.yml' @@ -31,7 +31,8 @@ To include a single configuration file, use either of these syntax options: You can include an array of configuration files: -- If you do not specify an `include` type, the type defaults to [`include:local`](index.md#includelocal): +- If you do not specify an `include` type, each array item defaults to [`include:local`](index.md#includelocal) + or [`include:remote`](index.md#includeremote), as needed: ```yaml include: diff --git a/doc/ci/yaml/index.md b/doc/ci/yaml/index.md index 3392304775a..dffe409b193 100644 --- a/doc/ci/yaml/index.md +++ b/doc/ci/yaml/index.md @@ -86,6 +86,7 @@ of the listed keywords use the value defined in the `default` section. - [`artifacts`](#artifacts) - [`before_script`](#before_script) - [`cache`](#cache) +- [`hooks`](#hooks) - [`image`](#image) - [`interruptible`](#interruptible) - [`retry`](#retry) @@ -394,12 +395,12 @@ Use [`workflow`](workflow.md) to control pipeline behavior. #### `workflow:name` -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/372538) in GitLab 15.5 [with a flag](../../administration/feature_flags.md) named `pipeline_name`. Disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/372538) in GitLab 15.5 [with a flag](../../administration/feature_flags.md) named `pipeline_name`. Disabled by default. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/376095) in GitLab 15.7. FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, -ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `pipeline_name`. -The feature is not ready for production use. +On self-managed GitLab, by default this feature is available. To hide the feature, +ask an administrator to [disable the feature flag](../../administration/feature_flags.md) named `pipeline_name`. You can use `name` in `workflow:` to define a name for pipelines. @@ -424,7 +425,7 @@ A configuration with different pipeline names depending on the pipeline conditio ```yaml variables: - PIPELINE_NAME: 'Default pipeline name' + PIPELINE_NAME: 'Default pipeline name' # A default is not required. workflow: name: '$PIPELINE_NAME' @@ -437,6 +438,11 @@ workflow: PIPELINE_NAME: 'Ruby 3 pipeline' ``` +**Additional details**: + +- If the name is an empty string, the pipeline is not assigned a name. A name consisting + of only CI/CD variables could evaluate to an empty string if all the variables are also empty. + #### `workflow:rules` The `rules` keyword in `workflow` is similar to [`rules` defined in jobs](#rules), @@ -1766,8 +1772,8 @@ deploy: **Additional details**: -- Enviroments created from this job definition are assigned a [tier](../environments/index.md#deployment-tier-of-environments) based on this value. -- Existing environments don't have their tier updated if this value is added later. Existing enviroments must have their tier updated via the [Environments API](../../api/environments.md#update-an-existing-environment). +- Environments created from this job definition are assigned a [tier](../environments/index.md#deployment-tier-of-environments) based on this value. +- Existing environments don't have their tier updated if this value is added later. Existing environments must have their tier updated via the [Environments API](../../api/environments.md#update-an-existing-environment). **Related topics**: @@ -1862,6 +1868,74 @@ rspec: - [Reuse configuration sections by using `extends`](yaml_optimization.md#use-extends-to-reuse-configuration-sections). - Use `extends` to reuse configuration from [included configuration files](yaml_optimization.md#use-extends-and-include-together). +### `hooks` + +> Introduced in GitLab 15.6 [with a flag](../../administration/feature_flags.md) named `ci_hooks_pre_get_sources_script`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, +ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `ci_hooks_pre_get_sources_script`. +The feature is not ready for production use. + +Use `hooks` to specify lists of commands to execute on the runner +at certain stages of job execution, like before retrieving the Git repository. + +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). + +**Possible inputs**: + +- A hash of hooks and their commands. Available hooks: `pre_get_sources_script`. + +#### `hooks:pre_get_sources_script` + +> Introduced in GitLab 15.6 [with a flag](../../administration/feature_flags.md) named `ci_hooks_pre_get_sources_script`. Disabled by default. + +Use `hooks:pre_get_sources_script` to specify a list of commands to execute on the runner +before retrieving the Git repository and any submodules. You can use it +to adjust the Git client configuration first, for example. + +**Related topics**: + +- [GitLab Runner configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) + +**Example of `hooks:pre_get_sources_script`**: + +```yaml +job1: + hooks: + pre_get_sources_script: + - echo 'hello job1 pre_get_sources_script' + script: echo 'hello job1 script' +``` + +### `id_tokens` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/356986) in GitLab 15.7. + +Use `id_tokens` to create [JSON web tokens (JWT)](https://www.rfc-editor.org/rfc/rfc7519) to authenticate with third party services. All +JWTs created this way support OIDC authentication. The required `aud` sub-keyword is used to configure the `aud` claim for the JWT. + +**Possible inputs**: + +- Token names with their `aud` claims. `aud` can be a single string or as an array of strings. + +**Example of `id_tokens`**: + +```yaml +job_with_id_tokens: + id_tokens: + ID_TOKEN_1: + aud: https://gitlab.com + ID_TOKEN_2: + aud: + - https://gcp.com + - https://aws.com + script: + - command_to_authenticate_with_gitlab $ID_TOKEN_1 + - command_to_authenticate_with_aws $ID_TOKEN_2 +``` + ### `image` Use `image` to specify a Docker image that the job runs in. @@ -2210,6 +2284,9 @@ This example creates four paths of execution: in a job's `needs` section. - In GitLab 13.9 and older, if `needs` refers to a job that might not be added to a pipeline because of `only`, `except`, or `rules`, the pipeline might fail to create. In GitLab 13.10 and later, use the [`needs:optional`](#needsoptional) keyword to resolve a failed pipeline creation. +- If a pipeline has jobs with `needs: []` and jobs in the [`.pre`](#stage-pre) stage, they will + all start as soon as the pipeline is created. Jobs with `needs: []` start immediately, + and jobs in the `.pre` stage also start immediately. #### `needs:artifacts` @@ -2268,12 +2345,12 @@ In this example: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14311) in GitLab 12.7. Use `needs:project` to download artifacts from up to five jobs in other pipelines. -The artifacts are downloaded from the latest successful pipeline for the specified ref. +The artifacts are downloaded from the latest successful specified job for the specified ref. To specify multiple jobs, add each as separate array items under the `needs` keyword. -If there is a pipeline running for the specified ref, a job with `needs:project` -does not wait for the pipeline to complete. Instead, the job downloads the artifact -from the latest pipeline that completed successfully. +If there is a pipeline running for the ref, a job with `needs:project` +does not wait for the pipeline to complete. Instead, the artifacts are downloaded +from the latest successful run of the specified job. `needs:project` must be used with `job`, `ref`, and `artifacts`. @@ -2963,7 +3040,7 @@ job: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/363024) in GitLab 15.3. Supported by `release-cli` v0.12.0 or later. -If the tag does not exist, the newly created tag is annotated with the message specifed by `tag_message`. +If the tag does not exist, the newly created tag is annotated with the message specified by `tag_message`. If omitted, a lightweight tag is created. **Keyword type**: Job keyword. You can use it only as part of a job. @@ -3670,7 +3747,7 @@ job: ### `services` -Use `services` to specify an additional Docker image to run scripts in. The [`services` image](../services/index.md) is linked +Use `services` to specify any additional Docker images that your scripts require to run successfully. The [`services` image](../services/index.md) is linked to the image specified in the [`image`](#image) keyword. **Keyword type**: Job keyword. You can use it only as part of a job or in the @@ -3706,9 +3783,11 @@ test: - bundle exec rake spec ``` -In this example, the job launches a Ruby container. Then, from that container, the job launches -another container that's running PostgreSQL. Then the job then runs scripts -in that container. +In this example, GitLab launches two containers for the job: + +- A Ruby container that runs the `script` commands. +- A PostgreSQL container. The `script` commands in the Ruby container can connect to + the PostgreSQL database at the `db-postgrest` hostname. **Related topics**: @@ -3882,6 +3961,12 @@ job2: - echo "This job runs in the test stage." ``` +**Additional details:** + +- If a pipeline has jobs with [`needs: []`](#needs) and jobs in the `.pre` stage, they will + all start as soon as the pipeline is created. Jobs with `needs: []` start immediately, + ignoring any stage configuration. + ### `tags` > - A limit of 50 tags per job [enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/338929) in GitLab 14.3. @@ -4228,7 +4313,8 @@ deploy_review_job: #### `variables:description` -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30101) in GitLab 13.7. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30101) in GitLab 13.7. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/363660) in GitLab 15.5, `variables:value` can contain an array of values. Use the `description` keyword to define a [pipeline-level (global) variable that is prefilled](../pipelines/index.md#prefill-variables-in-manual-pipelines) when [running a pipeline manually](../pipelines/index.md#run-a-pipeline-manually). @@ -4240,6 +4326,7 @@ If used with `value`, the variable value is also prefilled when running a pipeli **Possible inputs**: - A string. +- An array of strings. **Example of `variables:description`**: @@ -4254,10 +4341,17 @@ variables: - A global variable defined with `value` but no `description` behaves the same as [`variables`](#variables). +- `variables:value` can [contain an array of selectable values](../pipelines/index.md#configure-a-list-of-selectable-values-for-a-prefilled-variable). #### `variables:expand` -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353991) in GitLab 15.6 [with a flag](../../administration/feature_flags.md) named `ci_raw_variables_in_yaml_config`. Disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353991) in GitLab 15.6 [with a flag](../../administration/feature_flags.md) named `ci_raw_variables_in_yaml_config`. Disabled by default. +> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/375034) in GitLab 15.6. +> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/375034) in GitLab 15.7. + +FLAG: +On self-managed GitLab, by default this feature is available. To hide the feature per project, +ask an administrator to [disable the feature flag](../../administration/feature_flags.md) named `ci_raw_variables_in_yaml_config`. Use the `expand` keyword to configure a variable to be expandable or not. diff --git a/doc/ci/yaml/yaml_optimization.md b/doc/ci/yaml/yaml_optimization.md index f4774619713..5344a999b95 100644 --- a/doc/ci/yaml/yaml_optimization.md +++ b/doc/ci/yaml/yaml_optimization.md @@ -473,3 +473,27 @@ nested-references: ``` In this example, the `nested-references` job runs all three `echo` commands. + +### Configure your IDE to support `!reference` tags + +The [pipeline editor](../pipeline_editor/index.md) supports `!reference` tags. However, the schema rules for custom YAML +tags like `!reference` might be treated as invalid by your editor by default. +You can configure some editors to accept `!reference` tags. For example: + +- In VS Code, you can set `vscode-yaml` to parse `customTags` in your `settings.json` file: + + ```json + "yaml.customTags": [ + "!reference sequence" + ] + ``` + +- In Sublime Text, if you are using the `LSP-yaml` package, you can set `customTags` in your `LSP-yaml` user settings: + + ```json + { + "settings": { + "yaml.customTags": ["!reference sequence"] + } + } + ``` |
