diff options
Diffstat (limited to 'doc/ci')
83 files changed, 2098 insertions, 1526 deletions
diff --git a/doc/ci/caching/index.md b/doc/ci/caching/index.md index be5112251a4..7aeafce9352 100644 --- a/doc/ci/caching/index.md +++ b/doc/ci/caching/index.md @@ -535,7 +535,7 @@ and should only be disabled in an environment where all users with Developer rol To use the same cache for all branches: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **General pipelines**. 1. Clear the **Use separate caches for protected branches** checkbox. @@ -630,7 +630,7 @@ The next time the pipeline runs, the cache is stored in a different location. You can clear the cache in the GitLab UI: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Build > Pipelines**. 1. In the upper-right corner, select **Clear runner caches**. diff --git a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md index 7164fae10a1..1820cf77841 100644 --- a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md +++ b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md @@ -15,8 +15,8 @@ GitLab CI/CD can be used with Bitbucket Cloud by: To use GitLab CI/CD with a Bitbucket Cloud repository: 1. In GitLab, create a project: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). - 1. Select **View all your projects**. + 1. On the left sidebar, select **Search or go to**. + 1. Select **View all my projects**. 1. On the right of the page, select **New project**. 1. Select **Run CI/CD for external repository**. 1. Select **Repository by URL**. @@ -40,7 +40,7 @@ To use GitLab CI/CD with a Bitbucket Cloud repository: using the Personal Access Token we just generated for authentication. ```plaintext - https://gitlab.com/api/v4/projects/<PROJECT_ID>/mirror/pull?private_token=<PERSONAL_ACCESS_TOKEN> + https://gitlab.example.com/api/v4/projects/:project_id/mirror/pull?private_token=<your_personal_access_token> ``` The web hook Trigger should be set to 'Repository Push'. diff --git a/doc/ci/ci_cd_for_external_repos/github_integration.md b/doc/ci/ci_cd_for_external_repos/github_integration.md index 1fad7ad5a53..bc61990fcd8 100644 --- a/doc/ci/ci_cd_for_external_repos/github_integration.md +++ b/doc/ci/ci_cd_for_external_repos/github_integration.md @@ -34,8 +34,8 @@ repositories: `repo` and `admin:repo_hook` so that GitLab can access your project, update commit statuses, and create a web hook to notify GitLab of new commits. 1. In GitLab, create a project: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). - 1. Select **View all your projects**. + 1. On the left sidebar, select **Search or go to**. + 1. Select **View all my projects**. 1. On the right of the page, select **New project**. 1. Select **Run CI/CD for external repository**. 1. Select **GitHub**. @@ -63,8 +63,8 @@ To manually enable GitLab CI/CD for your repository: 1. Enter a **Token description** and update the scope to allow `repo` so that GitLab can access your project and update commit statuses. 1. In GitLab, create a project: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). - 1. Select **View all your projects**. + 1. On the left sidebar, select **Search or go to**. + 1. Select **View all my projects**. 1. Select **New project**. 1. Select **Run CI/CD for external repository** and **Repository by URL**. 1. In the **Git repository URL** field, enter the HTTPS URL for your GitHub repository. diff --git a/doc/ci/ci_cd_for_external_repos/index.md b/doc/ci/ci_cd_for_external_repos/index.md index a9093632a8c..76996294a96 100644 --- a/doc/ci/ci_cd_for_external_repos/index.md +++ b/doc/ci/ci_cd_for_external_repos/index.md @@ -18,14 +18,14 @@ external repository to get the benefits of GitLab CI/CD. Connecting an external repository sets up [repository mirroring](../../user/project/repository/mirror/index.md) and creates a lightweight project with issues, merge requests, wiki, and snippets disabled. These features -[can be re-enabled later](../../user/project/settings/index.md#configure-project-visibility-features-and-permissions). +[can be re-enabled later](../../user/project/settings/index.md#configure-project-features-and-permissions). ## Connect to an external repository To connect to an external repository: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). -1. Select **View all your projects**. +1. On the left sidebar, select **Search or go to**. +1. Select **View all my projects**. 1. Select **New project**. 1. Select **Run CI/CD for external repository**. 1. Select **GitHub** or **Repository by URL**. @@ -33,7 +33,7 @@ To connect to an external repository: If the **Run CI/CD for external repository** option is not available, the GitLab instance might not have any import sources configured. Ask an administrator for your instance to check -the [import sources configuration](../../administration/settings/visibility_and_access_controls.md#configure-allowed-import-sources). +the [import sources configuration](../../administration/settings/import_and_export_settings.md#configure-allowed-import-sources). ## Pipelines for external pull requests diff --git a/doc/ci/cloud_deployment/ecs/deploy_to_aws_ecs.md b/doc/ci/cloud_deployment/ecs/deploy_to_aws_ecs.md index f50b7be855a..2a73184eb7f 100644 --- a/doc/ci/cloud_deployment/ecs/deploy_to_aws_ecs.md +++ b/doc/ci/cloud_deployment/ecs/deploy_to_aws_ecs.md @@ -17,7 +17,7 @@ Ensure your own [runners are configured](../../runners/index.md). ## Prerequisites -- An [AWS account](https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/). +- An [AWS account](https://repost.aws/knowledge-center/create-and-activate-aws-account). Sign in with an existing AWS account or create a new one. - In this guide, you create an infrastructure in [`us-east-2` region](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html). You can use any region, but do not change it after you begin. diff --git a/doc/ci/cloud_services/aws/index.md b/doc/ci/cloud_services/aws/index.md index f0183fc7ba2..dd36da86cca 100644 --- a/doc/ci/cloud_services/aws/index.md +++ b/doc/ci/cloud_services/aws/index.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: `CI_JOB_JWT_V2` was [deprecated in GitLab 15.9](../../../update/deprecations.md#old-versions-of-json-web-tokens-are-deprecated) -and is scheduled to be removed in GitLab 16.5. Use [ID tokens](../../yaml/index.md#id_tokens) instead. +and is scheduled to be removed in GitLab 17.0. Use [ID tokens](../../yaml/index.md#id_tokens) instead. In this tutorial, we'll show you how to use a GitLab CI/CD job with a JSON web token (JWT) to retrieve temporary credentials from AWS without needing to store secrets. To do this, you must configure OpenID Connect (OIDC) for ID federation between GitLab and AWS. For background and requirements for integrating GitLab using OIDC, see [Connect to cloud services](../index.md). @@ -85,9 +85,12 @@ assume role: - `ROLE_ARN`: The role ARN defined in this [step](#configure-a-role-and-trust). - `GITLAB_OIDC_TOKEN`: An OIDC [ID token](../../yaml/index.md#id_tokens). -## Working example +## Working examples -See this [reference project](https://gitlab.com/guided-explorations/aws/configure-openid-connect-in-aws) for provisioning OIDC in AWS using Terraform and a sample script to retrieve temporary credentials. +- See this [reference project](https://gitlab.com/guided-explorations/aws/configure-openid-connect-in-aws) for provisioning OIDC in AWS using Terraform and a sample script to retrieve temporary credentials. +- [OIDC and Multi-Account Deployment with GitLab and ECS](https://gitlab.com/guided-explorations/aws/oidc-and-multi-account-deployment-with-ecs). +- AWS Partner (APN) Blog: [Setting up OpenID Connect with GitLab CI/CD](https://aws.amazon.com/blogs/apn/setting-up-openid-connect-with-gitlab-ci-cd-to-provide-secure-access-to-environments-in-aws-accounts/). +- [GitLab at AWS re:Inforce 2023: Secure GitLab CD pipelines to AWS w/ OpenID and JWT](https://www.youtube.com/watch?v=xWQGADDVn8g). ## Troubleshooting diff --git a/doc/ci/cloud_services/azure/index.md b/doc/ci/cloud_services/azure/index.md index fa8ff506dd0..3a882cf6820 100644 --- a/doc/ci/cloud_services/azure/index.md +++ b/doc/ci/cloud_services/azure/index.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: `CI_JOB_JWT_V2` was [deprecated in GitLab 15.9](../../../update/deprecations.md#old-versions-of-json-web-tokens-are-deprecated) -and is scheduled to be removed in GitLab 16.5. Use [ID tokens](../../yaml/index.md#id_tokens) instead. +and is scheduled to be removed in GitLab 17.0. Use [ID tokens](../../yaml/index.md#id_tokens) instead. This tutorial demonstrates how to use a JSON web token (JWT) in a GitLab CI/CD job to retrieve temporary credentials from Azure without needing to store secrets. @@ -135,7 +135,7 @@ The CI/CD variables are: - `AZURE_CLIENT_ID`: The [application client ID you saved earlier](#create-azure-ad-application-and-service-principal). - `AZURE_TENANT_ID`: Your Azure Active Directory. You can - [find it by using the Azure CLI or Azure Portal](https://learn.microsoft.com/en-us/azure/active-directory/fundamentals/active-directory-how-to-find-tenant). + [find it by using the Azure CLI or Azure Portal](https://learn.microsoft.com/en-us/azure/active-directory/fundamentals/how-to-find-tenant). - `GITLAB_OIDC_TOKEN`: An OIDC [ID token](../../yaml/index.md#id_tokens). ## Troubleshooting diff --git a/doc/ci/cloud_services/google_cloud/index.md b/doc/ci/cloud_services/google_cloud/index.md index f2318b818d8..a733f3d59cb 100644 --- a/doc/ci/cloud_services/google_cloud/index.md +++ b/doc/ci/cloud_services/google_cloud/index.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: `CI_JOB_JWT_V2` was [deprecated in GitLab 15.9](../../../update/deprecations.md#old-versions-of-json-web-tokens-are-deprecated) -and is scheduled to be removed in GitLab 16.5. Use [ID tokens](../../yaml/index.md#id_tokens) instead. +and is scheduled to be removed in GitLab 17.0. Use [ID tokens](../../yaml/index.md#id_tokens) instead. This tutorial demonstrates authenticating to Google Cloud from a GitLab CI/CD job using a JSON Web Token (JWT) token and Workload Identity Federation. This configuration diff --git a/doc/ci/cloud_services/index.md b/doc/ci/cloud_services/index.md index 6896fffcce7..87984c424c4 100644 --- a/doc/ci/cloud_services/index.md +++ b/doc/ci/cloud_services/index.md @@ -12,9 +12,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: `CI_JOB_JWT` and `CI_JOB_JWT_V2` were [deprecated in GitLab 15.9](../../update/deprecations.md#old-versions-of-json-web-tokens-are-deprecated) -and are scheduled to be removed in GitLab 16.5. Use [ID tokens](../yaml/index.md#id_tokens) instead. +and are scheduled to be removed in GitLab 17.0. Use [ID tokens](../yaml/index.md#id_tokens) instead. -GitLab CI/CD supports [OpenID Connect (OIDC)](https://openid.net/connect/faq/) to +GitLab CI/CD supports [OpenID Connect (OIDC)](https://openid.net/developers/how-connect-works/) to give your build and deployment jobs access to cloud credentials and services. Historically, teams stored secrets in projects or applied permissions on the GitLab Runner instance to build and deploy. OIDC capable [ID tokens](../yaml/index.md#id_tokens) are configurable diff --git a/doc/ci/components/catalog.md b/doc/ci/components/catalog.md new file mode 100644 index 00000000000..2194e72d56c --- /dev/null +++ b/doc/ci/components/catalog.md @@ -0,0 +1,33 @@ +--- +stage: Verify +group: Pipeline Authoring +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# CI/CD catalog **(PREMIUM ALL EXPERIMENT)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/407249) in GitLab 16.1. + +The CI/CD catalog is a list of [components repositories](index.md#components-repository), +each containing resources that you can add to your CI/CD pipelines. + +## Mark a components repository as a catalog resource + +After components are added to a components repository, they can immediately be [used](index.md#use-a-component-in-a-cicd-configuration) +to build pipelines in other projects. + +However, this repository is not discoverable. You must mark this project as a catalog resource +to allow it to be visible in the CI/CD Catalog so other users can discover it. + +To mark a project as a catalog resource: + +1. On the left sidebar, select **Search or go to** and find your project. +1. On the left sidebar, select **Settings > General**. +1. Expand **Visibility, project features, permissions**. +1. Scroll down to **CI/CD Catalog resource** and select the toggle to mark the project as a catalog resource. + +Ensure the project has a clear [description](../../user/project/settings/index.md#edit-project-name-and-description), +as the project description is displayed in the component list in the catalog. + +NOTE: +This action is not reversible. diff --git a/doc/ci/components/index.md b/doc/ci/components/index.md index 4a739bdfcf6..e73436522dc 100644 --- a/doc/ci/components/index.md +++ b/doc/ci/components/index.md @@ -2,29 +2,32 @@ stage: Verify group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -type: reference --- -# CI/CD Components (Experimental) +# CI/CD components **(FREE ALL EXPERIMENT)** > - Introduced as an [experimental feature](../../policy/experiment-beta-support.md) in GitLab 16.0, [with a flag](../../administration/feature_flags.md) named `ci_namespace_catalog_experimental`. Disabled by default. > - [Enabled on GitLab.com and self-managed](https://gitlab.com/groups/gitlab-org/-/epics/9897) in GitLab 16.2. -> - [Feature flag `ci_namespace_catalog_experimental` removed.](https://gitlab.com/gitlab-org/gitlab/-/issues/394772) in GitLab 16.3. +> - [Feature flag `ci_namespace_catalog_experimental` removed](https://gitlab.com/gitlab-org/gitlab/-/issues/394772) in GitLab 16.3. This feature is an experimental feature and [an epic exists](https://gitlab.com/groups/gitlab-org/-/epics/9897) to track future work. Tell us about your use case by leaving comments in the epic. -## Components Repository +## Components repository -A components repository is a GitLab project with a repository that hosts one or more pipeline components. A pipeline component is a reusable single pipeline configuration unit. You can use them to compose an entire pipeline configuration or a small part of a larger pipeline. It can optionally take [input parameters](../yaml/includes.md#define-input-parameters-with-specinputs). +A components repository is a GitLab project with a repository that hosts one or more pipeline components. +A pipeline component is a reusable single pipeline configuration unit. Use them to compose +an entire pipeline configuration or a small part of a larger pipeline. -### Create a components repository +A component can optionally take [input parameters](../yaml/inputs.md). + +## Create a components repository To create a components repository, you must: 1. [Create a new project](../../user/project/index.md#create-a-blank-project) with a `README.md` file. - -1. Create a `template.yml` file inside the project's root directory that contains the configuration you want to provide as a component. For example: +1. Create a `template.yml` file inside the project's root directory that contains the configuration you want to provide as a component. + For example: ```yaml spec: @@ -39,14 +42,71 @@ To create a components repository, you must: ### Directory structure -A components repository can host one or more components. +A components repository can host one or more components, and must follow a mandatory file structure. + +Component configurations can be saved through the following directory structure, containing: + +- A `templates` directory at the top level of your components repository. All component configuration files + should be saved under this directory. +- Files ending in `.yml` containing the component configurations, one file per component. +- A Markdown `README.md` file explaining the details of all the components in the repository. + +For example, if the project contains a single component and a pipeline to test the component, +the file structure should be similar to: + +```plaintext +├── templates/ +│ └── only_template.yml +├── README.md +└── .gitlab-ci.yml +``` + +This example component could be referenced with a path similar to `gitlab.com/my-username/my-component/only_template@<version>`, +if the project is: + +- On GitLab.com +- Named `my-component` +- In a personal namespace named `my-username` + +The templates directory and the suffix of the configuration file should be excluded from the referenced path. + +If the project contains multiple components, then the file structure should be similar to: + +```plaintext +├── README.md +├── .gitlab-ci.yml +└── templates/ + └── all-scans.yml + └── secret-detection.yml +``` + +These components would be referenced with these paths: -Components repositories must follow a mandatory file structure, containing: +- `gitlab.com/my-username/my-component/all-scans` +- `gitlab.com/my-username/my-component/secret-detection` + +You can omit the filename in the path if the configuration file is named `template.yml`. +For example, the following component could be referenced with `gitlab.com/my-username/my-component/dast`: + +```plaintext +├── README.md +├── .gitlab-ci.yml +├── templates/ +│ └── dast +│ └── template.yml +``` + +#### Component configurations saved in any directory (deprecated) + +NOTE: +Saving component configurations through this directory structure is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/415855). + +Components configurations can be saved through the following directory structure, containing: - `template.yml`: The component configuration, one file per component. If there is only one component, this file can be in the root of the project. If there are multiple components, each file must be in a separate subdirectory. -- `README.md`: A documentation file explaining the details of the all the components in the repository. +- `README.md`: A documentation file explaining the details of all the components in the repository. For example, if the project is on GitLab.com, named `my-component`, and in a personal namespace named `my-username`: @@ -60,6 +120,9 @@ namespace named `my-username`: └── .gitlab-ci.yml ``` + The `.gitlab-ci.yml` file is not required for a CI/CD component to work, but + [testing the component](#test-the-component) in a pipeline in the project is recommended. + This component is referenced with the path `gitlab.com/my-username/my-component@<version>`. - Containing one default component and multiple sub-components, then the file structure @@ -95,279 +158,281 @@ Nesting of components is not possible. For example: │ └── nested_template.yml ``` -### Test a component +## Release a component + +To create a release for a CI/CD component, use either: + +- The [`release`](../yaml/index.md#release) keyword in a CI/CD pipeline. Like in the + [component testing example](#test-the-component), you can set a component to automatically + be released after all tests pass in pipelines for new tags. +- The [UI for creating a release](../../user/project/releases/index.md#create-a-release). + +All released versions of the components are displayed in the CI/CD Catalog +page for the given resource, providing users with information about official releases. -Testing components as part of the development workflow to ensure that quality maintains high standards is strongly recommended. +Components [can be used](#use-a-component-in-a-cicd-configuration) without being released, +but only with a commit SHA or a branch name. To enable the use of tags or the `~latest` version keyword, +you must create a release. -Testing changes in a CI/CD pipeline can be done, like any other project, by creating a `.gitlab-ci.yml` in the root directory. +## Use a component in a CI/CD configuration +You can add a component to a CI/CD configuration with the `include: component` keyword. For example: ```yaml include: - # include the component located in the current project from the current SHA - - component: gitlab.com/$CI_PROJECT_PATH@$CI_COMMIT_SHA + - component: gitlab.example.com/my-namespace/my-component@1.0 inputs: stage: build - -stages: [build, test, release] - -# Expect `component-job` is added. -# This is an example of testing that the included component works as expected. -# You can leverage GitLab API endpoints or 3rd party tools to inspect data generated by the component. -ensure-job-added: - stage: test - image: badouralix/curl-jq - script: - - | - route="https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/pipelines/$CI_PIPELINE_ID/jobs" - count=`curl --silent --header "PRIVATE-TOKEN: $API_TOKEN" $route | jq 'map(select(.name | contains("component-job"))) | length'` - if [ "$count" != "1" ]; then - exit 1 - fi - -# If we are tagging a release with a specific convention ("v" + number) and all -# previous checks succeeded, we proceed with creating a release automatically. -create-release: - stage: release - image: registry.gitlab.com/gitlab-org/release-cli:latest - rules: - - if: $CI_COMMIT_TAG =~ /^v\d+/ - script: echo "Creating release $CI_COMMIT_TAG" - release: - tag_name: $CI_COMMIT_TAG - description: "Release $CI_COMMIT_TAG of components repository $CI_PROJECT_PATH" ``` -After committing and pushing changes, the pipeline tests the component then releases it if the test passes. +The component is identified by a unique address in the form `<fully-qualified-domain-name>/<component-path>@<specific-version>`, +where: -### Release a component +- `<fully-qualified-domain-name>` matches the GitLab host. You can only reference components + in the same GitLab instance as your project. +- `<component-path>` is the component project's full path and directory where the + component YAML file is located. +- `<specific-version>` is the version of the component. In order of highest priority first, + the version can be: + - A branch name, for example `main`. + - A commit SHA, for example `e3262fdd0914fa823210cdb79a8c421e2cef79d8`. + - A tag, for example: `1.0`. If a tag and branch exist with the same name, the tag + takes precedence over the branch. If a tag and commit SHA exist with the same name, + the commit SHA takes precedence over the tag. + - `~latest`, which is a special version that always points to the most recent released tag. + Available only if the component has been [released](#release-a-component). -Component repositories are released using the [`release`](../yaml/index.md#release) keyword within a CI pipeline. +For example, for a component repository located at `gitlab-org/dast` on `gitlab.com`, +the path: -Like in the [example above](#test-a-component), after all tests pass in a pipeline running for a tag ref, we can release a new version of the components repository. +- `gitlab.com/gitlab-org/dast@main` targets the `template.yml` in the root directory + on the `main` branch. +- `gitlab.com/gitlab-org/dast@e3262fdd0914fa823210cdb79a8c421e2cef79d8` targets the same file + for the specified commit SHA. +- `gitlab.com/gitlab-org/dast@1.0` targets the same file for the `1.0` tag. +- `gitlab.com/gitlab-org/dast@~latest` targets the same file for the latest release. +- `gitlab.com/gitlab-org/dast/api-scan@main` targets a different file, the `template.yml` + in the `/api-scan` directory in the component repository, for the `main` branch. -All released versions of the components repository are displayed in the Components Catalog page for the given resource, providing users with information about official releases. +## Best practices -### Use a component in a CI/CD configuration +### Avoid using global keywords -A pipeline component is identified by a unique address in the form `<fully-qualified-doman-name>/<component-path>@<version>` -containing: +Avoid using [global keywords](../yaml/index.md#global-keywords) in a component. +Using these keywords in a component affects all jobs in a pipeline, including jobs +directly defined in the main `.gitlab-ci.yml` or in other included components. -- **A fully qualified domain name (FQDN)**: The FQDN must match the GitLab host. -- **A specific version**: The version of the component can be (in order of highest priority first): - - A commit SHA, for example `gitlab.com/gitlab-org/dast@e3262fdd0914fa823210cdb79a8c421e2cef79d8`. - - A tag. for example: `gitlab.com/gitlab-org/dast@1.0`. - - `~latest`, which is a special version that always points to the most recent released tag, - for example `gitlab.com/gitlab-org/dast@~latest`. - - A branch name, for example `gitlab.com/gitlab-org/dast@main`. -- **A component path**: Contains the project's full path and the directory where the component YAML file `template.yml` is located. +As an alternative to global keywords, instead: -For example, for a component repository located at `gitlab-org/dast` on `gitlab.com`: +- Add the configuration directly to each job, even if it creates some duplication + in the component configuration. +- Use the [`extends`](../yaml/index.md#extends) keyword in the component. -- The path `gitlab.com/gitlab-org/dast` tries to load the `template.yml` from the root directory. -- The path `gitlab.com/gitlab-org/dast/api-scan` tries to load the `template.yml` from the `/api-scan` directory. +For example, using the `default` keyword is not recommended: -**Additional notes:** +```yaml +# Not recommended +default: + image: ruby:3.0 -- You can only reference components in the same GitLab instance as your project. -- If a tag and branch exist with the same name, the tag takes precedence over the branch. -- If a tag is named the same as a commit SHA that exists, like `e3262fdd0914fa823210cdb79a8c421e2cef79d8`, - the commit SHA takes precedence over the tag. +rspec-1: + script: bundle exec rspec dir1/ -### Best practices +rspec-2: + script: bundle exec rspec dir2/ +``` -#### Avoid using global keywords +Instead, you can: -When using [global keywords](../yaml/index.md#global-keywords) all jobs in the -pipeline are affected. Using these keywords in a component affects all jobs in a -pipeline, whether they are directly defined in the main `.gitlab-ci.yml` or -in any included components. +- Add the configuration to each job: -To make the composition of pipelines more deterministic, either: + ```yaml + rspec-1: + image: ruby:3.0 + script: bundle exec rspec dir1/ -- Duplicate the default configuration for each job. -- Use [`extends`](../yaml/index.md#extends) feature within the component. + rspec-2: + image: ruby:3.0 + script: bundle exec rspec dir2/ + ``` -```yaml -## -# BAD -default: - image: ruby:3.0 +- Use `extends` to reuse configuration: -rspec: - script: bundle exec rspec -``` + ```yaml + .rspec-image: + image: ruby:3.0 -```yaml -## -# GOOD -rspec: - image: ruby:3.0 - script: bundle exec rspec -``` + rspec-1: + extends: + - .rspec-image + script: bundle exec rspec dir1/ -#### Replace hard-coded values with inputs + rspec-2: + extends: + - .rspec-image + script: bundle exec rspec dir2/ + ``` -A typical hard-coded value found in CI templates is `stage:` value. Such hard coded values may force the user -of the component to know and adapt the pipeline to such implementation details. +### Replace hard-coded values with inputs -For example, if `stage: test` is hard-coded for a job in a component, the pipeline using the component must -define the `test` stage. Additionally, if the user of the component want to customize the stage value it has -to override the configuration: +Avoid hard-coding values in CI/CD components. Hard-coded values might force +component users to need to review the component's internal details and adapt their pipeline +to work with the component. -```yaml -## -# BAD: In order to use different stage name you need to override all the jobs -# included by the component. -include: - - component: gitlab.com/gitlab-org/ruby-test@1.0 +A common keyword with problematic hard-coded values is `stage`. If a component job's +stage is set to a specific value, the pipeline using the component **must** define +the exact same stage. Additionally, if the component user wants to use a different stage, +they must [override](../yaml/includes.md#override-included-configuration-values) the configuration. -stages: [verify, deploy] +The preferred method is to use the [`input` keyword](../yaml/inputs.md). +The component user can specify the exact value they need. -unit-test: - stage: verify +For example: -integration-test: - stage: verify -``` +- In the component configuration: -```yaml -## -# BAD: In order to use the component correctly you need to define the stage -# that is hard-coded in it. -include: - - component: gitlab.com/gitlab-org/ruby-test@1.0 + ```yaml + spec: + inputs: + stage: + default: test + --- + unit-test: + stage: $[[ inputs.stage ]] + script: echo unit tests + + integration-test: + stage: $[[ inputs.stage ]] + script: echo integration tests + ``` -stages: [test, deploy] -``` +- In the project using the component: -To improve this we can use [input parameters](../yaml/includes.md#define-input-parameters-with-specinputs) -allowing the user of a component to inject values that can be customized: + ```yaml + include: + - component: gitlab.com/gitlab-org/ruby-test@1.0 + inputs: + stage: verify -```yaml -## -# GOOD: We don't need to know the implementation details of a component and instead we can -# rely on the inputs. -include: - - component: gitlab.com/gitlab-org/ruby-test@1.0 - inputs: - stage: verify + stages: [verify, deploy] + ``` -stages: [verify, deploy] +### Replace custom CI/CD variables with inputs -## -# inside the component YAML: -spec: - inputs: - stage: - default: test ---- -unit-test: - stage: $[[ inputs.stage ]] - script: echo unit tests +When using CI/CD variables in a component, evaluate if the `inputs` keyword +should be used instead. Avoid requiring a user to define custom variables to change a component's +behavior. You should try to use `inputs` for any component customization. -integration-test: - stage: $[[ inputs.stage ]] - script: echo integration tests -``` +Inputs are explicitly defined in the component's specs, and are better validated than variables. +For example, if a required input is not passed to the component, GitLab returns a pipeline error. +By contrast, if a variable is not defined, its value is empty, and there is no error. -#### Prefer inputs over variables +For example, use `inputs` instead of variables to let users change a scanner's output format: -If variables are only used for YAML evaluation (for example `rules`) and not by the Runner -execution, it's advised to use inputs instead. -Inputs are explicitly defined in the component's contract and they are better validated -than variables. +- In the component configuration: -For example, if a required input is not passed an error is returned as soon as the component -is being used. By contrast, if a variable is not defined, it's value is empty. + ```yaml + spec: + inputs: + scanner-output: + default: json + --- + my-scanner: + script: my-scan --output $[[ inputs.scanner-output ]] + ``` -```yaml -## -# BAD: you need to configure an environment variable for a custom value that doesn't need -# to be used on the Runner -unit-test: - image: $MY_COMPONENT_X_IMAGE - script: echo unit tests - -integration-test: - image: $MY_COMPONENT_X_IMAGE - script: echo integration tests - -## -# Usage: -include: - - component: gitlab.com/gitlab-org/ruby-test@1.0 +- In the project using the component: -variables: - MY_COMPONENT_X_IMAGE: ruby:3.2 -``` + ```yaml + include: + - component: gitlab.example.com/my-scanner@1.0 + inputs: + scanner-output: yaml + ``` -```yaml -## -# GOOD: we define a customizable value and accept it as input -spec: - inputs: - image: - default: ruby:3.0 ---- -unit-test: - image: $[[ inputs.image ]] - script: echo unit tests +In other cases, CI/CD variables are still preferred, including: -integration-test: - image: $[[ inputs.image ]] - script: echo integration tests +- Using [predefined variables](../variables/predefined_variables.md) to automatically configure + a component to match a user's project. +- Requiring tokens or other sensitive values to be stored as [masked or protected variables in project settings](../variables/index.md#define-a-cicd-variable-in-the-ui). -## -# Usage: -include: - - component: gitlab.com/gitlab-org/ruby-test@1.0 - inputs: - image: ruby:3.2 -``` +### Use semantic versioning -#### Use semantic versioning +When tagging and releasing new versions of components, you should use [semantic versioning](https://semver.org). +Semantic versioning is the standard for communicating that a change is a major, minor, patch, +or other kind of change. -When tagging and releasing new versions of components we recommend using [semantic versioning](https://semver.org) -which is the standard for communicating bugfixes, minor and major or breaking changes. +You should use at least the `major.minor` format, as this is widely understood. For example, +`2.0` or `2.1`. -We recommend adopting at least the `MAJOR.MINOR` format. +Other examples of semantic versioning: -For example: `2.1`, `1.0.0`, `1.0.0-alpha`, `2.1.3`, `3.0.0-rc.1`. +- `1.0.0` +- `2.1.3` +- `1.0.0-alpha` +- `3.0.0-rc1` -## CI/CD Catalog **(PREMIUM ALL)** +### Test the component -The CI/CD Catalog is a list of [components repositories](#components-repository), -each containing resources that you can add to your CI/CD pipelines. +Testing CI/CD components as part of the development workflow is strongly recommended +and helps ensure consistent behavior. -### Mark the project as a catalog resource +Test changes in a CI/CD pipeline (like any other project) by creating a `.gitlab-ci.yml` +in the root directory. -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/407249) in GitLab 16.1. +For example: -After components are added to a components repository, they can immediately be [used](#use-a-component-in-a-cicd-configuration) to build pipelines in other projects. +```yaml +include: + # include the component located in the current project from the current SHA + - component: gitlab.com/$CI_PROJECT_PATH@$CI_COMMIT_SHA + inputs: + stage: build -However, this repository is not discoverable. You must mark this project as a catalog resource to allow it to be visible in the CI Catalog -so other users can discover it. +stages: [build, test, release] -To mark a project as a catalog resource: +# Expect `component-job` is added. +# This example tests that the included component works as expected. +# You can inspect data generated by the component, use GitLab API endpoints or third-party tools. +ensure-job-added: + stage: test + image: badouralix/curl-jq + script: + - | + route="https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/pipelines/$CI_PIPELINE_ID/jobs" + count=`curl --silent --header "PRIVATE-TOKEN: $API_TOKEN" $route | jq 'map(select(.name | contains("component-job"))) | length'` + if [ "$count" != "1" ]; then + exit 1 + fi -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. -1. On the left sidebar, select **Settings > General**. -1. Expand **Visibility, project features, permissions**. -1. Scroll down to **CI/CD Catalog resource** and select the toggle to mark the project as a catalog resource. +# If we are tagging a release with a specific convention ("v" + number) and all +# previous checks succeeded, we proceed with creating a release automatically. +create-release: + stage: release + image: registry.gitlab.com/gitlab-org/release-cli:latest + rules: + - if: $CI_COMMIT_TAG =~ /^v\d+/ + script: echo "Creating release $CI_COMMIT_TAG" + release: + tag_name: $CI_COMMIT_TAG + description: "Release $CI_COMMIT_TAG of components repository $CI_PROJECT_PATH" +``` -NOTE: -This action is not reversible. +After committing and pushing changes, the pipeline tests the component, then releases it if the test passes. -## Convert a CI template to component +## Convert a CI/CD template to a component -Any existing CI template, that you share with other projects via `include:` syntax, can be converted to a CI component. +Any existing CI/CD template that you use in projects by using the `include:` syntax +can be converted to a CI/CD component: -1. Decide whether you want the component to be part of an existing [components repository](#components-repository), - if you want to logically group components together. Create and setup a [components repository](#components-repository) otherwise. -1. Create a YAML file in the components repository according to the expected [directory structure](#directory-structure). -1. Copy the content of the template YAML file into the new component YAML file. -1. Refactor the component YAML to follow the [best practices](#best-practices) for components. -1. Leverage the `.gitlab-ci.yml` in the components repository to [test changes to the component](#test-a-component). -1. Tag and [release the component](#release-a-component). +1. Decide if you want the component to be part of an existing [components repository](index.md#components-repository) + to be grouped with other components, or create and set up a new components repository. +1. Create a YAML file in the components repository according to the expected [directory structure](index.md#directory-structure). +1. Copy the content of the original template YAML file into the new component YAML file. +1. Refactor the new component's configuration to: + - Follow the [best practices](index.md#best-practices) for components. + - Improve the configuration, for example by enabling [merge request pipelines](../pipelines/merge_request_pipelines.md) + or making it [more efficient](../pipelines/pipeline_efficiency.md). +1. Leverage the `.gitlab-ci.yml` in the components repository to [test changes to the component](index.md#test-the-component). +1. Tag and [release the component](index.md#release-a-component). diff --git a/doc/ci/docker/authenticate_registry.md b/doc/ci/docker/authenticate_registry.md index 52cc3071fda..28edb5140dd 100644 --- a/doc/ci/docker/authenticate_registry.md +++ b/doc/ci/docker/authenticate_registry.md @@ -18,9 +18,9 @@ login`: ```yaml default: - image: docker:20.10.16 + image: docker:24.0.5 services: - - docker:20.10.16-dind + - docker:24.0.5-dind variables: DOCKER_TLS_CERTDIR: "/certs" @@ -42,7 +42,7 @@ empty or remove it. If you are an administrator for GitLab Runner, you can mount a file with the authentication configuration to `~/.docker/config.json`. Then every job that the runner picks up is already authenticated. If you -are using the official `docker:20.10.16` image, the home directory is +are using the official `docker:24.0.5` image, the home directory is under `/root`. If you mount the configuration file, any `docker` command @@ -126,9 +126,9 @@ The same commands apply for any solution you implement. ```yaml default: - image: docker:20.10.16 + image: docker:24.0.5 services: - - docker:20.10.16-dind + - docker:24.0.5-dind variables: DOCKER_TLS_CERTDIR: "/certs" diff --git a/doc/ci/docker/buildah_rootless_tutorial.md b/doc/ci/docker/buildah_rootless_tutorial.md new file mode 100644 index 00000000000..fdb33fd4a8b --- /dev/null +++ b/doc/ci/docker/buildah_rootless_tutorial.md @@ -0,0 +1,149 @@ +--- +stage: Verify +group: Pipeline Execution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: howto +--- + +# Tutorial: Use Buildah in a rootless container with GitLab Runner Operator on OpenShift **(FREE)** + +This tutorial teaches you how to successfully build images using the `buildah` tool, +with GitLab Runner deployed using [GitLab Runner Operator](https://gitlab.com/gitlab-org/gl-openshift/gitlab-runner-operator) +on an OpenShift cluster. + +This guide is an adaptation of [using Buildah to build images in a rootless OpenShift container](https://github.com/containers/buildah/blob/main/docs/tutorials/05-openshift-rootless-build.md) +documentation for GitLab Runner Operator. + +To complete this tutorial, you will: + +1. [Configure the Buildah image](#configure-the-buildah-image) +1. [Configure the service account](#configure-the-service-account) +1. [Configure the job](#configure-the-job) + +## Prerequisites + +- A runner already deployed to a `gitlab-runner` namespace. + +## Configure the Buildah image + +We start by preparing a custom image based on the `quay.io/buildah/stable:v1.23.1` image. + +1. Create the `Containerfile-buildah` file: + + ```shell + cat > Containerfile-buildah <<EOF + FROM quay.io/buildah/stable:v1.23.1 + + RUN touch /etc/subgid /etc/subuid \ + && chmod g=u /etc/subgid /etc/subuid /etc/passwd \ + && echo build:10000:65536 > /etc/subuid \ + && echo build:10000:65536 > /etc/subgid + + # Use chroot since the default runc does not work when running rootless + RUN echo "export BUILDAH_ISOLATION=chroot" >> /home/build/.bashrc + + # Use VFS since fuse does not work + RUN mkdir -p /home/build/.config/containers \ + && (echo '[storage]';echo 'driver = "vfs"') > /home/build/.config/containers/storage.conf + + # The buildah container will run as `build` user + USER build + WORKDIR /home/build + EOF + ``` + +1. Build and push the Buildah image to a Container Registry. Let's push to the + [GitLab Container Registry](../../user/packages/container_registry/index.md): + + ```shell + docker build -f Containerfile-buildah -t registry.example.com/group/project/buildah:1.23.1 . + docker push registry.example.com/group/project/buildah:1.23.1 + ``` + +## Configure the service account + +For these steps, you need to run the commands in a terminal connected to the OpenShift cluster. + +1. Run this command to create a service account named `buildah-sa`: + + ```shell + oc create -f - <<EOF + apiVersion: v1 + kind: ServiceAccount + metadata: + name: buildah-sa + namespace: gitlab-runner + EOF + ``` + +1. Give the created service account the ability to run with `anyuid` [SCC](https://docs.openshift.com/container-platform/4.3/authentication/managing-security-context-constraints.html): + + ```shell + oc adm policy add-scc-to-user anyuid -z buildah-sa -n gitlab-runner + ``` + +1. Use a [runner configuration template](https://docs.gitlab.com/runner/configuration/configuring_runner_operator.html#customize-configtoml-with-a-configuration-template) + to configure Operator to use the service account we just created. Create a `custom-config.toml` file that contains: + + ```toml + [[runners]] + [runners.kubernetes] + service_account_overwrite_allowed = "buildah-*" + ``` + +1. Create a `ConfigMap` named `custom-config-toml` from the `custom-config.toml` file: + + ```shell + oc create configmap custom-config-toml --from-file config.toml=custom-config.toml -n gitlab-runner + ``` + +1. Set the `config` property of the `Runner` by updating its [Custom Resource Definition (CRD) file](https://docs.gitlab.com/runner/install/operator.html#install-gitlab-runner): + + ```yaml + apiVersion: apps.gitlab.com/v1beta2 + kind: Runner + metadata: + name: builah-runner + spec: + gitlabUrl: https://gitlab.example.com + token: gitlab-runner-secret + config: custom-config-toml + ``` + +## Configure the job + +The final step is to set up a GitLab CI/CD configuration file in you project to use +the image we built and the configured service account: + +```yaml +build: + stage: build + image: registry.example.com/group/project/buildah:1.23.1 + variables: + STORAGE_DRIVER: vfs + BUILDAH_FORMAT: docker + BUILDAH_ISOLATION: chroot + FQ_IMAGE_NAME: "$CI_REGISTRY_IMAGE/test" + KUBERNETES_SERVICE_ACCOUNT_OVERWRITE: "buildah-sa" + before_script: + # Log in to the GitLab container registry + - buildah login -u "$CI_REGISTRY_USER" --password $CI_REGISTRY_PASSWORD $CI_REGISTRY + script: + - buildah images + - buildah build -t $FQ_IMAGE_NAME + - buildah images + - buildah push $FQ_IMAGE_NAME +``` + +The job should use the image that we built as the value of `image` keyword. + +The `KUBERNETES_SERVICE_ACCOUNT_OVERWRITE` variable should have the value of the +service account name that we created. + +Congratulations, you've successfully built an image with Buildah in a rootless container! + +## Troubleshooting + +There is a [known issue](https://github.com/containers/buildah/issues/4049) with running as non-root. +You might need to use a [workaround](https://docs.gitlab.com/runner/configuration/configuring_runner_operator.html#configure-setfcap) +if you are using an OpenShift runner. diff --git a/doc/ci/docker/docker_layer_caching.md b/doc/ci/docker/docker_layer_caching.md index 861bea70cb7..b34b58ce964 100644 --- a/doc/ci/docker/docker_layer_caching.md +++ b/doc/ci/docker/docker_layer_caching.md @@ -29,9 +29,9 @@ This example `.gitlab-ci.yml` file shows how to use Docker caching: ```yaml default: - image: docker:20.10.16 + image: docker:24.0.5 services: - - docker:20.10.16-dind + - docker:24.0.5-dind before_script: - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index fb38de0f7de..269ce2c3212 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -80,7 +80,8 @@ For more information, see [security of the `docker` group](https://blog.zopyx.co "Docker-in-Docker" (`dind`) means: -- Your registered runner uses the [Docker executor](https://docs.gitlab.com/runner/executors/docker.html) or the [Kubernetes executor](https://docs.gitlab.com/runner/executors/kubernetes.html). +- Your registered runner uses the [Docker executor](https://docs.gitlab.com/runner/executors/docker.html) or + the [Kubernetes executor](https://docs.gitlab.com/runner/executors/kubernetes.html). - The executor uses a [container image of Docker](https://hub.docker.com/_/docker/), provided by Docker, to run your CI/CD jobs. @@ -90,7 +91,7 @@ the job script in context of the image in privileged mode. You should use Docker-in-Docker with TLS enabled, which is supported by [GitLab.com shared runners](../runners/index.md). -You should always pin a specific version of the image, like `docker:20.10.16`. +You should always pin a specific version of the image, like `docker:24.0.5`. If you use a tag like `docker:latest`, you have no control over which version is used. This can cause incompatibility problems when new versions are released. @@ -121,12 +122,12 @@ To use Docker-in-Docker with TLS enabled: --registration-token REGISTRATION_TOKEN \ --executor docker \ --description "My Docker Runner" \ - --docker-image "docker:20.10.16" \ + --docker-image "docker:24.0.5" \ --docker-privileged \ --docker-volumes "/certs/client" ``` - - This command registers a new runner to use the `docker:20.10.16` image. + - This command registers a new runner to use the `docker:24.0.5` image (if none is specified at the job level). To start the build and service containers, it uses the `privileged` mode. If you want to use Docker-in-Docker, you must always use `privileged = true` in your Docker containers. @@ -143,7 +144,7 @@ To use Docker-in-Docker with TLS enabled: executor = "docker" [runners.docker] tls_verify = false - image = "docker:20.10.16" + image = "docker:24.0.5" privileged = true disable_cache = false volumes = ["/certs/client", "/cache"] @@ -153,13 +154,13 @@ To use Docker-in-Docker with TLS enabled: ``` 1. You can now use `docker` in the job script. You should include the - `docker:20.10.16-dind` service: + `docker:24.0.5-dind` service: ```yaml default: - image: docker:20.10.16 + image: docker:24.0.5 services: - - docker:20.10.16-dind + - docker:24.0.5-dind before_script: - docker info @@ -202,7 +203,7 @@ Assuming that the runner's `config.toml` is similar to: executor = "docker" [runners.docker] tls_verify = false - image = "docker:20.10.16" + image = "docker:24.0.5" privileged = true disable_cache = false volumes = ["/cache"] @@ -212,13 +213,13 @@ Assuming that the runner's `config.toml` is similar to: ``` You can now use `docker` in the job script. You should include the -`docker:20.10.16-dind` service: +`docker:24.0.5-dind` service: ```yaml default: - image: docker:20.10.16 + image: docker:24.0.5 services: - - docker:20.10.16-dind + - docker:24.0.5-dind before_script: - docker info @@ -276,13 +277,13 @@ To use Docker-in-Docker with TLS enabled in Kubernetes: ``` 1. You can now use `docker` in the job script. You should include the - `docker:20.10.16-dind` service: + `docker:24.0.5-dind` service: ```yaml default: - image: docker:20.10.16 + image: docker:24.0.5 services: - - docker:20.10.16-dind + - docker:24.0.5-dind before_script: - docker info @@ -330,7 +331,7 @@ Docker-in-Docker is the recommended configuration, but you should be aware of th - **Storage drivers**: By default, earlier versions of Docker use the `vfs` storage driver, which copies the file system for each job. Docker 17.09 and later use `--storage-driver overlay2`, which is the recommended storage driver. See [Using the OverlayFS driver](#use-the-overlayfs-driver) for details. -- **Root file system**: Because the `docker:20.10.16-dind` container and the runner container do not share their +- **Root file system**: Because the `docker:24.0.5-dind` container and the runner container do not share their root file system, you can use the job's working directory as a mount point for child containers. For example, if you have files you want to share with a child container, you could create a subdirectory under `/builds/$CI_PROJECT_PATH` @@ -352,7 +353,7 @@ container. Docker is then available in the context of the image. If you bind the Docker socket and you are [using GitLab Runner 11.11 or later](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1261), -you can no longer use `docker:20.10.16-dind` as a service. Volume bindings also affect services, +you can no longer use `docker:24.0.5-dind` as a service. Volume bindings also affect services, making them incompatible. To make Docker available in the context of the image, you need to mount @@ -369,7 +370,7 @@ Your configuration should look similar to this example: executor = "docker" [runners.docker] tls_verify = false - image = "docker:20.10.16" + image = "docker:24.0.5" privileged = false disable_cache = false volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"] @@ -385,7 +386,7 @@ sudo gitlab-runner register -n \ --registration-token REGISTRATION_TOKEN \ --executor docker \ --description "My Docker Runner" \ - --docker-image "docker:20.10.16" \ + --docker-image "docker:24.0.5" \ --docker-volumes /var/run/docker.sock:/var/run/docker.sock ``` @@ -408,7 +409,7 @@ mirror: ```yaml services: - - name: docker:20.10.16-dind + - name: docker:24.0.5-dind command: ["--registry-mirror", "https://registry-mirror.example.com"] # Specify the registry mirror to use ``` @@ -431,7 +432,7 @@ Docker: ... privileged = true [[runners.docker.services]] - name = "docker:20.10.16-dind" + name = "docker:24.0.5-dind" command = ["--registry-mirror", "https://registry-mirror.example.com"] ``` @@ -445,7 +446,7 @@ Kubernetes: ... privileged = true [[runners.kubernetes.services]] - name = "docker:20.10.16-dind" + name = "docker:24.0.5-dind" command = ["--registry-mirror", "https://registry-mirror.example.com"] ``` @@ -552,12 +553,12 @@ the implications of this method are: docker run --rm -t -i -v $(pwd)/src:/home/app/src test-image:latest run_app_tests ``` -You do not need to include the `docker:20.10.16-dind` service, like you do when +You do not need to include the `docker:24.0.5-dind` service, like you do when you use the Docker-in-Docker executor: ```yaml default: - image: docker:20.10.16 + image: docker:24.0.5 before_script: - docker info @@ -638,30 +639,43 @@ To build Docker images without enabling privileged mode on the runner, you can use one of these alternatives: - [`kaniko`](using_kaniko.md). -- [`buildah`](https://github.com/containers/buildah). There is a [known issue](https://github.com/containers/buildah/issues/4049) with running as non-root, you might need this [workaround](https://docs.gitlab.com/runner/configuration/configuring_runner_operator.html#configure-setfcap) if you are using OpenShift Runner. +- [`buildah`](#buildah-example). -For example, with `buildah`: +### Buildah example -```yaml -# Some details from https://major.io/2019/05/24/build-containers-in-gitlab-ci-with-buildah/ +To use Buildah with GitLab CI/CD, you need [a runner](https://docs.gitlab.com/runner/) with one +of the following executors: + +- [Kubernetes](https://docs.gitlab.com/runner/executors/kubernetes.html). +- [Docker](https://docs.gitlab.com/runner/executors/docker.html). +- [Docker Machine](https://docs.gitlab.com/runner/executors/docker_machine.html). + +In this example, you use Buildah to: + +1. Build a Docker image. +1. Push it to [GitLab Container Registry](../../user/packages/container_registry/index.md). +In the last step, Buildah uses the `Dockerfile` under the +root directory of the project to build the Docker image. Finally, it pushes the image to the +project's Container Registry: + +```yaml build: stage: build image: quay.io/buildah/stable variables: - # Use vfs with buildah. Docker offers overlayfs as a default, but buildah + # Use vfs with buildah. Docker offers overlayfs as a default, but Buildah # cannot stack overlayfs on top of another overlayfs filesystem. STORAGE_DRIVER: vfs # Write all image metadata in the docker format, not the standard OCI format. # Newer versions of docker can handle the OCI format, but older versions, like # the one shipped with Fedora 30, cannot handle the format. BUILDAH_FORMAT: docker - # You may need this workaround for some errors: https://stackoverflow.com/a/70438141/1233435 - BUILDAH_ISOLATION: chroot FQ_IMAGE_NAME: "$CI_REGISTRY_IMAGE/test" before_script: - # Log in to the GitLab container registry - - export REGISTRY_AUTH_FILE=$HOME/auth.json + # GitLab Container Registry credentials taken from the + # [predefined CI/CD variables](../variables/index.md#predefined-cicd-variables) + # to authenticate to the registry. - echo "$CI_REGISTRY_PASSWORD" | buildah login -u "$CI_REGISTRY_USER" --password-stdin $CI_REGISTRY script: - buildah images @@ -670,6 +684,9 @@ build: - buildah push $FQ_IMAGE_NAME ``` +If you are using GitLab Runner Operator deployed to an OpenShift cluster, try the +[tutorial for using Buildah to build images in rootless container](buildah_rootless_tutorial.md). + ## Use the GitLab Container Registry After you've built a Docker image, you can push it to the @@ -702,9 +719,9 @@ This issue can occur when the service's image name ```yaml default: - image: docker:20.10.16 + image: docker:24.0.5 services: - - registry.hub.docker.com/library/docker:20.10.16-dind + - registry.hub.docker.com/library/docker:24.0.5-dind ``` A service's hostname is [derived from the full image name](../../ci/services/index.md#accessing-the-services). @@ -713,15 +730,94 @@ To allow service resolution and access, add an explicit alias for the service na ```yaml default: - image: docker:20.10.16 + image: docker:24.0.5 services: - - name: registry.hub.docker.com/library/docker:20.10.16-dind + - name: registry.hub.docker.com/library/docker:24.0.5-dind alias: docker ``` +### `Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?` + +You might get the following error when trying to run a `docker` command +to access a `dind` service: + +```shell +$ docker ps +Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? +``` + +Make sure your job has defined these environment variables: + +- `DOCKER_HOST` +- `DOCKER_TLS_CERTDIR` (optional) +- `DOCKER_TLS_VERIFY` (optional) + +You may also want to update the image that provides the Docker +client. For example, the [`docker/compose` images are obsolete](https://hub.docker.com/r/docker/compose) and should be +replaced with [`docker`](https://hub.docker.com/_/docker). + +As described in [runner issue 30944](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/30944#note_1514250909), +this error can happen if your job previously relied on environment variables derived from the deprecated +[Docker `--link` parameter](https://docs.docker.com/network/links/#environment-variables), +such as `DOCKER_PORT_2375_TCP`. Your job fails with this error if: + +- Your CI/CD image relies on a legacy variable, such as `DOCKER_PORT_2375_TCP`. +- The [runner feature flag `FF_NETWORK_PER_BUILD`](https://docs.gitlab.com/runner/configuration/feature-flags.html) is set to `true`. +- `DOCKER_HOST` is not explicitly set. + ### Error: `Error response from daemon: Get "https://registry-1.docker.io/v2/": unauthorized: incorrect username or password` This error appears when you use the deprecated variable, `CI_BUILD_TOKEN`. To prevent users from receiving this error, you should: - Use [CI_JOB_TOKEN](../jobs/ci_job_token.md) instead. - Change from `gitlab-ci-token/CI_BUILD_TOKEN` to `$CI_REGISTRY_USER/$CI_REGISTRY_PASSWORD`. + +### Error: `error during connect: Post "https://docker:2376/v1.24/auth": dial tcp: lookup docker on 127.0.0.11:53: no such host` + +This error appears when the `dind` service has failed to start. Check +the job log to see if `mount: permission denied (are you root?)` +appears. For example: + +```plaintext +Service container logs: +2023-08-01T16:04:09.541703572Z Certificate request self-signature ok +2023-08-01T16:04:09.541770852Z subject=CN = docker:dind server +2023-08-01T16:04:09.556183222Z /certs/server/cert.pem: OK +2023-08-01T16:04:10.641128729Z Certificate request self-signature ok +2023-08-01T16:04:10.641173149Z subject=CN = docker:dind client +2023-08-01T16:04:10.656089908Z /certs/client/cert.pem: OK +2023-08-01T16:04:10.659571093Z ip: can't find device 'ip_tables' +2023-08-01T16:04:10.660872131Z modprobe: can't change directory to '/lib/modules': No such file or directory +2023-08-01T16:04:10.664620455Z mount: permission denied (are you root?) +2023-08-01T16:04:10.664692175Z Could not mount /sys/kernel/security. +2023-08-01T16:04:10.664703615Z AppArmor detection and --privileged mode might break. +2023-08-01T16:04:10.665952353Z mount: permission denied (are you root?) +``` + +This indicates the GitLab Runner does not have permission to start the +`dind` service: + +1. Check that `privileged = true` is set in the `config.toml`. +1. Make sure the CI job has the right Runner tags to use these +privileged runners. + +### Error: `cgroups: cgroup mountpoint does not exist: unknown` + +There is a known incompatibility introduced by Docker Engine 20.10. + +When the host uses Docker Engine 20.10 or newer, then the `docker:dind` service in a version older than 20.10 does +not work as expected. + +While the service itself will start without problems, trying to build the container image results in the error: + +```plaintext +cgroups: cgroup mountpoint does not exist: unknown +``` + +To resolve this issue, update the `docker:dind` container to version at least 20.10.x, +for example `docker:24.0.5-dind`. + +The opposite configuration (`docker:24.0.5-dind` service and Docker Engine on the host in version +19.06.x or older) works without problems. For the best strategy, you should to frequently test and update +job environment versions to the newest. This brings new features, improved security and - for this specific +case - makes the upgrade on the underlying Docker Engine on the runner's host transparent for the job. diff --git a/doc/ci/docker/using_kaniko.md b/doc/ci/docker/using_kaniko.md index 568f4977c2f..8ab13c7154d 100644 --- a/doc/ci/docker/using_kaniko.md +++ b/doc/ci/docker/using_kaniko.md @@ -58,7 +58,7 @@ project's Container Registry while tagging it with the Git tag: build: stage: build image: - name: gcr.io/kaniko-project/executor:v1.9.0-debug + name: gcr.io/kaniko-project/executor:v1.14.0-debug entrypoint: [""] script: - /kaniko/executor @@ -96,7 +96,7 @@ build: https_proxy: <your-proxy> no_proxy: <your-no-proxy> image: - name: gcr.io/kaniko-project/executor:v1.9.0-debug + name: gcr.io/kaniko-project/executor:v1.14.0-debug entrypoint: [""] script: - /kaniko/executor @@ -158,3 +158,25 @@ on what other GitLab CI patterns are demonstrated are available at the project p If you receive this error, it might be due to an outside proxy. Setting the `http_proxy` and `https_proxy` [environment variables](../../administration/packages/container_registry.md#running-the-docker-daemon-with-a-proxy) can fix the problem. + +### Error: `kaniko should only be run inside of a container, run with the --force flag if you are sure you want to continue` + +There is a known incompatibility introduced by Docker Engine 20.10. + +When the host uses Docker Engine 20.10 or newer, then the `gcr.io/kaniko-project/executor:debug` image in a version +older than v1.9.0 does not work as expected. + +When you try to build the image, Kaniko fails with: + +```plaintext +kaniko should only be run inside of a container, run with the --force flag if you are sure you want to continue +``` + +To resolve this issue, update the `gcr.io/kaniko-project/executor:debug` container to version at least v1.9.0, +for example `gcr.io/kaniko-project/executor:v1.14.0-debug`. + +The opposite configuration (`gcr.io/kaniko-project/executor:v1.14.0-debug` image and Docker Engine +on the host in version 19.06.x or older) works without problems. For the best strategy, you should +frequently test and update job environment versions to the newest. This brings new features, improved +security and - for this specific case - makes the upgrade on underlying Docker Engine on the runner's +host transparent for the job. diff --git a/doc/ci/enable_or_disable_ci.md b/doc/ci/enable_or_disable_ci.md index 395a07f48c8..3081b8d1b39 100644 --- a/doc/ci/enable_or_disable_ci.md +++ b/doc/ci/enable_or_disable_ci.md @@ -30,7 +30,7 @@ When you disable GitLab CI/CD: To disable GitLab CI/CD in your project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. Expand **Visibility, project features, permissions**. 1. In the **Repository** section, turn off **CI/CD**. @@ -40,7 +40,7 @@ To disable GitLab CI/CD in your project: To enable GitLab CI/CD in your project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. Expand **Visibility, project features, permissions**. 1. In the **Repository** section, turn on **CI/CD**. diff --git a/doc/ci/environments/deployment_approvals.md b/doc/ci/environments/deployment_approvals.md index 2933b25e09b..754dcafb9f7 100644 --- a/doc/ci/environments/deployment_approvals.md +++ b/doc/ci/environments/deployment_approvals.md @@ -8,32 +8,24 @@ description: Require approvals prior to deploying to a Protected Environment # Deployment approvals **(PREMIUM ALL)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/343864) in GitLab 14.7 with a flag named `deployment_approvals`. Disabled by default. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/347342) in GitLab 14.8. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/347342) in GitLab 14.8. Feature flag `deployment_approvals` removed. -It may be useful to require additional approvals before deploying to certain protected environments (for example, production). This pre-deployment approval requirement is useful to accommodate testing, security, or compliance processes that must happen before each deployment. +You can require additional approvals for deployments to protected +environments. Deployments are blocked until all required approvals are +given. -When a protected environment requires one or more approvals, all deployments to that environment become blocked and wait for the required approvals from the `Allowed to Deploy` list before running. +Use deployment approvals to accommodate testing, +security, or compliance processes. For example, you might want to +require approvals for deployments to production environments. -NOTE: -See the [epic](https://gitlab.com/groups/gitlab-org/-/epics/6832) for planned features. - -## Prerequisites - -- Basic knowledge of [GitLab Environments and Deployments](index.md). -- Basic knowledge of [Protected Environments](protected_environments.md). +## Configure deployment approvals -## Configure deployment approvals for a project +You can require approvals for deployments to protected environments in +a project. To configure deployment approvals for a project: -1. [Create a deployment job](#create-a-deployment-job). -1. [Require approvals for a protected environment](#require-approvals-for-a-protected-environment). - -### Create a deployment job - -Create a deployment job in the `.gitlab-ci.yml` file of the desired project. The job does **not** need to be manual (`when: manual`). - -Example: +1. Create a deployment job in the `.gitlab-ci.yml` file of your project: ```yaml stages: @@ -47,22 +39,15 @@ Example: name: ${CI_JOB_NAME} ``` -### Require approvals for a protected environment + The job does not need to be manual (`when: manual`). -There are two ways to configure the approval requirements: +1. Add the required [approval rules](#multiple-approval-rules). -- [Unified approval setting](#unified-approval-setting-deprecated) ... You can define who can execute **and** approve deployments. - This is useful when there is no separation of duties between executors and approvers in your organization. -- [Multiple approval rules](#multiple-approval-rules) ... You can define who can execute **or** approve deployments. - This is useful when there is a separation of duties between executors and approvers in your organization. - -NOTE: -Multiple approval rules is a more flexible option than the unified approval setting, thus both configurations shouldn't -co-exist and multiple approval rules takes the precedence over the unified approval setting if it happens. +The environments in your project require approval before deployment. <!--- start_remove The following content will be removed on remove_date: '2024-05-22' --> -#### Unified approval setting (deprecated) +### Unified approval setting (deprecated) > - UI configuration [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/378447) in GitLab > 15.11. @@ -94,7 +79,7 @@ Maintainer role. <!--- end_remove --> -#### Multiple approval rules +### Multiple approval rules > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345678) in GitLab 14.10 with a flag named `deployment_approval_rules`. Disabled by default. > - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/345678) in GitLab 15.0. [Feature flag `deployment_approval_rules`](https://gitlab.com/gitlab-org/gitlab/-/issues/345678) removed. @@ -107,7 +92,7 @@ Maintainer role. - **Allowed to deploy** sets which entities can execute the deployment job. - **Approvers** sets which entities can approve the deployment job. -After this is configured, all jobs deploying to this environment automatically go into a blocked state and wait for approvals before running. Ensure that the number of required approvals is less than the number of users allowed to deploy. +After this is configured, all jobs deploying to this environment automatically go into a blocked state and wait for approvals before running. Ensure that the number of required approvals is less than the number of users allowed to deploy. Once a deployment job is approved, it must be [run manually](../jobs/job_control.md#run-a-manual-job). A configuration that uses the REST API might look like: @@ -128,7 +113,7 @@ NOTE: To protect, update, or unprotect an environment, you must have at least the Maintainer role. -#### Migrate to multiple approval rules +### Migrate to multiple approval rules You can migrate a protected environment from unified approval rules to multiple approval rules. Unified approval rules allow all entities that can deploy to an @@ -137,7 +122,7 @@ create a new approval rule for each entity allowed to deploy to the environment. To migrate with the UI: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Protected environments**. 1. From the **Environment** list, select your environment. @@ -171,7 +156,7 @@ require `Administrator` to approve every deployment job in `Production`. By default, the user who triggers a deployment pipeline can't also approve the deployment job. To allow self-approval of a deployment job: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Protected environments**. 1. From the **Approval options**, select the **Allow pipeline triggerer to approve deployment** checkbox. @@ -196,7 +181,7 @@ Prerequisites: To approve or reject a deployment to a protected environment using the UI: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Operate > Environments**. 1. Select the environment's name. 1. In the deployment's row, select **Approval options** (**{thumb-up}**). @@ -233,7 +218,7 @@ granted. To view the approval details of a deployment: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Operate > Environments**. 1. Select the environment's name. 1. In the deployment's row, select **Approval options** (**{thumb-up}**). @@ -249,7 +234,7 @@ The approval status details are shown: ### Using the UI -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Operate > Environments**. 1. Select the environment being deployed to. 1. Look for the `blocked` label. @@ -265,9 +250,9 @@ Use the [Deployments API](../../api/deployments.md#get-a-specific-deployment) to - When the [multiple approval rules](#multiple-approval-rules) is configured: - The `approval_summary` field contains the current approval status per rule. -## Related features +## Related topics -For details about other GitLab features aimed at protecting deployments, see [safe deployments](deployment_safety.md). +- [Deployment approvals feature epic](https://gitlab.com/groups/gitlab-org/-/epics/6832) <!-- ## Troubleshooting diff --git a/doc/ci/environments/environments_dashboard.md b/doc/ci/environments/environments_dashboard.md index e98205f45b9..0beaf2b8888 100644 --- a/doc/ci/environments/environments_dashboard.md +++ b/doc/ci/environments/environments_dashboard.md @@ -20,7 +20,7 @@ see which pipelines are green and which are red allowing you to diagnose if there is a block at a particular point, or if there's a more systemic problem you need to investigate. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Your work**. 1. Select **Environments**. diff --git a/doc/ci/environments/incremental_rollouts.md b/doc/ci/environments/incremental_rollouts.md index a498e525b54..309e311e252 100644 --- a/doc/ci/environments/incremental_rollouts.md +++ b/doc/ci/environments/incremental_rollouts.md @@ -22,9 +22,9 @@ Manual and Timed rollouts are included automatically in projects controlled by [Auto DevOps](../../topics/autodevops/index.md), but they are also configurable through GitLab CI/CD in the `.gitlab-ci.yml` configuration file. -Manually triggered rollouts can be implemented with your [Continuous Delivery](../introduction/index.md#continuous-delivery) -methodology, while timed rollouts do not require intervention and can be part of your -[Continuous Deployment](../introduction/index.md#continuous-deployment) strategy. +Manually triggered rollouts can be implemented with Continuous Delivery, +while timed rollouts do not require intervention and can be part of your +Continuous Deployment strategy. You can also combine both of them in a way that the app is deployed automatically unless you eventually intervene manually if necessary. diff --git a/doc/ci/environments/index.md b/doc/ci/environments/index.md index c91018980f0..713d58de326 100644 --- a/doc/ci/environments/index.md +++ b/doc/ci/environments/index.md @@ -51,7 +51,7 @@ Deployments show up in this list only after a deployment job has created them. To search environments by name: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Operate > Environments**. 1. In the search bar, enter your search term. - The length of your **search term should be 3 or more characters**. @@ -93,7 +93,7 @@ Prerequisites: To create a static environment in the UI: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Operate > Environments**. 1. Select **Create an environment**. 1. Complete the fields. @@ -339,13 +339,19 @@ It points to the commit you're rolling back to. For the rollback to succeed, the deployment process must be defined in the job's `script`. +Only the [deployment jobs](../jobs/index.md#deployment-jobs) are run. +In cases where a previous job generates artifacts that must be regenerated +on deploy, you must manually run the necessary jobs from the pipelines page. +For example, if you use Terraform and your `plan` and `apply` commands are separated +into multiple jobs, you must manually run the jobs to deploy or roll back. + #### Retry or roll back a deployment If there is a problem with a deployment, you can retry it or roll it back. To retry or roll back a deployment: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Operate > Environments**. 1. Select the environment. 1. To the right of the deployment name: @@ -359,7 +365,7 @@ In this case, see [job retries for rollback deployments](deployment_safety.md#jo ### Environment URL -> - [Fixed](https://gitlab.com/gitlab-org/gitlab/-/issues/337417) to persist arbitrary URLs in GitLab 15.2 [with a flag](../../administration/feature_flags.md) named `soft_validation_on_external_url`. Disabled by default. +> - [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/337417) to persist arbitrary URLs in GitLab 15.2 [with a flag](../../administration/feature_flags.md) named `soft_validation_on_external_url`. Disabled by default. > - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/337417) in GitLab 15.3. [Feature flag `soft_validation_on_external_url`](https://gitlab.com/gitlab-org/gitlab/-/issues/367206) removed. The [environment URL](../yaml/index.md#environmenturl) is displayed in a few @@ -560,7 +566,7 @@ you can view its expiration date and time. To view an environment's expiration date and time: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Operate > Environments**. 1. Select the name of the environment. @@ -573,7 +579,7 @@ you can override its expiration. To override an environment's expiration: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Operate > Environments**. 1. Select the deployment name. 1. in the upper-right corner, select the thumbtack (**{thumbtack}**). @@ -600,7 +606,7 @@ Environments view, the stop and deploy jobs must be in the same To stop an environment in the GitLab UI: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Operate > Environments**. 1. Next to the environment you want to stop, select **Stop**. 1. On the confirmation dialog, select **Stop environment**. @@ -664,7 +670,7 @@ Prerequisites: To delete an environment: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Operate > Environments**. 1. Select the **Stopped** tab. 1. Next to the environment you want to delete, select **Delete environment**. @@ -767,7 +773,7 @@ Limitations of GitLab Auto Rollback: GitLab Auto Rollback is turned off by default. To turn it on: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Automatic deployment rollbacks**. 1. Select the checkbox for **Enable automatic rollbacks**. @@ -910,6 +916,7 @@ the `review/feature-1` spec takes precedence over `review/*` and `*` specs. ## Related topics - [Dashboard for Kubernetes](kubernetes_dashboard.md) +- [Downstream pipelines for deployments](../pipelines/downstream_pipelines.md#downstream-pipelines-for-deployments) - [Deploy to multiple environments with GitLab CI/CD (blog post)](https://about.gitlab.com/blog/2021/02/05/ci-deployment-and-environments/) - [Review Apps](../review_apps/index.md) - [Protected environments](protected_environments.md) diff --git a/doc/ci/environments/kubernetes_dashboard.md b/doc/ci/environments/kubernetes_dashboard.md index 98b6731e469..0f9e1d808ec 100644 --- a/doc/ci/environments/kubernetes_dashboard.md +++ b/doc/ci/environments/kubernetes_dashboard.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Dashboard for Kubernetes (Beta) **(FREE ALL)** +# Dashboard for Kubernetes **(FREE ALL BETA)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/390769) in GitLab 16.1, with [flags](../../administration/feature_flags.md) named `environment_settings_to_graphql`, `kas_user_access`, `kas_user_access_project`, and `expose_authorized_cluster_agents`. This feature is in [Beta](../../policy/experiment-beta-support.md#beta). > - Feature flag `environment_settings_to_graphql` [removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/124177) in GitLab 16.2. @@ -15,17 +15,12 @@ Use the Dashboard for Kubernetes to understand the status of your clusters with The dashboard works with every connected Kubernetes cluster, whether you deployed them with CI/CD or GitOps. -For Flux users, the synchronization status of a given environment is not displayed in the dashboard. -[Issue 391581](https://gitlab.com/gitlab-org/gitlab/-/issues/391581) proposes to add this functionality. - ## Configure a dashboard > - Filtering resources by namespace [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/403618) in GitLab 16.2 [with a flag](../../administration/feature_flags.md) named `kubernetes_namespace_for_environment`. Disabled by default. > - Filtering resources by namespace [enabled by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/127043) in GitLab 16.3. Feature flag `kubernetes_namespace_for_environment` removed. -> - Selecting the related Flux resource [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/128857) in GitLab 16.3 [with a flag](../../administration/feature_flags.md) named `flux_resource_for_environment`. Disabled by default. - -FLAG: -On self-managed GitLab, by default selecting a Flux resource is not available. To make it available, an administrator can [enable the feature flag](../../administration/feature_flags.md) named `flux_resource_for_environment`. On GitLab.com, this feature is not available. +> - Selecting the related Flux resource [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/128857) in GitLab 16.3 [with a flag](../../administration/feature_flags.md) named `flux_resource_for_environment`. +> - Selecting the related Flux resource [generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/130648) in GitLab 16.4. Feature flag `flux_resource_for_environment` removed. Configure a dashboard to use it for a given environment. You can configure dashboard for an environment that already exists, or @@ -38,9 +33,9 @@ Prerequisites: ### The environment already exists -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Operate > Environments**. -1. Select the environment to be associated with the Kubernetes. +1. Select the environment to be associated with the agent for Kubernetes. 1. Select **Edit**. 1. Select a GitLab agent for Kubernetes. 1. Optional. From the **Kubernetes namespace** dropdown list, select a namespace. @@ -49,7 +44,7 @@ Prerequisites: ### The environment doesn't exist -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Operate > Environments**. 1. Select **New environment**. 1. Complete the **Name** field. @@ -62,17 +57,25 @@ Prerequisites: To view a configured dashboard: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Operate > Environments**. -1. Expand the environment associated with GitLab agent for Kubernetes. +1. Expand the environment associated with the agent for Kubernetes. 1. Expand **Kubernetes overview**. ### Flux sync status > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/391581) in GitLab 16.3. -> - Customizing the name of the Flux resource [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/128857) in GitLab 16.3 [with a flag](../../administration/feature_flags.md) named `flux_resource_for_environment`. Disabled by default. +> - Customizing the name of the Flux resource [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/128857) in GitLab 16.3 [with a flag](../../administration/feature_flags.md) named `flux_resource_for_environment`. +> - Customizing the name of the Flux resource [generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/130648) in GitLab 16.4. Feature flag `flux_resource_for_environment` removed. -A dashboard displays the sync status of your Flux deployments. +You can review the sync status of your Flux deployments from a dashboard. +To display the deployment status, your dashboard must be able to retrieve the `Kustomization` and `HelmRelease` resources, +which requires a namespace to be configured for the environment. + +By default, GitLab searches the `Kustomization` and `HelmRelease` resources for the name of the project slug. +You can specify the resource names with the **Flux resource** dropdown list in the environment settings. + +A dashboard displays one of the following status badges: | Status | Description | |---------|-------------| @@ -83,11 +86,6 @@ A dashboard displays the sync status of your Flux deployments. | **Unknown** | The sync status of the deployment couldn't be retrieved. | | **Unavailable** | The `Kustomization` or `HelmRelease` resource couldn't be retrieved. | -Deployments rely on Flux `Kustomization` and `HelmRelease` resources to gather -the status of a given environment, which requires a namespace to be configured for the environment. -By default, GitLab searches the `Kustomization` and `HelmRelease` resources for the name of the project slug. -You can customize the name GitLab looks for in the environment settings. - ## Troubleshooting When working with the Dashboard for Kubernetes, you might encounter the following issues. diff --git a/doc/ci/environments/protected_environments.md b/doc/ci/environments/protected_environments.md index fd789ea798d..21b58ac5ab4 100644 --- a/doc/ci/environments/protected_environments.md +++ b/doc/ci/environments/protected_environments.md @@ -30,7 +30,7 @@ Prerequisites: To protect an environment: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Protected environments**. 1. Select **Protect an environment**. @@ -256,7 +256,7 @@ To protect a group-level environment, make sure your environments have the corre > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/325249) in GitLab 15.1. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > CI/CD**. 1. Expand **Protected environments**. 1. From the **Environment** list, select the [deployment tier of environments](index.md#deployment-tier-of-environments) you want to protect. diff --git a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md index 737c95cf747..647669385d8 100644 --- a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md +++ b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md @@ -9,7 +9,7 @@ type: tutorial WARNING: Authenticating with `CI_JOB_JWT` was [deprecated in GitLab 15.9](../../../update/deprecations.md#old-versions-of-json-web-tokens-are-deprecated) -and the token is scheduled to be removed in GitLab 16.5. Use +and the token is scheduled to be removed in GitLab 17.0. Use [ID tokens to authenticate with HashiCorp Vault](../../secrets/id_token_authentication.md#automatic-id-token-authentication-with-hashicorp-vault) instead, as demonstrated on this page. diff --git a/doc/ci/examples/deployment/index.md b/doc/ci/examples/deployment/index.md index f9360faedac..2be800efcbb 100644 --- a/doc/ci/examples/deployment/index.md +++ b/doc/ci/examples/deployment/index.md @@ -121,7 +121,7 @@ We also use two secure variables: To store API keys as secure variables: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Variables**. diff --git a/doc/ci/examples/semantic-release.md b/doc/ci/examples/semantic-release.md index b284e2b2dc1..356a3d1d63e 100644 --- a/doc/ci/examples/semantic-release.md +++ b/doc/ci/examples/semantic-release.md @@ -96,7 +96,7 @@ As part of publishing a package, semantic-release increases the version number i 1. Under **select scopes**, select the **api** checkbox. 1. Select **Create project access token**. 1. Copy the value. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Variables**. 1. Select **Add variable**. diff --git a/doc/ci/index.md b/doc/ci/index.md index 21fe122f503..8d9108e4fc5 100644 --- a/doc/ci/index.md +++ b/doc/ci/index.md @@ -6,174 +6,86 @@ description: "Learn how to use GitLab CI/CD, the GitLab built-in Continuous Inte type: index --- -# GitLab CI/CD **(FREE ALL)** - -GitLab CI/CD is a tool for software development using the continuous methodologies: - -- [Continuous Integration (CI)](introduction/index.md#continuous-integration) -- [Continuous Delivery (CD)](introduction/index.md#continuous-delivery) -- [Continuous Deployment (CD)](introduction/index.md#continuous-deployment) - -NOTE: -Out-of-the-box management systems can decrease hours spent on maintaining toolchains by 10% or more. -Watch our ["Mastering continuous software development"](https://about.gitlab.com/webcast/mastering-ci-cd/) -webcast to learn about continuous methods and how GitLab CI/CD can help you simplify and scale software development. - -Use GitLab CI/CD to catch bugs and errors early in -the development cycle. Ensure that all the code deployed to -production complies with the code standards you established for -your app. - -GitLab CI/CD can automatically build, test, deploy, and -monitor your applications by using [Auto DevOps](../topics/autodevops/index.md). - -For a complete overview of these methodologies and GitLab CI/CD, -read the [Introduction to CI/CD with GitLab](introduction/index.md). - -<div class="video-fallback"> - Video demonstration of continuous integration with GitLab CI/CD: <a href="https://www.youtube.com/watch?v=ljth1Q5oJoo">Continuous Integration with GitLab (overview demo)</a>. -</div> -<figure class="video-container"> - <iframe src="https://www.youtube-nocookie.com/embed/ljth1Q5oJoo" frameborder="0" allowfullscreen> </iframe> -</figure> - -## Concepts - -GitLab CI/CD uses a number of concepts to describe and run your build and deploy. - -| Concept | Description | -|:--------------------------------------------------------|:--------------------------------------------------------------------------------------| -| [Pipelines](pipelines/index.md) | Structure your CI/CD process through pipelines. | -| [CI/CD variables](variables/index.md) | Reuse values based on a variable/value key pair. | -| [Environments](environments/index.md) | Deploy your application to different environments (for example, staging, production). | -| [Job artifacts](jobs/job_artifacts.md) | Output, use, and reuse job artifacts. | -| [Cache dependencies](caching/index.md) | Cache your dependencies for a faster execution. | -| [GitLab Runner](https://docs.gitlab.com/runner/) | Configure your own runners to execute your scripts. | -| [Pipeline efficiency](pipelines/pipeline_efficiency.md) | Configure your pipelines to run quickly and efficiently. | -| [Test cases](test_cases/index.md) | Create testing scenarios. | - -## Configuration - -GitLab CI/CD supports numerous configuration options: - -| Configuration | Description | -|:---------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------| -| [Schedule pipelines](pipelines/schedules.md) | Schedule pipelines to run as often as you need. | -| [Custom path for `.gitlab-ci.yml`](pipelines/settings.md#specify-a-custom-cicd-configuration-file) | Define a custom path for the CI/CD configuration file. | -| [Git submodules for CI/CD](git_submodules.md) | Configure jobs for using Git submodules. | -| [SSH keys for CI/CD](ssh_keys/index.md) | Using SSH keys in your CI pipelines. | -| [Pipeline triggers](triggers/index.md) | Trigger pipelines through the API. | -| [Merge request pipelines](pipelines/merge_request_pipelines.md) | Design a pipeline structure for running a pipeline in merge requests. | -| [Integrate with Kubernetes clusters](../user/infrastructure/clusters/index.md) | Connect your project to Google Kubernetes Engine (GKE) or an existing Kubernetes cluster. | -| [Optimize GitLab and GitLab Runner for large repositories](large_repositories/index.md) | Recommended strategies for handling large repositories. | -| [`.gitlab-ci.yml` full reference](yaml/index.md) | All the attributes you can use with GitLab CI/CD. | - -Certain operations can only be performed according to the -[user](../user/permissions.md#gitlab-cicd-permissions) and [job](../user/permissions.md#job-permissions) permissions. - -## Features - -GitLab CI/CD features, grouped by DevOps stage, include: - -| Feature | Description | -|:---------------------------------------------------------------------------------------------|:------------| -| **Configure** | | -| [Auto DevOps](../topics/autodevops/index.md) | Set up your app's entire lifecycle. | -| [ChatOps](chatops/index.md) | Trigger CI jobs from chat, with results sent back to the channel. | -| [Connect to cloud services](cloud_services/index.md) | Connect to cloud providers using OpenID Connect (OIDC) to retrieve temporary credentials to access services or secrets. | -| **Verify** | | -| [CI services](services/index.md) | Link Docker containers with your base image. | -| [GitLab CI/CD for external repositories](ci_cd_for_external_repos/index.md) | Get the benefits of GitLab CI/CD combined with repositories in GitHub and Bitbucket Cloud. | -| [Interactive Web Terminals](interactive_web_terminal/index.md) | Open an interactive web terminal to debug the running jobs. | -| [Review Apps](review_apps/index.md) | Configure GitLab CI/CD to preview code changes. | -| [Unit test reports](testing/unit_test_reports.md) | Identify test failures directly on merge requests. | -| [Using Docker images](docker/using_docker_images.md) | Use GitLab and GitLab Runner with Docker to build and test applications. | -| **Release** | | -| [Auto Deploy](../topics/autodevops/stages.md#auto-deploy) | Deploy your application to a production environment in a Kubernetes cluster. | -| [Building Docker images](docker/using_docker_build.md) | Maintain Docker-based projects using GitLab CI/CD. | -| [Canary Deployments](../user/project/canary_deployments.md) | Ship features to only a portion of your pods and let a percentage of your user base to visit the temporarily deployed feature. | -| [Deploy boards](../user/project/deploy_boards.md) | Check the current health and status of each CI/CD environment running on Kubernetes. | -| [Feature flags](../operations/feature_flags.md) | Deploy your features behind Feature flags. | -| [GitLab Pages](../user/project/pages/index.md) | Deploy static websites. | -| [GitLab Releases](../user/project/releases/index.md) | Add release notes to Git tags. | -| [Cloud deployment](cloud_deployment/index.md) | Deploy your application to a main cloud provider. | -| **Secure** | | -| [Code Quality](testing/code_quality.md) | Analyze your source code quality. | -| [Container Scanning](../user/application_security/container_scanning/index.md) | Scan your container images for known vulnerabilities. | -| [Coverage-guided fuzz testing](../user/application_security/coverage_fuzzing/index.md) | Test your application's behavior by providing randomized input. | -| [Dynamic Application Security Testing](../user/application_security/dast/index.md) | Test your application's runtime behavior for vulnerabilities. | -| [Dependency Scanning](../user/application_security/dependency_scanning/index.md) | Analyze your dependencies for known vulnerabilities. | -| [Infrastructure as Code scanning](../user/application_security/iac_scanning/index.md) | Scan your IaC configuration files for known vulnerabilities. | -| [License Scanning](../user/compliance/license_scanning_of_cyclonedx_files/index.md) | Search your project dependencies for their licenses. | Search your project dependencies for their licenses. | -| [Secret Detection](../user/application_security/secret_detection/index.md) | Search your application's source code for secrets. | -| [Static Application Security Testing](../user/application_security/sast/index.md) | Test your application's source code for known vulnerabilities. | -| [Web API fuzz testing](../user/application_security/api_fuzzing/index.md) | Test your application's API behavior by providing randomized input. | -| **Govern** | | -| [Compliance frameworks](../user/group/compliance_frameworks.md) | Enforce a GitLab CI/CD configuration on all projects in a group. | -| [Scan execution policies](../user/application_security/policies/scan-execution-policies.md) | Enforce security scans run on a specified schedule or with the project pipeline. | -| [Scan results policies](../user/application_security/policies/scan-result-policies.md) | Enforce action based on results of a pipeline security scan. | - -## Examples - -See the [CI/CD examples](examples/index.md) page for example project code and tutorials for -using GitLab CI/CD with various: - -- App frameworks -- Languages -- Platforms - -## Administration - -You can change the default behavior of GitLab CI/CD for: - -- An entire GitLab instance in the [CI/CD administration settings](../administration/cicd.md). -- Specific projects in the [pipelines settings](pipelines/settings.md). - -See also: - -- [Enable or disable GitLab CI/CD in a project](enable_or_disable_ci.md). +# Get started with GitLab CI/CD **(FREE ALL)** + +CI/CD is a continuous method of software development, where you continuously build, +test, deploy, and monitor iterative code changes. + +This iterative process helps reduce the chance that you develop new code based on +buggy or failed previous versions. GitLab CI/CD can catch bugs early in the development cycle, +and help ensure that all the code deployed to production complies with your established code standards. + +## Common terms + +If you're new to GitLab CI/CD, start by reviewing some of the commonly used terms. + +### The `.gitlab-ci.yml` file + +To use GitLab CI/CD, you start with a `.gitlab-ci.yml` file at the root of your project. +In this file, you specify the list of things you want to do, like test and deploy your application. +This file follows the YAML format and has its own special syntax. + +You can name this file anything you want, but `.gitlab-ci.yml` is the most common name. +Use the pipeline editor to edit the `.gitlab-ci.yml` file and test the syntax before you commit changes. + +**Get started:** + +- [Create your first `.gitlab-ci.yml` file](quick_start/index.md). +- [View all the possible keywords that you can use in the `.gitlab-ci.yml` file](yaml/index.md). + +### Runners + +Runners are the agents that run your jobs. These agents can run on physical machines or virtual instances. +In your `.gitlab-ci.yml` file, you can specify a container image you want to use when running the job. +The runner loads the image and runs the job either locally or in the container. + +If you use GitLab.com, free shared runners are already available for you. And you can register your own +runners on GitLab.com if you'd like. + +If you don't use GitLab.com, you can: + +- Register runners or use runners already registered for your self-managed instance. +- Create a runner on your local machine. + +**Get started:** + +- [Create a runner on your local machine](../tutorials/create_register_first_runner/index.md). +- [Learn more about runners](https://docs.gitlab.com/runner/). + +### Pipelines + +Pipelines are made up of jobs and stages: + +- **Jobs** define what you want to do. For example, test code changes, or deploy + to a staging environment. +- Jobs are grouped into **stages**. Each stage contains at least one job. + Typical stages might be `build`, `test`, and `deploy`. + +**Get started:** + +- [Learn more about pipelines](pipelines/index.md). + +### CI/CD variables + +CI/CD variables help you customize jobs by making values defined elsewhere accessible to jobs. +They can be hard-coded in your `.gitlab-ci.yml` file, project settings, or dynamically generated +[predefined variables](variables/predefined_variables.md). + +**Get started:** + +- [Learn more about CI/CD variables](variables/index.md). + +## Videos + +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [GitLab CI/CD demo](https://www.youtube-nocookie.com/embed/ljth1Q5oJoo). +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [GitLab CI/CD and the Web IDE](https://youtu.be/l5705U8s_nQ?t=369). +- Webcast: [Mastering continuous software development](https://about.gitlab.com/webcast/mastering-ci-cd/). ## Related topics -- [Why you might choose GitLab CI/CD](https://about.gitlab.com/blog/2016/10/17/gitlab-ci-oohlala/) -- [Reasons you might migrate from another platform](https://about.gitlab.com/blog/2016/07/22/building-our-web-app-on-gitlab-ci/) -- [Five teams that made the switch to GitLab CI/CD](https://about.gitlab.com/blog/2019/04/25/5-teams-that-made-the-switch-to-gitlab-ci-cd/) -- If you use VS Code to edit your GitLab CI/CD configuration, the - [GitLab Workflow VS Code extension](../user/project/repository/vscode.md) helps you +- [Five teams that made the switch to GitLab CI/CD](https://about.gitlab.com/blog/2019/04/25/5-teams-that-made-the-switch-to-gitlab-ci-cd/). +- [Make the case for CI/CD in your organization](https://about.gitlab.com/devops-tools/github-vs-gitlab/). +- Learn how [Verizon reduced rebuilds](https://about.gitlab.com/blog/2019/02/14/verizon-customer-story/) from 30 days to under 8 hours with GitLab. +- Use the [GitLab Workflow VS Code extension](../user/project/repository/vscode.md) to [validate your configuration](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow#validate-gitlab-ci-configuration) - and [view your pipeline status](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow#information-about-your-branch-pipelines-mr-closing-issue) - -See also the [Why CI/CD?](https://docs.google.com/presentation/d/1OGgk2Tcxbpl7DJaIOzCX4Vqg3dlwfELC3u2jEeCBbDk) presentation. - -### Major version changes (breaking) - -As GitLab CI/CD has evolved, certain breaking changes have -been necessary. - -For GitLab 15.0 and later, all breaking changes are documented on the following pages: - -- [Deprecations](../update/deprecations.md) -- [Removals](../update/removals.md) - -The breaking changes for [GitLab Runner](https://docs.gitlab.com/runner/) in earlier -major version releases are: - -- 14.0: No breaking changes. -- 13.0: - - [Remove Backported `os.Expand`](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4915). - - [Remove Fedora 29 package support](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/16158). - - [Remove macOS 32-bit support](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/25466). - - [Removed `debug/jobs/list?v=1` endpoint](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6361). - - [Remove support for array of strings when defining services for Docker executor](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4922). - - [Remove `--docker-services` flag on register command](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6404). - - [Remove legacy build directory caching](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4180). - - [Remove `FF_USE_LEGACY_VOLUMES_MOUNTING_ORDER` feature flag](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6581). - - [Remove support for Windows Server 1803](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6553). -- 12.0: - - [Use `refspec` to clone/fetch Git repository](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4069). - - [Old cache configuration](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4070). - - [Old metrics server configuration](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4072). - - [Remove `FF_K8S_USE_ENTRYPOINT_OVER_COMMAND`](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4073). - - [Remove Linux distributions that reach EOL](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1130). - - [Update command line API for helper images](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4013). - - [Remove old `git clean` flow](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4175). + and [view your pipeline status](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow#information-about-your-branch-pipelines-mr-closing-issue). diff --git a/doc/ci/introduction/img/gitlab_workflow_example_11_9.png b/doc/ci/introduction/img/gitlab_workflow_example_11_9.png Binary files differdeleted file mode 100644 index 1f11db55f81..00000000000 --- a/doc/ci/introduction/img/gitlab_workflow_example_11_9.png +++ /dev/null diff --git a/doc/ci/introduction/img/gitlab_workflow_example_extended_v12_3.png b/doc/ci/introduction/img/gitlab_workflow_example_extended_v12_3.png Binary files differdeleted file mode 100644 index 6e1066d4868..00000000000 --- a/doc/ci/introduction/img/gitlab_workflow_example_extended_v12_3.png +++ /dev/null diff --git a/doc/ci/introduction/index.md b/doc/ci/introduction/index.md index 35692ed7b7e..536dd224ac5 100644 --- a/doc/ci/introduction/index.md +++ b/doc/ci/introduction/index.md @@ -1,116 +1,11 @@ --- -stage: Verify -group: Pipeline Authoring -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -description: "An overview of Continuous Integration, Continuous Delivery, and Continuous Deployment, as well as an introduction to GitLab CI/CD." -type: concepts +redirect_to: 'index.md' +remove_date: '2023-11-24' --- -# CI/CD concepts **(FREE ALL)** +This document was moved to [another location](../index.md). -With the continuous method of software development, you continuously build, -test, and deploy iterative code changes. This iterative process helps reduce -the chance that you develop new code based on buggy or failed previous versions. -With this method, you strive to have less human intervention or even no intervention at all, -from the development of new code until its deployment. - -The three primary approaches for the continuous method are: - -- [Continuous Integration](#continuous-integration) -- [Continuous Delivery](#continuous-delivery) -- [Continuous Deployment](#continuous-deployment) - -Out-of-the-box management systems can decrease hours spent on maintaining toolchains by 10% or more. -Watch our ["Mastering continuous software development"](https://about.gitlab.com/webcast/mastering-ci-cd/) -webcast to learn about continuous methods and how built-in GitLab CI/CD can help you simplify and scale software development. - -- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i>Learn how to: [configure CI/CD](https://www.youtube.com/watch?v=opdLqwz6tcE). -- [Make the case for CI/CD in your organization](https://about.gitlab.com/devops-tools/github-vs-gitlab/). -- Learn how [Verizon reduced rebuilds](https://about.gitlab.com/blog/2019/02/14/verizon-customer-story/) from 30 days to under 8 hours with GitLab. - -## Continuous Integration - -Consider an application that has its code stored in a Git -repository in GitLab. Developers push code changes every day, -multiple times a day. For every push to the repository, you -can create a set of scripts to build and test your application -automatically. These scripts help decrease the chances that you introduce errors in your application. - -This practice is known as [Continuous Integration](https://en.wikipedia.org/wiki/Continuous_integration). -Each change submitted to an application, even to development branches, -is built and tested automatically and continuously. These tests ensure the -changes pass all tests, guidelines, and code compliance -standards you established for your application. - -[GitLab itself](https://gitlab.com/gitlab-org/gitlab) is an -example of a project that uses Continuous Integration as a software -development method. For every push to the project, a set -of checks run against the code. - -## Continuous Delivery - -[Continuous Delivery](https://continuousdelivery.com/) is a step -beyond Continuous Integration. Not only is your application -built and tested each time a code change is pushed to the codebase, -the application is also deployed continuously. However, with continuous -delivery, you trigger the deployments manually. - -Continuous Delivery checks the code automatically, but it requires -human intervention to manually and strategically trigger the deployment -of the changes. - -## Continuous Deployment - -Continuous Deployment is another step beyond Continuous Integration, similar to -Continuous Delivery. The difference is that instead of deploying your -application manually, you set it to be deployed automatically. -Human intervention is not required. - -## GitLab CI/CD - -[GitLab CI/CD](../quick_start/index.md) is the part of GitLab that you use -for all of the continuous methods (Continuous Integration, -Delivery, and Deployment). With GitLab CI/CD, you can test, build, -and publish your software with no third-party application or integration needed. - -<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For an overview, see [Introduction to GitLab CI/CD](https://www.youtube.com/watch?v=l5705U8s_nQ&t=397) from an April 2020 GitLab meetup. - -### GitLab CI/CD workflow - -GitLab CI/CD fits in a common development workflow. - -You can start by discussing a code implementation in an issue -and working locally on your proposed changes. Then you can push your -commits to a feature branch in a remote repository that's hosted in GitLab. -The push triggers the CI/CD pipeline for your project. Then, GitLab CI/CD: - -- Runs automated scripts (sequentially or in parallel) to: - - Build and test your application. - - Preview the changes in a Review App, the same as you - would see on your `localhost`. - -After the implementation works as expected: - -- Get your code reviewed and approved. -- Merge the feature branch into the default branch. - - GitLab CI/CD deploys your changes automatically to a production environment. - -If something goes wrong, you can roll back your changes. - - - -This workflow shows the major steps in the GitLab process. -You don't need any external tools to deliver your software and -you can visualize all the steps in the GitLab UI. - -### A deeper look into the CI/CD workflow - -If you look deeper into the workflow, you can see -the features available in GitLab at each stage of the DevOps -lifecycle. - - - -<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -[Get a deeper look at GitLab CI/CD](https://youtu.be/l5705U8s_nQ?t=369). +<!-- This redirect file can be deleted after <2023-11-24>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/ci/jobs/ci_job_token.md b/doc/ci/jobs/ci_job_token.md index dee078c21e0..f1aa834a038 100644 --- a/doc/ci/jobs/ci_job_token.md +++ b/doc/ci/jobs/ci_job_token.md @@ -121,7 +121,7 @@ Prerequisite: To disable the job token scope allowlist: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Token Access**. 1. Toggle **Limit access _to_ this project** to disabled. @@ -144,7 +144,7 @@ Prerequisite: To add a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Token Access**. 1. Verify **Limit access _to_ this project** is enabled. @@ -188,7 +188,7 @@ Prerequisite: To configure the job token scope: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Token Access**. 1. Toggle **Limit access _from_ this project** to enabled. @@ -255,7 +255,7 @@ While troubleshooting CI/CD job token authentication issues, be aware that: - A [GraphQL example mutation](../../api/graphql/getting_started.md#update-project-settings) is available to toggle the scope settings per project. - [This comment](https://gitlab.com/gitlab-org/gitlab/-/issues/351740#note_1335673157) - demonstrates how to use graphQL with Bash and cURL to: + demonstrates how to use GraphQL with Bash and cURL to: - Enable the inbound token access scope. - Give access to project B from project A, or add B to A's allowlist. - To remove project access. diff --git a/doc/ci/jobs/index.md b/doc/ci/jobs/index.md index 762fb717505..90a64ea7569 100644 --- a/doc/ci/jobs/index.md +++ b/doc/ci/jobs/index.md @@ -48,7 +48,7 @@ Selecting an individual job shows you its job log, and allows you to: To view the full list of jobs that ran in a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Build > Jobs**. You can filter the list by [job status](#the-order-of-jobs-in-a-pipeline). diff --git a/doc/ci/jobs/job_artifacts.md b/doc/ci/jobs/job_artifacts.md index b080a4fd1e9..b6269918ed9 100644 --- a/doc/ci/jobs/job_artifacts.md +++ b/doc/ci/jobs/job_artifacts.md @@ -299,7 +299,7 @@ You can also delete individual artifacts from the [**Artifacts** page](#bulk-del You can delete multiple artifacts at the same time: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Build > Artifacts**. 1. Select the checkboxes next to the artifacts you want to delete. You can select up to 50 artifacts. 1. Select **Delete selected**. @@ -341,7 +341,7 @@ Keeping the latest artifacts can use a large amount of storage space in projects with a lot of jobs or large artifacts. If the latest artifacts are not needed in a project, you can disable this behavior to save space: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Artifacts**. 1. Clear the **Keep artifacts from most recent successful jobs** checkbox. diff --git a/doc/ci/jobs/job_control.md b/doc/ci/jobs/job_control.md index 770e1d38297..1065ee93389 100644 --- a/doc/ci/jobs/job_control.md +++ b/doc/ci/jobs/job_control.md @@ -867,7 +867,7 @@ linux:rspec: parallel: matrix: - PROVIDER: aws - - STACK: app1 + STACK: app1 script: echo "Running rspec on linux..." mac:rspec: @@ -877,7 +877,7 @@ mac:rspec: parallel: matrix: - PROVIDER: [gcp, vultr] - - STACK: [data] + STACK: [data] script: echo "Running rspec on mac..." production: diff --git a/doc/ci/large_repositories/index.md b/doc/ci/large_repositories/index.md index da5ebe21341..cef0554bf6c 100644 --- a/doc/ci/large_repositories/index.md +++ b/doc/ci/large_repositories/index.md @@ -1,254 +1,11 @@ --- -stage: Verify -group: Runner -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -type: reference +redirect_to: '../../user/project/repository/managing_large_repositories.md' +remove_date: '2023-11-30' --- -# Optimize GitLab for large repositories **(FREE ALL)** +This document was moved to [another location](../../user/project/repository/managing_large_repositories.md). -Large repositories consisting of more than 50k files in a worktree -may require more optimizations beyond -[pipeline efficiency](../pipelines/pipeline_efficiency.md) -because of the time required to clone and check out. - -GitLab and GitLab Runner handle this scenario well -but require optimized configuration to efficiently perform its -set of operations. - -The general guidelines for handling big repositories are simple. -Each guideline is described in more detail in the sections below: - -- Always fetch incrementally. Do not clone in a way that results in recreating all of the worktree. -- Always use shallow clone to reduce data transfer. Be aware that this puts more burden - on GitLab instance due to higher CPU impact. -- Control the clone directory if you heavily use a fork-based workflow. -- Optimize `git clean` flags to ensure that you remove or keep data that might affect or speed-up your build. - -## Shallow cloning - -> Introduced in GitLab Runner 8.9. - -GitLab and GitLab Runner perform a [shallow clone](../pipelines/settings.md#limit-the-number-of-changes-fetched-during-clone) -by default. - -Ideally, you should always use `GIT_DEPTH` with a small number -like 10. This instructs GitLab Runner to perform shallow clones. -Shallow clones make Git request only the latest set of changes for a given branch, -up to desired number of commits as defined by the `GIT_DEPTH` variable. - -This significantly speeds up fetching of changes from Git repositories, -especially if the repository has a very long backlog consisting of number -of big files as we effectively reduce amount of data transfer. - -The following example makes the runner shallow clone to fetch only a given branch; -it does not fetch any other branches nor tags. - -```yaml -variables: - GIT_DEPTH: 10 - -test: - script: - - ls -al -``` - -## Git strategy - -> Introduced in GitLab Runner 8.9. - -By default, GitLab is configured to use the [`fetch` Git strategy](../runners/configure_runners.md#git-strategy), -which is recommended for large repositories. -This strategy reduces the amount of data to transfer and -does not really impact the operations that you might do on a repository from CI. - -## Git clone path - -> Introduced in GitLab Runner 11.10. - -[`GIT_CLONE_PATH`](../runners/configure_runners.md#custom-build-directories) allows you to -control where you clone your sources. This can have implications if you -heavily use big repositories with fork workflow. - -Fork workflow from GitLab Runner's perspective is stored as a separate repository -with separate worktree. That means that GitLab Runner cannot optimize the usage -of worktrees and you might have to instruct GitLab Runner to use that. - -In such cases, ideally you want to make the GitLab Runner executor be used only -for the given project and not shared across different projects to make this -process more efficient. - -The [`GIT_CLONE_PATH`](../runners/configure_runners.md#custom-build-directories) has to be -within the `$CI_BUILDS_DIR`. Currently, it is impossible to pick any path -from disk. - -## Git clean flags - -> Introduced in GitLab Runner 11.10. - -[`GIT_CLEAN_FLAGS`](../runners/configure_runners.md#git-clean-flags) allows you to control -whether or not you require the `git clean` command to be executed for each CI -job. By default, GitLab ensures that you have your worktree on the given SHA, -and that your repository is clean. - -[`GIT_CLEAN_FLAGS`](../runners/configure_runners.md#git-clean-flags) is disabled when set -to `none`. On very big repositories, this might be desired because `git -clean` is disk I/O intensive. Controlling that with `GIT_CLEAN_FLAGS: -ffdx --e .build/` (for example) allows you to control and disable removal of some -directories within the worktree between subsequent runs, which can speed-up -the incremental builds. This has the biggest effect if you re-use existing -machines and have an existing worktree that you can re-use for builds. - -For exact parameters accepted by -[`GIT_CLEAN_FLAGS`](../runners/configure_runners.md#git-clean-flags), see the documentation -for [`git clean`](https://git-scm.com/docs/git-clean). The available parameters -are dependent on Git version. - -## Git fetch extra flags - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4142) in GitLab Runner 13.1. - -[`GIT_FETCH_EXTRA_FLAGS`](../runners/configure_runners.md#git-fetch-extra-flags) allows you -to modify `git fetch` behavior by passing extra flags. - -For example, if your project contains a large number of tags that your CI jobs don't rely on, -you could add [`--no-tags`](https://git-scm.com/docs/git-fetch#Documentation/git-fetch.txt---no-tags) -to the extra flags to make your fetches faster and more compact. - -Also in the case where you repository does _not_ contain a lot of -tags, `--no-tags` can [make a big difference in some cases](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/746). -If your CI builds do not depend on Git tags it is worth trying. - -See the [`GIT_FETCH_EXTRA_FLAGS` documentation](../runners/configure_runners.md#git-fetch-extra-flags) -for more information. - -## Fork-based workflow - -> Introduced in GitLab Runner 11.10. - -Following the guidelines above, let's imagine that we want to: - -- Optimize for a big project (more than 50k files in directory). -- Use forks-based workflow for contributing. -- Reuse existing worktrees. Have preconfigured runners that are pre-cloned with repositories. -- Runner assigned only to project and all forks. - -Let's consider the following two examples, one using `shell` executor and -other using `docker` executor. - -### `shell` executor example - -Let's assume that you have the following [`config.toml`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html). - -```toml -concurrent = 4 - -[[runners]] - url = "GITLAB_URL" - token = "TOKEN" - executor = "shell" - builds_dir = "/builds" - cache_dir = "/cache" - - [runners.custom_build_dir] - enabled = true -``` - -This `config.toml`: - -- Uses the `shell` executor, -- Specifies a custom `/builds` directory where all clones are stored. -- Enables the ability to specify `GIT_CLONE_PATH`, -- Runs at most 4 jobs at once. - -### `docker` executor example - -Let's assume that you have the following [`config.toml`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html). - -```toml -concurrent = 4 - -[[runners]] - url = "GITLAB_URL" - token = "TOKEN" - executor = "docker" - builds_dir = "/builds" - cache_dir = "/cache" - - [runners.docker] - volumes = ["/builds:/builds", "/cache:/cache"] -``` - -This `config.toml`: - -- Uses the `docker` executor, -- Specifies a custom `/builds` directory on disk where all clones are stored. - We host mount the `/builds` directory to make it reusable between subsequent runs - and be allowed to override the cloning strategy. -- Doesn't enable the ability to specify `GIT_CLONE_PATH` as it is enabled by default. -- Runs at most 4 jobs at once. - -### Our `.gitlab-ci.yml` - -Once we have the executor configured, we need to fine tune our `.gitlab-ci.yml`. - -Our pipeline is most performant if we use the following `.gitlab-ci.yml`: - -```yaml -variables: - GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/$CI_PROJECT_NAME - -build: - script: ls -al -``` - -This YAML setting configures a custom clone path. This path makes it possible to re-use worktrees -between the parent project and forks because we use the same clone path for all forks. - -Why use `$CI_CONCURRENT_ID`? The main reason is to ensure that worktrees used are not conflicting -between projects. The `$CI_CONCURRENT_ID` represents a unique identifier within the given executor. -When we use it to construct the path, this directory does not conflict -with other concurrent jobs running. - -### Store custom clone options in `config.toml` - -Ideally, all job-related configuration should be stored in `.gitlab-ci.yml`. -However, sometimes it is desirable to make these schemes part of the runner's configuration. - -In the above example of Forks, making this configuration discoverable for users may be preferred, -but this brings administrative overhead as the `.gitlab-ci.yml` needs to be updated for each branch. -In such cases, it might be desirable to keep the `.gitlab-ci.yml` clone path agnostic, but make it -a configuration of the runner. - -We can extend our [`config.toml`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html) -with the following specification that is used by the runner if `.gitlab-ci.yml` does not override it: - -```toml -concurrent = 4 - -[[runners]] - url = "GITLAB_URL" - token = "TOKEN" - executor = "docker" - builds_dir = "/builds" - cache_dir = "/cache" - - environment = [ - "GIT_CLONE_PATH=$CI_BUILDS_DIR/$CI_CONCURRENT_ID/$CI_PROJECT_NAME" - ] - - [runners.docker] - volumes = ["/builds:/builds", "/cache:/cache"] -``` - -This makes the cloning configuration to be part of the given runner -and does not require us to update each `.gitlab-ci.yml`. - -## Git fetch caching step - -For very active repositories with a large number of references and files, consider using the -[Gitaly pack-objects cache](../../administration/gitaly/configure_gitaly.md#pack-objects-cache). -The pack-objects cache: - -- Benefits all repositories on your GitLab server. -- Automatically works for forks. +<!-- This redirect file can be deleted after <2023-11-30>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/ci/lint.md b/doc/ci/lint.md index 0ca0469f5e9..2b54c645807 100644 --- a/doc/ci/lint.md +++ b/doc/ci/lint.md @@ -24,7 +24,7 @@ configuration added with the [`includes` keyword](yaml/index.md#include). To check CI/CD configuration with the CI lint tool: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Build > Pipelines**. 1. In the upper-right corner, select **CI lint**. 1. Paste a copy of the CI/CD configuration you want to check into the text box. @@ -45,7 +45,7 @@ Prerequisites: To simulate a pipeline: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Build > Pipelines**. 1. In the upper-right corner, select **CI lint**. 1. Paste a copy of the CI/CD configuration you want to check into the text box. diff --git a/doc/ci/migration/examples/img/maven-freestyle-plugin.png b/doc/ci/migration/examples/img/maven-freestyle-plugin.png Binary files differnew file mode 100644 index 00000000000..bb3308a1595 --- /dev/null +++ b/doc/ci/migration/examples/img/maven-freestyle-plugin.png diff --git a/doc/ci/migration/examples/img/maven-freestyle-shell.png b/doc/ci/migration/examples/img/maven-freestyle-shell.png Binary files differnew file mode 100644 index 00000000000..73ecbb9401d --- /dev/null +++ b/doc/ci/migration/examples/img/maven-freestyle-shell.png diff --git a/doc/ci/migration/examples/jenkins-maven.md b/doc/ci/migration/examples/jenkins-maven.md new file mode 100644 index 00000000000..0ed08357eef --- /dev/null +++ b/doc/ci/migration/examples/jenkins-maven.md @@ -0,0 +1,232 @@ +--- +stage: Verify +group: Pipeline Authoring +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: index, howto +--- + +# Migrate a Maven build from Jenkins to GitLab CI/CD + +If you have a Maven build in Jenkins, you can use a [Java Spring](https://gitlab.com/gitlab-org/project-templates/spring) +project template to migrate to GitLab. The template uses Maven for its underlying dependency management. + +## Sample Jenkins configurations + +The following three Jenkins examples each use different methods to test, build, and install a +Maven project into a shell agent: + +- Freestyle with shell execution +- Freestyle with the Maven task plugin +- A declarative pipeline using a Jenkinsfile + +All three examples run the same three commands in order, in three different stages: + +- `mvn test`: Run any tests found in the codebase +- `mvn package -DskipTests`: Compile the code into an executable type defined in the POM + and skip running any tests because that was done in the first stage. +- `mvn install -DskipTests`: Install the compiled executable into the agent's local Maven + `.m2` repository and again skip running the tests. + +These examples use a single, persistent Jenkins agent, which requires Maven to be +pre-installed on the agent. This method of execution is similar to a GitLab Runner +using the [shell executor](https://docs.gitlab.com/runner/executors/shell.html). + +### Freestyle with shell execution + +If using Jenkins' built-in shell execution option to directly call `mvn` commands +from the shell on the agent, the configuration might look like: + + + +### Freestyle with Maven task plugin + +If using the Maven plugin in Jenkins to declare and execute any specific goals +in the [Maven build lifecycle](https://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html), +the configuration might look like: + + + +This plugin requires Maven to be installed on the Jenkins agent, and uses a script wrapper +for calling Maven commands. + +### Using a declarative pipeline + +If using a declarative pipeline, the configuration might look like: + +```groovy +pipeline { + agent any + tools { + maven 'maven-3.6.3' + jdk 'jdk11' + } + stages { + stage('Build') { + steps { + sh "mvn package -DskipTests" + } + } + stage('Test') { + steps { + sh "mvn test" + } + } + stage('Install') { + steps { + sh "mvn install -DskipTests" + } + } + } +} +``` + +This example uses shell execution commands instead of plugins. + +By default, a declarative pipeline configuration is stored either in the Jenkins +pipeline configuration or directly in the Git repository in a `Jenksinfile`. + +## Convert Jenkins configuration to GitLab CI/CD + +While the examples above are all slightly different, they can all be migrated to GitLab CI/CD +with the same pipeline configuration. + +Prerequisites: + +- A GitLab Runner with a Shell executor +- Maven 3.6.3 and Java 11 JDK installed on the shell runner + +This example mimics the behavior and syntax of building, testing, and installing on Jenkins. + +In a GitLab CI/CD pipeline, the commands run in "jobs", which are grouped into stages. +The migrated configuration in the `.gitlab-ci.yml` configuration file consists of +two global keywords (`stages` and `variables`) followed by 3 jobs: + +```yaml +stages: + - build + - test + - install + +variables: + MAVEN_OPTS: >- + -Dhttps.protocols=TLSv1.2 + -Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository + MAVEN_CLI_OPTS: >- + -DskipTests + +build-JAR: + stage: build + script: + - mvn $MAVEN_CLI_OPTS package + +test-code: + stage: test + script: + - mvn test + +install-JAR: + stage: install + script: + - mvn $MAVEN_CLI_OPTS install +``` + +In this example: + +- `stages` defines three stages that run in order. Like the Jenkins examples above, + the test job runs first, followed by the build job, and finally the install job. +- `variables` defines [CI/CD variables](../../variables/index.md) that can be used by all jobs: + - `MAVEN_OPTS` are Maven environment variables needed whenever Maven is executed: + - `-Dhttps.protocols=TLSv1.2` sets the TLS protocol to version 1.2 for any HTTP requests in the pipeline. + - `-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository` sets the location of the + local Maven repository to the GitLab project directory on the runner, so the job + can access and modify the repository. + - `MAVEN_CLI_OPTS` are specific arguments to be added to `mvn` commands: + - `-DskipTests` skips the `test` stage in the Maven build lifecycle. +- `test-code`, `build-JAR`, and `install-JAR` are the user-defined names for the jobs + to run in the pipeline: + - `stage` defines which stage the job runs in. A pipeline contains one or more stages + and a stage contains one or more jobs. This example has three stages, each with a single job. + - `script` defines the commands to run in that job, similar to `steps` in a `Jenkinsfile`. + Jobs can run multiple commands in sequence, which run in the image container, + but in this example the jobs run only one command each. + +### Run jobs in Docker containers + +Instead of using a persistent machine for handling this build process like the Jenkins samples, +this example uses an ephemeral Docker container to handle execution. Using a container +removes the need for maintaining a virtual machine and the Maven version installed on it. +It also increases flexibility for expanding and extending the functionality of the pipeline. + +Prerequisites: + +- A GitLab Runner with the Docker executor that can be used by the project. + If you are using GitLab.com, you can use the public shared runners. + +This migrated pipeline configuration consists of three global keywords (`stages`, `default`, and `variables`) +followed by 3 jobs. This configuration makes use of additional GitLab CI/CD features +for an improved pipeline compared to the [example above](#convert-jenkins-configuration-to-gitlab-cicd): + +```yaml +stages: + - build + - test + - install + +default: + image: maven:3.6.3-openjdk-11 + cache: + key: $CI_COMMIT_REF_SLUG + paths: + - .m2/ + +variables: + MAVEN_OPTS: >- + -Dhttps.protocols=TLSv1.2 + -Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository + MAVEN_CLI_OPTS: >- + -DskipTests + +build-JAR: + stage: build + script: + - mvn $MAVEN_CLI_OPTS package + +test-code: + stage: test + script: + - mvn test + +install-JAR: + stage: install + script: + - mvn $MAVEN_CLI_OPTS install +``` + +In this example: + +- `stages` defines three stages that run in order. Like the Jenkins examples above, + the test job runs first, followed by the build job, and finally the install job. +- `default` defines standard configuration to reuse in all jobs by default: + - `image` defines the Docker image container to use and execute commands in. In this example, + it's an official Maven Docker image with everything needed already installed. + - `cache` is used to cache and reuse dependencies: + - `key` is the unique identifier for the specific cache archive. In this example, + it's a shortened version of the Git commit ref, autogenerated as a [predefined CI/CD variable](../../variables/predefined_variables.md). + Any job that runs for the same commit ref reuses the same cache. + - `paths` are the directories or files to include in the cache. In this example, + we cache the `.m2/` directory to avoid re-installing dependencies between job runs. +- `variables` defines [CI/CD variables](../../variables/index.md) that can be used by all jobs: + - `MAVEN_OPTS` are Maven environment variables needed whenever Maven is executed: + - `-Dhttps.protocols=TLSv1.2` sets the TLS protocol to version 1.2 for any HTTP requests in the pipeline. + - `-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository` sets the location of the + local Maven repository to the GitLab project directory on the runner, so the job + can access and modify the repository. + - `MAVEN_CLI_OPTS` are specific arguments to be added to `mvn` commands: + - `-DskipTests` skips the `test` stage in the Maven build lifecycle. +- `test-code`, `build-JAR`, and `install-JAR` are the user-defined names for the jobs + to run in the pipeline: + - `stage` defines which stage the job runs in. A pipeline contains one or more stages + and a stage contains one or more jobs. This example has three stages, each with a single job. + - `script` defines the commands to run in that job, similar to `steps` in a `Jenkinsfile`. + Jobs can run multiple commands in sequence, which run in the image container, + but in this example the jobs run only one command each. diff --git a/doc/ci/migration/jenkins.md b/doc/ci/migration/jenkins.md index 54ea3812029..d02c2f9c54e 100644 --- a/doc/ci/migration/jenkins.md +++ b/doc/ci/migration/jenkins.md @@ -7,215 +7,175 @@ type: index, howto # Migrating from Jenkins **(FREE ALL)** -A lot of GitLab users have successfully migrated to GitLab CI/CD from Jenkins. -We've collected several resources here that you might find informative if you're just getting started. -Think of this page as a "GitLab CI/CD for Jenkins Users" guide. +If you're migrating from Jenkins to GitLab CI/CD, you should be able +to create CI/CD pipelines that do everything you need. + +You can start by watching the [Migrating from Jenkins to GitLab](https://www.youtube.com/watch?v=RlEVGOpYF5Y) +video for examples of: + +- Converting a Jenkins pipeline into a GitLab CI/CD pipeline. +- Using Auto DevOps to test your code automatically. + +## Get started The following list of recommended steps was created after observing organizations -that were able to quickly complete this migration: - -1. Start by reading the GitLab CI/CD [Quick Start Guide](../quick_start/index.md) and [important product differences](#important-product-differences). -1. Learn the importance of [managing the organizational transition](#manage-organizational-transition). -1. [Add runners](../runners/index.md) to your GitLab instance. -1. Educate and enable your developers to independently perform the following steps in their projects: - 1. Review the [Quick Start Guide](../quick_start/index.md) and [Pipeline Configuration Reference](../yaml/index.md). - 1. Use the [Jenkins Wrapper](#jenkinsfile-wrapper) to temporarily maintain fragile Jenkins jobs. - 1. Migrate the build and CI jobs and configure them to show results directly in your merge requests. They can use [Auto DevOps](../../topics/autodevops/index.md) as a starting point, and [customize](../../topics/autodevops/customize.md) or [decompose](../../topics/autodevops/customize.md#use-individual-components-of-auto-devops) the configuration as needed. - 1. Add [Review Apps](../review_apps/index.md). - 1. Migrate the deployment jobs using [cloud deployment templates](../cloud_deployment/index.md), adding [environments](../environments/index.md), and [deploy boards](../../user/project/deploy_boards.md). - 1. Work to unwrap any jobs still running with the use of the Jenkins wrapper. -1. Take stock of any common CI/CD job definitions then create and share [templates](#templates) for them. -1. Check the [pipeline efficiency documentation](../pipelines/pipeline_efficiency.md) - to learn how to make your GitLab CI/CD pipelines faster and more efficient. - -Watch the [Migrating from Jenkins to GitLab](https://www.youtube.com/watch?v=RlEVGOpYF5Y) video for examples of how to: - -- Convert a Jenkins pipeline into a GitLab CI/CD pipeline. -- Use Auto DevOps to test your code automatically. - -Otherwise, read on for important information that helps you get the ball rolling. Welcome -to GitLab! +that were able to quickly complete this migration. + +Before doing any migration work, you should [start with a migration plan](plan_a_migration.md). + +Engineers that need to migrate projects to GitLab CI/CD should: + +- Read about some [key GitLab CI/CD features](#key-gitlab-cicd-features). +- Follow tutorials to create: + - [Your first GitLab pipeline](../quick_start/index.md). + - [A more complex pipeline](../quick_start/tutorial.md) that builds, tests, + and deploys a static site. +- Review the [`.gitlab-ci.yml` keyword reference](../yaml/index.md). +- Ensure [runners](../runners/index.md) are available, either by using shared GitLab.com runners + or installing new runners. +- Migrate build and CI jobs and configure them to show results directly in merge requests. + You can use [Auto DevOps](../../topics/autodevops/index.md) as a starting point, + and [customize](../../topics/autodevops/customize.md) or [decompose](../../topics/autodevops/customize.md#use-individual-components-of-auto-devops) + the configuration as needed. +- Migrate deployment jobs by using [cloud deployment templates](../cloud_deployment/index.md), + [environments](../environments/index.md), and the [GitLab agent for Kubernetes](../../user/clusters/agent/index.md). +- Check if any CI/CD configuration can be reused across different projects, then create + and share [templates](#templates). +- Check the [pipeline efficiency documentation](../pipelines/pipeline_efficiency.md) + to learn how to make your GitLab CI/CD pipelines faster and more efficient. If you have questions that are not answered here, the [GitLab community forum](https://forum.gitlab.com/) can be a great resource. -## Manage organizational transition - -An important part of transitioning from Jenkins to GitLab is the cultural and organizational -changes that come with the move, and successfully managing them. A few -things we have found that help this are: - -- Setting and communicating a clear vision of what your migration goals are helps - your users understand why the effort is worth it. The value is clear when - the work is done, but people need to be aware while it's in progress too. -- Sponsorship and alignment from the relevant leadership team helps with the point above. -- Spending time educating your users on what's different and sharing this document - with them helps ensure you are successful. -- Finding ways to sequence or delay parts of the migration can help a lot, but you - don't want to leave things in a non-migrated (or partially-migrated) state for too - long. To gain all the benefits of GitLab, moving your existing Jenkins setup over - as-is, including any current problems, isn't enough. You need to take advantage - of the improvements that GitLab offers, and this requires (eventually) updating - your implementation as part of the transition. - -## JenkinsFile Wrapper - -We are building a [JenkinsFile Wrapper](https://gitlab.com/gitlab-org/jfr-container-builder/) which -you can use to run a complete Jenkins instance inside of a GitLab job, including plugins. This can help ease the process -of transition, by letting you delay the migration of less urgent pipelines for a period of time. - -If you are interested in helping GitLab test the wrapper, join our [public testing issue](https://gitlab.com/gitlab-org/gitlab/-/issues/215675) for instructions and to provide your feedback. - -NOTE: -If you have a paid GitLab subscription, the JenkinsFile Wrapper is not packaged with GitLab and falls outside of the scope of support. For more information, see the [Statement of Support](https://about.gitlab.com/support/statement-of-support/). - -## Important product differences - -Some high level differences between the products worth mentioning are: - -- With GitLab you don't need a root `pipeline` keyword to wrap everything. -- The way pipelines are triggered and [trigger other pipelines](../yaml/index.md#trigger) - is different than Jenkins. GitLab pipelines can be triggered: - - - on push - - on [schedule](../pipelines/schedules.md) - - from the [GitLab UI](../pipelines/index.md#run-a-pipeline-manually) - - by [API call](../triggers/index.md) - - by [webhook](../triggers/index.md#use-a-webhook) - - by [ChatOps](../chatops/index.md) - -- You can control which jobs run in which cases, depending on how they are triggered, - with the [`rules` syntax](../yaml/index.md#rules). -- GitLab [pipeline scheduling concepts](../pipelines/schedules.md) are also different from Jenkins. -- You can reuse pipeline configurations using the [`include` keyword](../yaml/index.md#include) - and [templates](#templates). Your templates can be kept in a central repository (with different - permissions), and then any project can use them. This central project could also - contain scripts or other reusable code. -- You can also use the [`extends` keyword](../yaml/index.md#extends) to reuse configuration - in a single pipeline configuration. -- All jobs in a single stage always run in parallel, and all stages run in sequence. - Certain jobs might break this sequencing as needed with our [directed acyclic graph](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/47063) - feature. +### Key GitLab CI/CD features + +GitLab CI/CD key features might be different or not exist in Jenkins. For example, +in GitLab: + +- Pipelines can be triggered with: + - A Git push + - A [Schedule](../pipelines/schedules.md) + - The [GitLab UI](../pipelines/index.md#run-a-pipeline-manually) + - An [API call](../triggers/index.md) + - A [webhook](../triggers/index.md#use-a-webhook) +- You can control which jobs run in which cases with the [`rules` syntax](../yaml/index.md#rules). +- You can reuse pipeline configurations: + - Use the [`extends` keyword](../yaml/index.md#extends) to reuse configuration + in a single pipeline configuration. + - Use the [`include` keyword](../yaml/index.md#include) to reuse configuration across + multiple pipelines and projects. +- Jobs are grouped into stages, and jobs in the same stage can run at the same time. + Stages run in sequence. Jobs can be configured to run outside of the stage ordering with the + [`needs` keyword](../yaml/index.md#needs). - The [`parallel`](../yaml/index.md#parallel) keyword can automatically parallelize tasks, - like tests that support parallelization. -- Usually all jobs in a single stage run in parallel, and all stages run in sequence. - Different [pipeline architectures](../pipelines/pipeline_architectures.md) allow you to change this behavior. -- The new [`rules` syntax](../yaml/index.md#rules) is the recommended method of - controlling when different jobs run. It is more powerful than the `only/except` syntax. -- One important difference is that jobs run independently of each other and have a - fresh environment in each job. Passing artifacts between jobs is controlled using the - [`artifacts`](../yaml/index.md#artifacts) and [`dependencies`](../yaml/index.md#dependencies) - keywords. When finished, use the planned [Workspaces](https://gitlab.com/gitlab-org/gitlab/-/issues/29265) - feature to persist a common workspace between serial jobs. -- The `.gitlab-ci.yml` file is checked in to the root of your repository, much like a Jenkinsfile, but - is in the YAML format (see [complete reference](../yaml/index.md)) instead of a Groovy DSL. It's most - analogous to the declarative Jenkinsfile format. -- Manual approvals or gates can be set up as [`when:manual` jobs](../jobs/job_control.md#create-a-job-that-must-be-run-manually). These can - also leverage [`protected environments`](../jobs/job_control.md#run-a-job-after-a-delay) - to control who is able to approve them. -- GitLab comes with a [container registry](../../user/packages/container_registry/index.md), so you can use - container images to set up your build environment. For example, set up one pipeline that builds your build environment - itself and publish that to the container registry. Then, have your pipelines use this instead of each building their - own environment, which is slower and may be less consistent. We have extensive documentation on [how to use the Container Registry](../../user/packages/container_registry/index.md). -- A central utilities repository can be a great place to put assorted scheduled jobs - or other manual jobs that function like utilities. Jenkins installations tend to - have a few of these. - -## Agents vs. runners - -Both Jenkins agents and GitLab runners are the hosts that run jobs. To convert the -Jenkins agent, uninstall it and then [install and register the runner](../runners/index.md). -Runners do not require much overhead, so you can size them similarly to the Jenkins -agents you were using. - -Some important differences in the way runners work in comparison to agents are: - -- Runners can be set up as [shared across an instance, be added at the group level, or set up at the project level](../runners/runners_scope.md). - They self-select jobs from the scopes you've defined automatically. -- You can also [use tags](../runners/configure_runners.md#use-tags-to-control-which-jobs-a-runner-can-run) for finer control, and - associate runners with specific jobs. For example, you can use a tag for jobs that + especially tests that support parallelization. +- Jobs run independently of each other and have a fresh environment for each job. + Passing artifacts between jobs is controlled using the [`artifacts`](../yaml/index.md#artifacts) + and [`dependencies`](../yaml/index.md#dependencies) keywords. +- The `.gitlab-ci.yml` configuration file exists in your Git repository, like a `Jenkinsfile`, + but is [a YAML file](#yaml-configuration-file), not Groovy. +- GitLab comes with a [container registry](../../user/packages/container_registry/index.md). + You can build and store custom container images to run your jobs in. + +## Runners + +Like Jenkins agents, GitLab runners are the hosts that run jobs. If you are using GitLab.com, +you can use the [shared runner fleet](../runners/index.md) to run jobs without provisioning +your own runners. + +To convert a Jenkins agent for use with GitLab CI/CD, uninstall the agent and then +[install and register a runner](../runners/index.md). Runners do not require much overhead, +so you might be able to use similar provisioning as the Jenkins agents you were using. + +Some key details about runners: + +- Runners can be [configured](../runners/runners_scope.md) to be shared across an instance, + a group, or dedicated to a single project. +- You can use the [`tags` keyword](../runners/configure_runners.md#use-tags-to-control-which-jobs-a-runner-can-run) + for finer control, and associate runners with specific jobs. For example, you can use a tag for jobs that require dedicated, more powerful, or specific hardware. - GitLab has [autoscaling for runners](https://docs.gitlab.com/runner/configuration/autoscale.html). Use autoscaling to provision runners only when needed and scale down when not needed, similar to ephemeral agents in Jenkins. -If you are using `gitlab.com`, you can take advantage of our [shared runner fleet](../runners/index.md) -to run jobs without provisioning your own runners. We are investigating making them -[available for self-managed instances](https://gitlab.com/groups/gitlab-org/-/epics/835) -as well. +## YAML configuration file -## Groovy vs. YAML +GitLab pipeline configuration files use the [YAML](https://yaml.org/) format instead of +the [Groovy](https://groovy-lang.org/) format that Jenkins uses. -Jenkins Pipelines are based on [Groovy](https://groovy-lang.org/), so the pipeline specification is written as code. -GitLab works a bit differently, using the more highly structured [YAML](https://yaml.org/) format. -The scripting elements are in `script` blocks separate from the pipeline specification itself. - -Using YAML is a strength of GitLab, in that it helps keep the learning curve much simpler to get up and running. -It also avoids some of the problem of unconstrained complexity which can make your Jenkinsfile hard to understand -and manage. - -We do of course still value DRY (don't repeat yourself) principles. We want to ensure that -behaviors of your jobs can be codified once and applied as needed. You can use the `extends` syntax to -[reuse configuration in your jobs](../yaml/index.md#extends), and `include` can -be used to [reuse pipeline configurations](../yaml/index.md#include) in pipelines -in different projects: +Using YAML is a strength of GitLab CI/CD, as it is a simple format to understand +and start using. For example, a small configuration file with two jobs and some +shared configuration in a hidden job: ```yaml -.in-docker: +.test-config: tags: - - docker - image: alpine + - docker-runners + stage: test -rspec: +test-job: extends: - - .in-docker + - .docker-config script: - - rake rspec + - bundle exec rake rspec + +lint-job: + extends: + - .docker-config + script: + - yarn run prettier ``` -## Artifact publishing +In this example: + +- The commands to run in jobs are added with the [`script` keyword](../yaml/index.md#script). +- The [`extends` keyword](../yaml/index.md#extends) reduces duplication in the configuration + by adding the same `tags` and `stage` configuration defined in `.test-config` to both jobs. + +### Artifacts + +In GitLab, any job can use the [`artifacts` keyword](../yaml/index.md#artifacts) +to define a set of [artifacts](../jobs/job_artifacts.md) to be stored when a job completes. +Artifacts are files that can be used in later jobs, for example for testing or deployment. -Artifacts may work a bit differently than you've used them with Jenkins. In GitLab, any job can define -a set of artifacts to be saved by using the `artifacts` keyword. This can be configured to point to a file -or set of files that can then be persisted from job to job. Read more on our detailed -[artifacts documentation](../jobs/job_artifacts.md): +For example: ```yaml pdf: script: xelatex mycv.tex artifacts: paths: - - ./mycv.pdf - - ./output/ + - mycv.pdf + - output/ expire_in: 1 week ``` -Additionally, we have package management features like built-in container and package registries that you -can leverage. You can see the complete list of packaging features in the -[Packages and registries](../../user/packages/index.md) documentation. +In this example: -## Integrated features +- The `mycv.pdf` file and all the files in `output/` are stored and could be used + in later jobs. +- To save resources, the artifacts expire and are deleted after one week. -You may have used plugins to get things like code quality, unit tests, and security scanning working in Jenkins. -GitLab takes advantage of our connected ecosystem to automatically pull these kinds of results into -your merge requests, pipeline details pages, and other locations. You may find that you actually don't -need to configure anything to have these appear. +### Scanning features -Our [CI/CD feature index](../index.md#features) has the full list of bundled features and links to the documentation for each. -Refer to this index if these features aren't working as expected, or if you'd like to see what's available. +You might have used plugins for things like code quality, security, or static application scanning +in Jenkins. Tools like these are already available in GitLab and can be used in your +pipeline. + +GitLab features including [code quality](../testing/code_quality.md), [security scanning](../../user/application_security/index.md), +[SAST](../../user/application_security/sast/index.md), and many others generate reports +when they complete. These reports can be displayed in merge requests and pipeline details pages. ### Templates -For advanced CI/CD teams, project templates can enable the reuse of pipeline configurations, -as well as encourage inner sourcing. +For organizations with many CI/CD pipelines, you can use project templates to configure +custom CI/CD configuration templates and reuse them across projects. + +Group maintainers can configure a group to use as the source for [custom project templates](../../administration/custom_project_templates.md). +These templates can be used by all projects in the group. -In self-managed GitLab instances, you can build an [Instance Template Repository](../../administration/settings/instance_template_repository.md). -Development teams across the whole organization can select templates from a dropdown list. -A group maintainer or a group owner is able to set a group to use as the source for the -[custom project templates](../../administration/custom_project_templates.md). This can -be used by all projects in the group. An instance administrator can set a group as -the source for [instance project templates](../../user/group/custom_project_templates.md), -which can be used by projects in that instance. +An instance administrator can set a group as the source for [instance project templates](../../user/group/custom_project_templates.md), +which can be used by all projects in that instance. ## Convert a declarative Jenkinsfile @@ -361,7 +321,48 @@ my_job: - if: $CI_COMMIT_BRANCH ``` +## Secrets Management + +Privileged information, often referred to as "secrets", is sensitive information +or credentials you need in your CI/CD workflow. You might use secrets to unlock protected resources +or sensitive information in tools, applications, containers, and cloud-native environments. + +Secrets management in Jenkins is usually handled with the `Secret` type field or the +Credentials Plugin. Credentials stored in the Jenkins settings can be exposed to +jobs as environment variables by using the Credentials Binding plugin. + +For secrets management in GitLab, you can use one of the supported integrations +for an external service. These services securely store secrets outside of your GitLab project, +though you must have a subscription for the service: + +- [HashiCorp Vault](../secrets/id_token_authentication.md#automatic-id-token-authentication-with-hashicorp-vault) +- [Azure Key Vault](../secrets/azure_key_vault.md). + +GitLab also supports [OIDC authentication](../secrets/id_token_authentication.md) +for other third party services that support OIDC. + +Additionally, you can make credentials available to jobs by storing them in CI/CD variables, though secrets +stored in plain text are susceptible to accidental exposure, [the same as in Jenkins](https://www.jenkins.io/doc/developer/security/secrets/#storing-secrets). +You should always store sensitive information in [masked](../variables/index.md#mask-a-cicd-variable) +and [protected](../variables/index.md#protect-a-cicd-variable) variables, which mitigates +some of the risk. + +Also, never store secrets as variables in your `.gitlab-ci.yml` file, which is public to all +users with access to the project. Storing sensitive information in variables should +only be done in [the project, group, or instance settings](../variables/index.md#define-a-cicd-variable-in-the-ui). + +Review the [security guidelines](../variables/index.md#cicd-variable-security) to improve +the safety of your CI/CD variables. + ## Additional resources -For help making your pipelines faster and more efficient, see the -[pipeline efficiency documentation](../pipelines/pipeline_efficiency.md). +- You can use the [JenkinsFile Wrapper](https://gitlab.com/gitlab-org/jfr-container-builder/) + to run a complete Jenkins instance inside of a GitLab CI/CD job, including plugins. Use this tool to + help ease the transition to GitLab CI/CD, by delaying the migration of less urgent pipelines. + + NOTE: + The JenkinsFile Wrapper is not packaged with GitLab and falls outside of the scope of support. + For more information, see the [Statement of Support](https://about.gitlab.com/support/statement-of-support/). +- If your tooling outputs packages that you want to make accessible, you can store them + in a [package registry](../../user/packages/index.md). +- Use [review Apps](../review_apps/index.md) to preview changes before merging them. diff --git a/doc/ci/migration/plan_a_migration.md b/doc/ci/migration/plan_a_migration.md new file mode 100644 index 00000000000..488b2abf3a2 --- /dev/null +++ b/doc/ci/migration/plan_a_migration.md @@ -0,0 +1,71 @@ +--- +stage: Verify +group: Pipeline Authoring +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: index, howto +--- + +# Plan a migration from another tool to GitLab CI/CD + +Before starting a migration from another tool to GitLab CI/CD, you should begin by +developing a migration plan. + +Review the advice on [managing organizational changes](#manage-organizational-changes) +first for advice on initial steps for larger migrations. + +Users involved in the migration itself should review the [questions to ask before starting a migration](#technical-questions-to-ask-before-starting-a-migration), +as an important technical step for setting expectations. CI/CD tools differ in approach, +structure, and technical specifics. While some concepts map one-to-one, others require +interactive conversion. + +It's important to focus on your desired end state instead of strictly translating +the behavior of your old tool. + +## Manage organizational changes + +An important part of transitioning to GitLab CI/CD is the cultural and organizational +changes that come with the move, and successfully managing them. + +A few things that organizations have reported as helping: + +- Set and communicate a clear vision of what your migration goals are, which helps + your users understand why the effort is worth it. The value is clear when + the work is done, but people need to be aware while it's in progress too. +- Sponsorship and alignment from the relevant leadership teams helps with the point above. +- Spend time educating your users on what's different, and share this guide + with them. +- Finding ways to sequence or delay parts of the migration can help a lot. Importantly though, + try not to leave things in a non-migrated (or partially-migrated) state for too + long. +- To gain all the benefits of GitLab, moving your existing configuration over as-is, + including any current problems, isn't enough. Take advantage of the improvements + that GitLab CI/CD offers, and update your implementation as part of the transition. + +## Technical questions to ask before starting a migration + +Asking some initial technical questions about your CI/CD needs helps quickly define +the migration requirements: + +- How many projects use this pipeline? +- What branching strategy is used? Feature branches? Mainline? Release branches? +- What tools do you use to build your code? For example, Maven, Gradle, or NPM? +- What tools do you use to test your code? For example JUnit, Pytest, or Jest? +- Do you use any security scanners? +- Where do you store any built packages? +- How do you deploy your code? +- Where do you deploy your code? + +### Jenkins + +If you are [migrating from Jenkins](jenkins.md), these additional questions can help with planning +the migration: + +- What plugins are used by jobs in Jenkins today? + - Do you know what these plugins do exactly? + - Do any plugin wrap a common build tool? For example, Maven, Gradle, or NPM? +- What is installed on the Jenkins agents? +- Are there any shared libraries in use? +- How are you authenticating from Jenkins? Are you using SSH keys, API tokens, or other secrets? +- Are there other projects that you need to access from your pipeline? +- Are there credentials in Jenkins to access outside services? For example Ansible Tower, + Artifactory, or other Cloud Providers or deployment targets? diff --git a/doc/ci/mobile_devops.md b/doc/ci/mobile_devops.md index 6969b3d4ef4..4e1a0f4683c 100644 --- a/doc/ci/mobile_devops.md +++ b/doc/ci/mobile_devops.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Mobile DevOps (Experiment) +# Mobile DevOps **(EXPERIMENT)** Use GitLab Mobile DevOps to quickly build, sign, and release native and cross-platform mobile apps for Android and iOS using GitLab CI/CD. Mobile DevOps is an experimental feature developed by @@ -189,8 +189,7 @@ fastlane init ``` This command creates a `fastlane` folder in the project with an `Appfile` and a stubbed-out `fastfile`. -This process asks you for login credentials to App Store Connect -to generate an app identifier and App Store app if they don't already exist. +During this process, you are prompted for App Store Connect login credentials to generate an app identifier and an App Store app if they don't already exist. The next step sets up fastlane match to manage code signing files for the project. Run the following command to generate a `Matchfile` with the configuration: @@ -295,7 +294,7 @@ Use the [Google Play integration](../user/project/integrations/google_play.md), to configure your CI/CD pipelines to connect to the [Google Play Console](https://play.google.com/console) to build and release Android apps. To enable the integration: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **Google Play**. 1. Under **Enable integration**, select the **Active** checkbox. @@ -352,7 +351,7 @@ Use the [Apple App Store integration](../user/project/integrations/apple_app_sto to configure your CI/CD pipelines to connect to [App Store Connect](https://appstoreconnect.apple.com/) to build and release apps for iOS, iPadOS, macOS, tvOS, and watchOS. To enable the integration: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **Apple App Store**. 1. Under **Enable integration**, select the **Active** checkbox. diff --git a/doc/ci/pipeline_editor/index.md b/doc/ci/pipeline_editor/index.md index 5291a8abb49..1dc6bc05e71 100644 --- a/doc/ci/pipeline_editor/index.md +++ b/doc/ci/pipeline_editor/index.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Pipeline Editor **(FREE ALL)** +# Pipeline editor **(FREE ALL)** > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4540) in GitLab 13.8. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/270059) in GitLab 13.10. diff --git a/doc/ci/pipelines/cicd_minutes.md b/doc/ci/pipelines/cicd_minutes.md index 4e822cf3edd..fa4a5e691a7 100644 --- a/doc/ci/pipelines/cicd_minutes.md +++ b/doc/ci/pipelines/cicd_minutes.md @@ -54,7 +54,7 @@ Prerequisite: To change the default quota that applies to all namespaces: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > CI/CD**. 1. Expand **Continuous Integration and Deployment**. @@ -76,7 +76,7 @@ Prerequisite: To set a compute quota for a namespace: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Overview > Groups**. 1. For the group you want to update, select **Edit**. @@ -107,7 +107,7 @@ Prerequisite: To view compute usage for your group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to +1. On the left sidebar, select **Search or go to** and find your group. The group must not be a subgroup. 1. Select **Settings > Usage Quotas**. 1. Select the **Pipelines** tab. @@ -167,7 +167,7 @@ You can purchase additional compute minutes for your group. You cannot transfer purchased compute minutes from one group to another, so be sure to select the correct group. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > Usage Quotas**. 1. Select **Pipelines**. 1. Select **Buy additional compute minutes**. diff --git a/doc/ci/pipelines/downstream_pipelines.md b/doc/ci/pipelines/downstream_pipelines.md index fca6e8407ef..6dc58882f37 100644 --- a/doc/ci/pipelines/downstream_pipelines.md +++ b/doc/ci/pipelines/downstream_pipelines.md @@ -439,6 +439,7 @@ upstream pipeline: Use [`needs:project`](../yaml/index.md#needsproject) to fetch artifacts from an upstream pipeline: +1. In GitLab 15.9 and later, [add the downstream project to the job token scope allowlist](../jobs/ci_job_token.md#add-a-project-to-the-job-token-scope-allowlist) of the upstream project. 1. In the upstream pipeline, save the artifacts in a job with the [`artifacts`](../yaml/index.md#artifacts) keyword, then trigger the downstream pipeline with a trigger job: @@ -491,6 +492,7 @@ because the downstream pipeline attempts to fetch artifacts from the latest bran To fetch the artifacts from the upstream `merge request` pipeline instead of the `branch` pipeline, pass `CI_MERGE_REQUEST_REF_PATH` to the downstream pipeline using [variable inheritance](#pass-yaml-defined-cicd-variables): +1. In GitLab 15.9 and later, [add the downstream project to the job token scope allowlist](../jobs/ci_job_token.md#add-a-project-to-the-job-token-scope-allowlist) of the upstream project. 1. In a job in the upstream pipeline, save the artifacts using the [`artifacts`](../yaml/index.md#artifacts) keyword. 1. In the job that triggers the downstream pipeline, pass the `$CI_MERGE_REQUEST_REF_PATH` variable: @@ -718,6 +720,80 @@ Use the [`trigger:forward` keyword](../yaml/index.md#triggerforward) to specify what type of variables to forward to the downstream pipeline. Forwarded variables are considered trigger variables, which have the [highest precedence](../variables/index.md#cicd-variable-precedence). +## Downstream pipelines for deployments + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/369061) in GitLab 16.4. + +You can use the [`environment`](../yaml/index.md#environment) keyword with [`trigger`](../yaml/index.md#trigger). +You might want to use `environment` from a trigger job if your deployment and application projects are separately managed. + +```yaml +deploy: + trigger: + project: project-group/my-downstream-project + environment: production +``` + +A downstream pipeline can provision infrastructure, deploy to a designated environment, and return the deployment status +to the upstream project. + +You can [view the environment and deployment](../environments/index.md#view-environments-and-deployments) +from the upstream project. + +### Advanced example + +This example configuration has the following behaviors: + +- The upstream project dynamically composes an environment name based on a branch name. +- The upstream project passes the context of the deployment to the downstream project with `UPSTREAM_*` variables. + +The `.gitlab-ci.yml` in an upstream project: + +```yaml +stages: + - deploy + - cleanup + +.downstream-deployment-pipeline: + variables: + UPSTREAM_PROJECT_ID: $CI_PROJECT_ID + UPSTREAM_ENVIRONMENT_NAME: $CI_ENVIRONMENT_NAME + UPSTREAM_ENVIRONMENT_ACTION: $CI_ENVIRONMENT_ACTION + trigger: + project: project-group/deployment-project + branch: main + strategy: depend + +deploy-review: + stage: deploy + extends: .downstream-deployment-pipeline + environment: + name: review/$CI_COMMIT_REF_SLUG + on_stop: stop-review + +stop-review: + stage: cleanup + extends: .downstream-deployment-pipeline + environment: + name: review/$CI_COMMIT_REF_SLUG + action: stop + when: manual +``` + +The `.gitlab-ci.yml` in a downstream project: + +```yaml +deploy: + script: echo "Deploy to ${UPSTREAM_ENVIRONMENT_NAME} for ${UPSTREAM_PROJECT_ID}" + rules: + - if: $CI_PIPELINE_SOURCE == "pipeline" && $UPSTREAM_ENVIRONMENT_ACTION == "start" + +stop: + script: echo "Stop ${UPSTREAM_ENVIRONMENT_NAME} for ${UPSTREAM_PROJECT_ID}" + rules: + - if: $CI_PIPELINE_SOURCE == "pipeline" && $UPSTREAM_ENVIRONMENT_ACTION == "stop" +``` + ## Troubleshooting ### Trigger job fails and does not create multi-project pipeline @@ -747,3 +823,9 @@ You cannot trigger a multi-project pipeline with a tag when a branch exists with name. The downstream pipeline fails to create with the error: `downstream pipeline can not be created, Ref is ambiguous`. Only trigger multi-project pipelines with tag names that do not match branch names. + +### `403 Forbidden` error when downloading a job artifact from an upstream pipeline + +In GitLab 15.9 and later, CI/CD job tokens are scoped to the project that the pipeline executes under. Therefore, the job token in a downstream pipeline cannot be used to access an upstream project by default. + +To resolve this, [add the downstream project to the job token scope allowlist](../jobs/ci_job_token.md#add-a-project-to-the-job-token-scope-allowlist). diff --git a/doc/ci/pipelines/index.md b/doc/ci/pipelines/index.md index 7cde8b50524..f0f0f6d29d2 100644 --- a/doc/ci/pipelines/index.md +++ b/doc/ci/pipelines/index.md @@ -139,7 +139,7 @@ operation of the pipeline. To execute a pipeline manually: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Build > Pipelines**. 1. Select **Run pipeline**. 1. In the **Run for branch name or tag** field, select the branch or tag to run the pipeline for. @@ -310,9 +310,9 @@ related objects, such as builds, logs, artifacts, and triggers. A strict security model is enforced when pipelines are executed on [protected branches](../../user/project/protected_branches.md). -The following actions are allowed on protected branches only if the user is +The following actions are allowed on protected branches if the user is [allowed to merge or push](../../user/project/protected_branches.md) -on that specific branch: +to that specific branch: - Run manual pipelines (using the [Web UI](#run-a-pipeline-manually) or [pipelines API](#pipelines-api)). - Run scheduled pipelines. @@ -321,15 +321,13 @@ on that specific branch: - Trigger manual actions on existing pipelines. - Retry or cancel existing jobs (using the Web UI or pipelines API). -**Variables** marked as **protected** are accessible only to jobs that -run on protected branches, preventing untrusted users getting unintended access to -sensitive information like deployment credentials and tokens. +**Variables** marked as **protected** are accessible to jobs that run in pipelines for protected branches. Only assign users the right to merge to protected branches if they have permission to access sensitive information like deployment credentials and tokens. **Runners** marked as **protected** can run jobs only on protected branches, preventing untrusted code from executing on the protected runner and preserving deployment keys and other credentials from being unintentionally accessed. To ensure that jobs intended to be executed on protected -runners do not use regular runners, they must be tagged accordingly. +runners do not use regular runners, they must be [tagged](../yaml/index.md#tags) accordingly. Review the [deployment safety](../environments/deployment_safety.md) page for additional security recommendations for securing your pipelines. @@ -349,7 +347,7 @@ Prerequisites: To trigger the pipeline when the upstream project is rebuilt: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Pipeline subscriptions**. 1. Select **Add project**. diff --git a/doc/ci/pipelines/merge_request_pipelines.md b/doc/ci/pipelines/merge_request_pipelines.md index 356b97aacc0..7b8a2a16734 100644 --- a/doc/ci/pipelines/merge_request_pipelines.md +++ b/doc/ci/pipelines/merge_request_pipelines.md @@ -169,8 +169,13 @@ To use the UI to run a pipeline in the parent project for a merge request from a 1. Select **Run pipeline**. You must read and accept the warning, or the pipeline does not run. You can disable this feature by using [the projects API](../../api/projects.md#edit-project) -to disable the `ci_allow_fork_pipelines_to_run_in_parent_project` setting. -The setting is `enabled` by default. +to disable the `ci_allow_fork_pipelines_to_run_in_parent_project` setting (enabled by default). +When you disable this setting, new pipelines from forks in the parent project are prevented. + +WARNING: +Older pipelines created before the setting was disabled are not affected and continue to run. +If you rerun a job in an older pipeline, the job uses the same context as when the +pipeline was originally created. ## Available predefined variables diff --git a/doc/ci/pipelines/merge_trains.md b/doc/ci/pipelines/merge_trains.md index c2bf9743e4f..d7f03490c68 100644 --- a/doc/ci/pipelines/merge_trains.md +++ b/doc/ci/pipelines/merge_trains.md @@ -101,7 +101,7 @@ Prerequisites: To enable merge trains: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Merge requests**. 1. In the **Merge method** section, verify that **Merge commit** is selected. 1. In the **Merge options** section: diff --git a/doc/ci/pipelines/merged_results_pipelines.md b/doc/ci/pipelines/merged_results_pipelines.md index 51678e64b10..1a21a2a9a00 100644 --- a/doc/ci/pipelines/merged_results_pipelines.md +++ b/doc/ci/pipelines/merged_results_pipelines.md @@ -41,7 +41,7 @@ To use merged results pipelines: To enable merged results pipelines in a project, you must have at least the Maintainer role: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Merge requests**. 1. In the **Merge options** section, select **Enable merged results pipelines**. 1. Select **Save changes**. diff --git a/doc/ci/pipelines/pipeline_architectures.md b/doc/ci/pipelines/pipeline_architectures.md index d0324f16ffb..305e48ca9f5 100644 --- a/doc/ci/pipelines/pipeline_architectures.md +++ b/doc/ci/pipelines/pipeline_architectures.md @@ -306,5 +306,4 @@ deploy_b: environment: production ``` -It's also possible to set jobs to run before or after triggering child pipelines, -for example if you have common setup steps or a unified deployment at the end. +Jobs can be set to run before or after triggering child pipelines in GitLab, allowing common setup steps or unified deployment. diff --git a/doc/ci/pipelines/pipeline_artifacts.md b/doc/ci/pipelines/pipeline_artifacts.md deleted file mode 100644 index 79435c0276d..00000000000 --- a/doc/ci/pipelines/pipeline_artifacts.md +++ /dev/null @@ -1,14 +0,0 @@ ---- -stage: Verify -group: Pipeline Security -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -redirect_to: '../testing/test_coverage_visualization.md' -remove_date: '2023-08-31' ---- - -This document was moved to [another location](../testing/test_coverage_visualization.md). - -<!-- This redirect file can be deleted after <2023-08-31>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/ci/pipelines/pipeline_efficiency.md b/doc/ci/pipelines/pipeline_efficiency.md index 1d7d6c3289a..3d24be0f8e1 100644 --- a/doc/ci/pipelines/pipeline_efficiency.md +++ b/doc/ci/pipelines/pipeline_efficiency.md @@ -28,7 +28,7 @@ The easiest indicators to check for inefficient pipelines are the runtimes of th stages, and the total runtime of the pipeline itself. The total pipeline duration is heavily influenced by the: -- [Size of the repository](../large_repositories/index.md) +- [Size of the repository](../../user/project/repository/managing_large_repositories.md) - Total number of stages and jobs. - Dependencies between jobs. - The ["critical path"](#directed-acyclic-graphs-dag-visualization), which represents @@ -227,7 +227,7 @@ Methods to reduce Docker image size: - Use a small base image, for example `debian-slim`. - Do not install convenience tools such as vim or curl if they aren't strictly needed. - Create a dedicated development image. -- Disable man pages and docs installed by packages to save space. +- Disable man pages and documentation installed by packages to save space. - Reduce the `RUN` layers and combine software installation steps. - Use [multi-stage builds](https://blog.alexellis.io/mutli-stage-docker-builds/) to merge multiple Dockerfiles that use the builder pattern into one Dockerfile, which can reduce image size. diff --git a/doc/ci/pipelines/schedules.md b/doc/ci/pipelines/schedules.md index 75a7d373203..1003c03bebf 100644 --- a/doc/ci/pipelines/schedules.md +++ b/doc/ci/pipelines/schedules.md @@ -26,7 +26,7 @@ Otherwise, the pipeline is not created. No error message is displayed. To add a pipeline schedule: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Build > Pipeline schedules**. 1. Select **New schedule** and fill in the form. - **Interval Pattern**: Select one of the preconfigured intervals, or enter a custom @@ -47,7 +47,7 @@ you must delete unused schedules before you can add another. The owner of a pipeline schedule can edit it: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Build > Pipeline schedules**. 1. Next to the schedule, select **Edit** (**{pencil}**) and fill in the form. @@ -60,7 +60,7 @@ of the schedule. To trigger a pipeline schedule manually, so that it runs immediately instead of the next scheduled time: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Build > Pipeline schedules**. 1. On the right of the list, for the pipeline you want to run, select **Play** (**{play}**). @@ -79,7 +79,7 @@ including [protected environments](../environments/protected_environments.md) an To take ownership of a pipeline created by a different user: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Build > Pipeline schedules**. 1. On the right of the list, for the pipeline you want to become owner of, select **Take ownership**. diff --git a/doc/ci/pipelines/settings.md b/doc/ci/pipelines/settings.md index b9c95c63098..02559da75a0 100644 --- a/doc/ci/pipelines/settings.md +++ b/doc/ci/pipelines/settings.md @@ -24,7 +24,7 @@ For public and internal projects, you can change who can see your: To change the visibility of your pipelines and related features: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **General pipelines**. 1. Select or clear the **Public pipelines** checkbox. @@ -56,7 +56,7 @@ This setting has no effect when: To change the pipeline visibility for non-project members: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. Expand **Visibility, project features, permissions**. 1. For **CI/CD**, choose: @@ -72,7 +72,7 @@ is selected. You can set pending or running pipelines to cancel automatically when a pipeline for new changes runs on the same branch. You can enable this in the project settings: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **General Pipelines**. 1. Select the **Auto-cancel redundant pipelines** checkbox. @@ -94,7 +94,7 @@ newer one, which may not be what you want. To avoid this scenario: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **General pipelines**. 1. Select the **Prevent outdated deployment jobs** checkbox. @@ -112,7 +112,7 @@ directory. However, you can specify an alternate filename path, including locati To customize the path: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **General pipelines**. 1. In the **CI/CD configuration file** field, enter the filename. If the file: @@ -161,7 +161,7 @@ able to edit it. You can choose how your repository is fetched from GitLab when a job runs. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **General pipelines**. 1. Under **Git strategy**, select an option: @@ -169,7 +169,7 @@ You can choose how your repository is fetched from GitLab when a job runs. for every job. However, the local working copy is always pristine. - `git fetch` is faster because it re-uses the local working copy (and falls back to clone if it doesn't exist). This is recommended, especially for - [large repositories](../large_repositories/index.md#git-strategy). + [large repositories](../../user/project/repository/managing_large_repositories.md#git-strategy). The configured Git strategy can be overridden by the [`GIT_STRATEGY` variable](../runners/configure_runners.md#git-strategy) in the `.gitlab-ci.yml` file. @@ -182,7 +182,7 @@ in the `.gitlab-ci.yml` file. You can limit the number of changes that GitLab CI/CD fetches when it clones a repository. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **General pipelines**. 1. Under **Git strategy**, under **Git shallow clone**, enter a value. @@ -192,14 +192,14 @@ a repository. In GitLab versions 14.7 and later, newly created projects have a default `git depth` value of `20`. GitLab versions 14.6 and earlier have a default `git depth` value of `50`. -This value can be overridden by the [`GIT_DEPTH` variable](../large_repositories/index.md#shallow-cloning) +This value can be overridden by the [`GIT_DEPTH` variable](../../user/project/repository/managing_large_repositories.md#shallow-cloning) in the `.gitlab-ci.yml` file. ## Set a limit for how long jobs can run You can define how long a job can run before it times out. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **General pipelines**. 1. In the **Timeout** field, enter the number of minutes, or a human-readable value like `2 hours`. diff --git a/doc/ci/quick_start/tutorial.md b/doc/ci/quick_start/tutorial.md index acc47a07a02..41239615590 100644 --- a/doc/ci/quick_start/tutorial.md +++ b/doc/ci/quick_start/tutorial.md @@ -8,10 +8,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w This tutorial walks you through configuring a progressively more complex CI/CD pipeline through small, iterative steps. The pipeline is always fully functional, -but it gains more functionality with each step. +but it gains more functionality with each step. The goal is to build, test, and deploy +a documentation site. -When you finish this tutorial, you will have a new project on GitLab.com and a working documentation site on -[Docusaurus](https://docusaurus.io/). +When you finish this tutorial, you will have a new project on GitLab.com and a working documentation site +using [Docusaurus](https://docusaurus.io/). To complete this tutorial, you will: @@ -36,8 +37,8 @@ Before adding the pipeline configuration, you must first set up a Docusaurus pro on GitLab.com: 1. Create a new project under your username (not a group): - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). - 1. Select **View all your projects**. + 1. On the left sidebar, select **Search or go to**. + 1. Select **View all my projects**. 1. On the right of the page, select **New project**. 1. Select **Create blank project**. 1. Enter the project details: diff --git a/doc/ci/resource_groups/index.md b/doc/ci/resource_groups/index.md index cf29ab62240..a0f6a8008d0 100644 --- a/doc/ci/resource_groups/index.md +++ b/doc/ci/resource_groups/index.md @@ -9,7 +9,7 @@ description: Control the job concurrency in GitLab CI/CD > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15536) in GitLab 12.7. -By default, pipelines in GitLab CI/CD run in parallel. The parallelization is an important factor to improve +By default, pipelines in GitLab CI/CD run concurrently. Concurrency is an important factor to improve the feedback loop in merge requests, however, there are some situations that you may want to limit the concurrency on deployment jobs to run them one by one. @@ -130,13 +130,13 @@ pipelines run almost at the same time: Depending on the process mode of the resource group: - If the process mode is set to `unordered`: - - `deploy-1`, `deploy-2`, and `deploy-3` do not run in parallel. + - `deploy-1`, `deploy-2`, and `deploy-3` do not run concurrently. - There is no guarantee on the job execution order, for example, `deploy-1` could run before or after `deploy-3` runs. - If the process mode is `oldest_first`: - - `deploy-1`, `deploy-2`, and `deploy-3` do not run in parallel. + - `deploy-1`, `deploy-2`, and `deploy-3` do not run concurrently. - `deploy-1` runs first, `deploy-2` runs second, and `deploy-3` runs last. - If the process mode is `newest_first`: - - `deploy-1`, `deploy-2`, and `deploy-3` do not run in parallel. + - `deploy-1`, `deploy-2`, and `deploy-3` do not run concurrently. - `deploy-3` runs first, `deploy-2` runs second and `deploy-1` runs last. ## Pipeline-level concurrency control with cross-project/parent-child pipelines diff --git a/doc/ci/review_apps/img/enable_review_app_v16.png b/doc/ci/review_apps/img/enable_review_app_v16.png Binary files differnew file mode 100644 index 00000000000..00e305d6a6b --- /dev/null +++ b/doc/ci/review_apps/img/enable_review_app_v16.png diff --git a/doc/ci/review_apps/index.md b/doc/ci/review_apps/index.md index d05861818e2..f831c53c024 100644 --- a/doc/ci/review_apps/index.md +++ b/doc/ci/review_apps/index.md @@ -76,14 +76,14 @@ Prerequisite: To use the review apps template: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to +1. On the left sidebar, select **Search or go to** and find the project you want to create a review app job for. -1. Select **Build > Environments**. +1. Select **Operate > Environments**. 1. Select **Enable review apps**. 1. Copy the provided code snippet and paste it into your `.gitlab-ci.yml` file: -  +  You can edit this template as needed. diff --git a/doc/ci/runners/configure_runners.md b/doc/ci/runners/configure_runners.md index 3e26c120f74..fd1716cc58a 100644 --- a/doc/ci/runners/configure_runners.md +++ b/doc/ci/runners/configure_runners.md @@ -64,9 +64,9 @@ of shared runners on large GitLab instances. This ensures that you control access to your GitLab instances and secure [runner executors](https://docs.gitlab.com/runner/executors/). If certain executors run a job, the file system, the code the runner executes, -and the runner token may be exposed. This means that anyone that runs jobs +and the runner authentication token may be exposed. This means that anyone that runs jobs on a _shared runner_ can access another user's code that runs on the runner. -Users with access to the runner token can use it to create a clone of +Users with access to the runner authentication token can use it to create a clone of a runner and submit false jobs in a vector attack. For more information, see [Security Considerations](https://docs.gitlab.com/runner/security/). ### Prevent runners from revealing sensitive information @@ -76,7 +76,7 @@ on [protected branches](../../user/project/protected_branches.md), or jobs that To prevent runners from revealing sensitive information: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Runners**. 1. Find the runner you want to protect or unprotect. Make sure the runner is enabled. @@ -111,7 +111,7 @@ That new runner may then be used to obtain the values of secret variables or to To reset the registration token: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Runners**. 1. To the right of **New project runner**, select the vertical ellipsis (**{ellipsis_v}**). @@ -124,19 +124,19 @@ you use to provision and register new values. ### Reset the runner authentication token -If an authentication token is revealed, an attacker could use the token to [clone a runner](https://docs.gitlab.com/runner/security/#cloning-a-runner). +If a runner authentication token is revealed, an attacker could use the token to [clone a runner](https://docs.gitlab.com/runner/security/#cloning-a-runner). -To reset the authentication token: +To reset the runner authentication token: 1. Delete the runner: - [Delete a shared runner](runners_scope.md#delete-shared-runners). - [Delete a group runner](runners_scope.md#delete-a-group-runner). - [Delete a project runner](runners_scope.md#delete-a-project-runner). -1. Create a new runner so that it is assigned a new authentication token: - - [Create a shared runner](runners_scope.md#create-a-shared-runner-with-an-authentication-token). - - [Create a group runner](runners_scope.md#create-a-group-runner-with-an-authentication-token). - - [Create a project runner](runners_scope.md#create-a-project-runner-with-an-authentication-token). -1. Optional. To verify that the previous authentication token has been revoked, use the [Runners API](../../api/runners.md#verify-authentication-for-a-registered-runner). +1. Create a new runner so that it is assigned a new runner authentication token: + - [Create a shared runner](runners_scope.md#create-a-shared-runner-with-a-runner-authentication-token). + - [Create a group runner](runners_scope.md#create-a-group-runner-with-a-runner-authentication-token). + - [Create a project runner](runners_scope.md#create-a-project-runner-with-a-runner-authentication-token). +1. Optional. To verify that the previous runner authentication token has been revoked, use the [Runners API](../../api/runners.md#verify-authentication-for-a-registered-runner). ## Use tags to control which jobs a runner can run @@ -874,7 +874,7 @@ defaults to the number of CPUs available. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30942) in GitLab 15.3 [with a flag](../../administration/feature_flags.md) named `enforce_runner_token_expires_at`. Disabled by default. > - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/377902) in GitLab 15.5. Feature flag `enforce_runner_token_expires_at` removed. -Each runner has an [authentication token](../../api/runners.md#registration-and-authentication-tokens) +Each runner has an [runner authentication token](../../api/runners.md#registration-and-authentication-tokens) to connect with the GitLab instance. To help prevent the token from being compromised, you can have the @@ -883,12 +883,12 @@ they are updated for each runner, regardless of the runner's status (`online` or No manual intervention should be required, and no running jobs should be affected. -If you need to manually update the authentication token, you can run a +If you need to manually update the runner authentication token, you can run a command to [reset the token](https://docs.gitlab.com/runner/commands/#gitlab-runner-reset-token). -### Automatically rotate authentication tokens +### Automatically rotate runner authentication tokens -You can specify an interval for authentication tokens to rotate. +You can specify an interval for runner authentication tokens to rotate. This rotation helps ensure the security of the tokens assigned to your runners. Prerequisites: @@ -897,11 +897,11 @@ Prerequisites: To automatically rotate runner authentication tokens: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**.. 1. On the left sidebar, select **Settings > CI/CD**. 1. Expand **Continuous Integration and Deployment** 1. Set a **Runners expiration** time for runners, leave empty for no expiration. 1. Select **Save**. -Before the interval expires, runners automatically request a new authentication token. +Before the interval expires, runners automatically request a new runner authentication token. diff --git a/doc/ci/runners/index.md b/doc/ci/runners/index.md index 5427911e1ce..7e597264e1e 100644 --- a/doc/ci/runners/index.md +++ b/doc/ci/runners/index.md @@ -17,10 +17,15 @@ Your jobs can run on: - [Windows runners](saas/windows_saas_runner.md) ([Beta](../../policy/experiment-beta-support.md#beta)) - [macOS runners](saas/macos_saas_runner.md) ([Beta](../../policy/experiment-beta-support.md#beta)) -Refer to the compute minutes [cost factor](../../ci/pipelines/cicd_minutes.md#cost-factor) for the cost factor applied to the machine type based on size. -The number of minutes you can use on these runners depends on the [maximum number of compute minutes](../pipelines/cicd_minutes.md) +For more information about the cost factor applied to the machine type based on size, see [cost factor](../../ci/pipelines/cicd_minutes.md#cost-factor). +The number of minutes you can use on these runners depends on the [maximum number of units of compute](../pipelines/cicd_minutes.md) in your [subscription plan](https://about.gitlab.com/pricing/). +[Untagged](../../ci/runners/configure_runners.md#use-tags-to-control-which-jobs-a-runner-can-run) jobs automatically run in containers +on the `small` Linux runners. + +The objective is to make 90% of CI/CD jobs start executing in 120 seconds or less. The error rate should be less than 0.5%. + ## How SaaS runners work When you use SaaS runners: @@ -30,9 +35,6 @@ When you use SaaS runners: - The virtual machine where your job runs has `sudo` access with no password. - The storage is shared by the operating system, the image with pre-installed software, and a copy of your cloned repository. This means that the available free disk space for your jobs to use is reduced. -- [Untagged](../../ci/runners/configure_runners.md#use-tags-to-control-which-jobs-a-runner-can-run) jobs automatically run in containers -on the `small` Linux runners. -- The objective is to make 90% of CI jobs start executing in 120 seconds or less. The error rate target will be less than 0.5%. NOTE: Jobs handled by SaaS runners on GitLab.com **time out after 3 hours**, regardless of the timeout configured in a project. @@ -67,3 +69,65 @@ takes over the task of securely deleting the virtual machine and associated data - Inbound communication from the public internet to the temporary VM is not allowed. - Firewall rules do not permit communication between VMs. - The only internal communication allowed to the temporary VMs is from the runner manager. + +## Supported image lifecycle + +For runners on macOS and Windows, you can only run jobs on supported images. You cannot bring your own image. Supported images have the following lifecycle: + +- Beta +- Generally Available +- Deprecated + +### Beta + +To gather feedback on an image prior to making the image Generally Available (GA) and to address +any issues, new images are released as Beta. Any jobs running on Beta images are not +covered by the service-level agreement. If you use Beta images, you can provide feedback +by creating an issue. + +### Generally Available + +A Generally Available (GA) image is released after the image completes a Beta phase +and is considered suitable for general use. To become GA, the +image must fulfill the following requirements: + +- Successful completion of a Beta phase by resolving all reported significant bugs +- Compatibility of installed software with the underlying OS + +Jobs running on GA images are covered by the defined service-level agreement. Over time, these images are deprecated. + +### Deprecated + +A maximum of two Generally Available (GA) images are supported at a time. After a new GA image is released, +the oldest GA image becomes deprecated. A deprecated image is no longer +updated and is deleted after 3 months in accordance with the [deprecation guidelines](../../development/deprecation_guidelines/index.md). + +## Major version changes (breaking) + +As GitLab CI/CD and Runner have evolved, certain breaking changes have been necessary. + +For GitLab 15.0 and later, all breaking changes are documented on the following page: + +- [Deprecations and removals](../../update/deprecations.md) + +The breaking changes for GitLab Runner in earlier major version releases are: + +- 14.0: No breaking changes. +- 13.0: + - [Remove Backported `os.Expand`](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4915). + - [Remove Fedora 29 package support](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/16158). + - [Remove macOS 32-bit support](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/25466). + - [Removed `debug/jobs/list?v=1` endpoint](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6361). + - [Remove support for array of strings when defining services for Docker executor](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4922). + - [Remove `--docker-services` flag on register command](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6404). + - [Remove legacy build directory caching](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4180). + - [Remove `FF_USE_LEGACY_VOLUMES_MOUNTING_ORDER` feature flag](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6581). + - [Remove support for Windows Server 1803](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6553). +- 12.0: + - [Use `refspec` to clone/fetch Git repository](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4069). + - [Old cache configuration](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4070). + - [Old metrics server configuration](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4072). + - [Remove `FF_K8S_USE_ENTRYPOINT_OVER_COMMAND`](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4073). + - [Remove Linux distributions that reach EOL](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1130). + - [Update command line API for helper images](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4013). + - [Remove old `git clean` flow](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4175). diff --git a/doc/ci/runners/new_creation_workflow.md b/doc/ci/runners/new_creation_workflow.md index 55ff5165ff7..c4c2fbebb12 100644 --- a/doc/ci/runners/new_creation_workflow.md +++ b/doc/ci/runners/new_creation_workflow.md @@ -14,15 +14,13 @@ As with all projects, the items mentioned on this page are subject to change or The development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc. -In GitLab 16.0, we introduced a new runner creation workflow that uses authentication tokens to register -runners. The legacy workflow that uses registration tokens is deprecated and will be removed in GitLab 17.0. +In GitLab 16.0, we introduced a new runner creation workflow that uses runner authentication tokens to register +runners. The legacy workflow that uses registration tokens is deprecated and will be removed in GitLab 18.0. For information about the current development status of the new workflow, see [epic 7663](https://gitlab.com/groups/gitlab-org/-/epics/7663). For information about the technical design and reasons for the new architecture, see [Next GitLab Runner Token Architecture](../../architecture/blueprints/runner_tokens/index.md). -## Feedback - If you experience problems or have concerns about the new runner registration workflow, or if the following information is not sufficient, you can let us know in the [feedback issue](https://gitlab.com/gitlab-org/gitlab/-/issues/387993). @@ -32,8 +30,8 @@ you can let us know in the [feedback issue](https://gitlab.com/gitlab-org/gitlab For the new runner registration workflow, you: 1. [Create a runner](register_runner.md) directly in the GitLab UI. -1. Receive an authentication token. -1. Use the authentication token instead of the registration token when you register +1. Receive a runner authentication token. +1. Use the runner authentication token instead of the registration token when you register a runner with this configuration. Runner managers registered in multiple hosts appear under the same runner in the GitLab UI, but with an identifying system ID. @@ -46,54 +44,63 @@ multiple runners. For more information, see [Reusing a GitLab Runner configurati ## Estimated time frame for planned changes - In GitLab 15.10 and later, you can use the new runner registration workflow. -- In GitLab 16.6, we plan to disable registration tokens. -- In GitLab 17.0, we plan to completely remove support for runner registration tokens. +- In GitLab 17.0, we plan to disable runner registration tokens. +- In GitLab 18.0, we plan to completely remove support for runner registration tokens. ## Prevent your runner registration workflow from breaking -Until GitLab 16.6, you can still use the legacy runner registration workflow. +Until GitLab 17.0, you can still use the legacy runner registration workflow. -In GitLab 16.6, the legacy runner registration workflow will be disabled automatically. You will be able to manually re-enable the legacy runner registration workflow for a limited time. For more information, see -[Using registration tokens after GitLab 16.6](#using-registration-tokens-after-gitlab-166). +In GitLab 17.0, the legacy runner registration workflow will be disabled automatically. You will be able to manually re-enable the legacy runner registration workflow for a limited time. For more information, see +[Using registration tokens after GitLab 17.0](#using-registration-tokens-after-gitlab-170). -If no action is taken before your GitLab instance is upgraded to GitLab 16.6, then your runner registration +If no action is taken before your GitLab instance is upgraded to GitLab 17.0, then your runner registration workflow will break. To avoid a broken workflow, you must: -1. [Create a shared runner](runners_scope.md#create-a-shared-runner-with-an-authentication-token) and obtain the authentication token. +1. [Create a shared runner](runners_scope.md#create-a-shared-runner-with-a-runner-authentication-token) and obtain the authentication token. 1. Replace the registration token in your runner registration workflow with the authentication token. -## Using registration tokens after GitLab 16.6 +## Using registration tokens after GitLab 17.0 -To continue using registration tokens after GitLab 16.6: +To continue using registration tokens after GitLab 17.0: -- On GitLab.com, you can manually re-enable the legacy runner registration process in the top-level group settings until GitLab 16.8. -- On GitLab self-managed, you can manually re-enable the legacy runner registration process in the Admin Area settings until GitLab 17.0. +- On GitLab.com, you can manually re-enable the legacy runner registration process in the top-level group settings until GitLab 18.0. +- On GitLab self-managed, you can manually re-enable the legacy runner registration process in the Admin Area settings until GitLab 18.0. Plans to implement a UI setting to re-enable registration tokens are proposed in [issue 411923](https://gitlab.com/gitlab-org/gitlab/-/issues/411923) +## Runners registered with a registration token will continue to work after 18.0 + +Existing runners will not be affected by these changes, they will still work even after the legacy registration method is removed. + ## Changes to the `gitlab-runner register` command syntax -The `gitlab-runner register` command will stop accepting registration tokens and instead accept new +The `gitlab-runner register` command will stop accepting registration tokens and instead accept new runner authentication tokens generated in the GitLab runners administration page. -These authentication tokens are recognizable by their `glrt-` prefix. +The runner authentication tokens are recognizable by their `glrt-` prefix. When you create a runner in the GitLab UI, you specify configuration values that were previously command-line options prompted by the `gitlab-runner register` command. These command-line options have been [deprecated](../../update/deprecations.md#registration-tokens-and-server-side-runner-arguments-in-post-apiv4runners-endpoint). -If you specify an authentication token with: +If you specify a runner authentication token with: - the `--token` command-line option, the `gitlab-runner register` command does not accept the configuration values. - the `--registration-token` command-line option, the `gitlab-runner register` command ignores the configuration values. +| Token | Registration command | +|----------------------------------------|-----------------------------------------------------------------------------------------------------------| +| Runner authentication token | `gitlab-runner register --token $RUNNER_AUTHENTICATION_TOKEN` | +| Runner registration token (deprecated) | `gitlab-runner register --registration-token $RUNNER_REGISTRATION_TOKEN <runner configuration arguments>` | + Authentication tokens have the prefix, `glrt-`. To ensure minimal disruption to your automation workflow, [legacy-compatible registration processing](https://docs.gitlab.com/runner/register/#legacy-compatible-registration-processing) -triggers if an authentication token is specified in the legacy parameter `--registration-token`. +triggers if a runner authentication token is specified in the legacy parameter `--registration-token`. Example command for GitLab 15.9: @@ -126,7 +133,7 @@ gitlab-runner register \ ## Impact on autoscaling In autoscaling scenarios such as GitLab Runner Operator or GitLab Runner Helm Chart, the -registration token is replaced with the authentication token generated from the UI. +registration token is replaced with the runner authentication token generated from the UI. This means that the same runner configuration is reused across jobs, instead of creating a runner for each job. The specific runner can be identified by the unique system ID that is generated when the runner @@ -138,33 +145,13 @@ Existing runners will continue to work as usual. This change only affects regist ## Creating runners programmatically -A new [POST /user/runners REST API](../../api/users.md#create-a-runner) was introduced in -GitLab 15.11, which allows a runner to be created in the context of an authenticated user. This should only be used in -scenarios where the runner configuration is dynamic, or not reusable. If the runner configuration is static, it is -preferable to reuse the authentication token of an existing runner. +In GitLab 15.11 and later, you can use the [POST /user/runners REST API](../../api/users.md#create-a-runner) +to create a runner as an authenticated user. This should only be used if the runner configuration is dynamic +or not reusable. If the runner configuration is static, you should reuse the runner authentication token of +an existing runner. -The following snippet shows how a group runner could be created and registered with a -[Group Access Token](../../user/group/settings/group_access_tokens.md) using the new creation flow. -The process is very similar when using [Project Access Tokens](../../user/project/settings/project_access_tokens.md) -or [Personal Access Tokens](../../user/profile/personal_access_tokens.md): - -```shell -# `GROUP_ID` contains the numerical ID of the group where the runner will be created -# `GITLAB_TOKEN` can be a Personal Access Token for a group owner, or a Group Access Token on the respective group -# created with `owner` access and `api` scope. -# -# The output will be parsed by `jq` to extract the token of the newly created runner -RUNNER_TOKEN=$(curl --silent --request POST "https://gitlab.com/api/v4/user/runners" \ - --header "private-token: $GITLAB_TOKEN" \ - --data runner_type=group_type --data group_id=$GROUP_ID --data 'description=My runner' --data 'tag_list=java,linux' \ - | jq -r '.token') - -gitlab-runner register \ - --non-interactive \ - --executor "shell" \ - --url "https://gitlab.com/" \ - --token "$RUNNER_TOKEN" -``` +For instructions about how to automate runner creation and registration, see the tutorial, +[Automate runner creation and registration](../../tutorials/automate_runner_creation/index.md). ## Installing GitLab Runner with Helm chart @@ -185,7 +172,7 @@ runUntagged: true protected: true ``` -If you store the runner token in `secrets`, you must also modify them. +If you store the runner authentication token in `secrets`, you must also modify them. In the legacy runner registration workflow, fields were specified with: @@ -212,3 +199,7 @@ data: runner-registration-token: "" # need to leave as an empty string for compatibility reasons runner-token: "REDACTED" ``` + +NOTE: +If your secret management solution doesn't allow you to set an empty string for `runner-registration-token`, +you can set it to any string - it will be ignored when `runner-token` is present. diff --git a/doc/ci/runners/runners_scope.md b/doc/ci/runners/runners_scope.md index fff695eb606..c6387a60495 100644 --- a/doc/ci/runners/runners_scope.md +++ b/doc/ci/runners/runners_scope.md @@ -34,7 +34,7 @@ If you are using GitLab.com: - The shared runners consume the [compute minutes](../pipelines/cicd_minutes.md) included with your account. -### Create a shared runner with an authentication token +### Create a shared runner with a runner authentication token > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/383139) in GitLab 15.10. Deployed behind the `create_runner_workflow_for_admin` [flag](../../administration/feature_flags.md) > - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/389269) in GitLab 16.0. @@ -44,23 +44,32 @@ Prerequisite: - You must be an administrator. -When you create a runner, it is assigned an authentication token that you use to register it. The runner uses the token to authenticate with GitLab when picking up jobs from the job queue. +When you create a runner, it is assigned a runner authentication token that you use to register it. The runner uses the token to authenticate with GitLab when picking up jobs from the job queue. To create a shared runner: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **CI/CD > Runners**. 1. Select **New instance runner**. -1. Select a platform. -1. Optional. Enter configurations for the runner. -1. Select **Submit**. -1. Follow the on-screen instructions to register the runner from the command line. - -You can also [create a runner](../../api/users.md#create-a-runner) with the API to generate an authentication token. +1. Select the operating system where GitLab Runner is installed. +1. In the **Tags** section, in the **Tags** field, enter the job tags to specify jobs the runner can run. + If there are no job tags for this runner, select **Run untagged**. +1. Optional. In the **Runner description** field, to add a runner description + that displays in GitLab, enter a runner description. +1. Optional. In the **Configuration** section, add additional configurations. +1. Select **Create runner**. +1. Follow the on-screen instructions to register the runner from the command line. When prompted by the command line: + - For the `GitLab instance URL`, use the URL for your GitLab instance. For example, if your project + is hosted on `gitlab.example.com/yourname/yourproject`, your GitLab instance URL is `https://gitlab.example.com`. + - For the `executor`, enter the type of [executor](https://docs.gitlab.com/runner/executors/). The executor is the + environment where the runner executes the job. + +You can also [use the API](../../api/users.md#create-a-runner) to create a runner. NOTE: -The authentication token displays in the UI for only a short period of time during registration. +The runner authentication token displays in the UI for a limited period of time during registration. After you register the runner, +the authentication token is stored in the `config.toml`. ### Create a shared runner with a registration token (deprecated) @@ -75,7 +84,7 @@ Prerequisite: To create a shared runner: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **CI/CD > Runners**. 1. Select **Register an instance runner**. @@ -90,7 +99,7 @@ Prerequisite: You can pause a runner so that it does not accept jobs from groups and projects in the GitLab instance. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **CI/CD > Runners**. 1. In the search box, enter the runner description or filter the runner list. @@ -110,7 +119,7 @@ jobs, you can [pause](#pause-or-resume-a-shared-runner) the runner instead. To delete a single or multiple shared runners: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **CI/CD > Runners**. 1. In the search box, enter the runner description or filter the list of runners. @@ -134,7 +143,7 @@ For existing projects, an administrator must To enable shared runners for a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Runners**. 1. Turn on the **Enable shared runners for this project** toggle. @@ -143,7 +152,7 @@ To enable shared runners for a project: To enable shared runners for a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > CI/CD**. 1. Expand **Runners**. 1. Turn on the **Enable shared runners for this group** toggle. @@ -156,7 +165,7 @@ or group. To disable shared runners for a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Runners**. 1. In the **Shared runners** area, turn off the **Enable shared runners for this project** toggle. @@ -170,7 +179,7 @@ Shared runners are automatically disabled for a project: To disable shared runners for a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > CI/CD**. 1. Expand **Runners**. 1. Turn off the **Enable shared runners for this group** toggle. @@ -222,7 +231,7 @@ to have access to a set of runners. Group runners process jobs by using a first in, first out queue. -### Create a group runner with an authentication token +### Create a group runner with a runner authentication token > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/383143) in GitLab 15.10. Deployed behind the `create_runner_workflow_for_namespace` [flag](../../administration/feature_flags.md). Disabled by default. > - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/393919) in GitLab 16.0. @@ -233,22 +242,31 @@ Prerequisites: - You must have the Owner role for the group. You can create a group runner for your self-managed GitLab instance or for GitLab.com. -When you create a runner, it is assigned an authentication token that you use to register it. The runner uses the token to authenticate with GitLab when picking up jobs from the job queue. +When you create a runner, it is assigned a runner authentication token that you use to register it. +The runner uses the token to authenticate with GitLab when it picks up jobs from the job queue. To create a group runner: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Build > Runners**. 1. Select **New group runner**. -1. Select a platform. -1. Optional. Enter configurations for the runner. -1. Select **Submit**. -1. Follow the on-screen instructions to register the runner from the command line. - -You can also [create a runner](../../api/users.md#create-a-runner) with the API to generate an authentication token. +1. Select the operating system where GitLab Runner is installed. +1. In the **Tags** section, in the **Tags** field, enter the job tags to specify jobs the runner can run. + If there are no job tags for this runner, select **Run untagged**. +1. Optional. In the **Runner description** field, add a runner description + that displays in GitLab. +1. Optional. In the **Configuration** section, add additional configurations. +1. Select **Create runner**. +1. Follow the on-screen instructions to register the runner from the command line. When prompted by the command line: + - For the `GitLab instance URL`, use the URL for your GitLab instance. For example, if your project + is hosted on `gitlab.example.com/yourname/yourproject`, your GitLab instance URL is `https://gitlab.example.com`. + - For the `executor`, enter the type of [executor](https://docs.gitlab.com/runner/executors/). The executor is the + environment where the runner executes the job. + +You can also [use the API](../../api/users.md#create-a-runner) to create a runner. NOTE: -The authentication token displays in the UI for only a short period of time during registration. +The runner authentication token displays in the UI for only a short period of time during registration. ### Create a group runner with a registration token (deprecated) @@ -256,7 +274,7 @@ The authentication token displays in the UI for only a short period of time duri WARNING: The ability to pass a runner registration token, and support for certain configuration arguments was -[deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/380872) in GitLab 15.6 and will be removed in GitLab 17.0. Authentication tokens +[deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/380872) in GitLab 15.6 and will be removed in GitLab 18.0. Authentication tokens should be used instead. For more information, see [Migrating to the new runner registration workflow](new_creation_workflow.md). You must have the Owner role for the group. @@ -264,7 +282,7 @@ You must have the Owner role for the group. To create a group runner: 1. [Install GitLab Runner](https://docs.gitlab.com/runner/install/). -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Build > Runners**. 1. In the upper-right corner, select **Register a group runner**. 1. Select **Show runner installation and registration instructions**. @@ -273,17 +291,20 @@ To create a group runner: Alternately, you can copy the registration token and follow the documentation for how to [register a runner](https://docs.gitlab.com/runner/register/). -### View and manage group runners +### View group runners + +> Ability for users with the Maintainer role to view group runners [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/384179) in GitLab 16.4. + +Prerequisite: + +- You must have the Maintainer or Owner role for the group. -You can view and manage all runners for a group, its subgroups, and projects. +You can view all runners for a group and its subgroups and projects. You can do this for your self-managed GitLab instance or for GitLab.com. -You must have the Owner role for the group. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Build > Runners**. -From this page, you can edit, pause, and remove runners from the group, its subgroups, and projects. - #### Filter group runners to show only inherited > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/337838/) in GitLab 15.5. @@ -297,7 +318,7 @@ By default, only those that are inherited are shown. To show all runners available in the instance, including shared runners and those in other groups: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Build > Runners**. 1. Above the list, turn off the **Show only inherited** toggle. @@ -310,7 +331,7 @@ Prerequisite: You can pause a runner so that it does not accept jobs from subgroups and projects in the GitLab instance. If you pause a group runner that is used by multiple projects, the runner pauses for all projects. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Build > Runners**. 1. In the search box, enter the runner description or filter the runner list. 1. In the runner list, to the right of the runner: @@ -331,7 +352,7 @@ jobs, you can [pause](#pause-or-resume-a-group-runner) the runner instead. To delete a single or multiple group runners: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Build > Runners**. 1. In the search box, enter the runner description or filter the list of runners. 1. Delete the group runner: @@ -344,11 +365,15 @@ To delete a single or multiple group runners: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/363012) in GitLab 15.1. +Prerequisite: + +- You must have the Owner role for the group. + You can clean up group runners that have been inactive for more than three months. Group runners are those that were created at the group level. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > CI/CD**. 1. Expand **Runners**. 1. Turn on the **Enable stale runner cleanup** toggle. @@ -399,7 +424,7 @@ NOTE: Project runners do not get shared with forked projects automatically. A fork *does* copy the CI/CD settings of the cloned repository. -### Create a project runner with an authentication token +### Create a project runner with a runner authentication token > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/383143) in GitLab 15.10. Deployed behind the `create_runner_workflow_for_namespace` [flag](../../administration/feature_flags.md). Disabled by default. > - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/393919) in GitLab 16.0. @@ -409,23 +434,33 @@ Prerequisites: - You must have the Maintainer role for the project. -You can create a project runner for your self-managed GitLab instance or for GitLab.com. When you create a runner, it is assigned an authentication token that you use to register to the runner. The runner uses the token to authenticate with GitLab when picking up jobs from the job queue. +You can create a project runner for your self-managed GitLab instance or for GitLab.com. When you create a runner, +it is assigned a runner authentication token that you use to register to the runner. The runner uses the token to +authenticate with GitLab when it picks up jobs from the job queue. To create a project runner: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand the **Runners** section. 1. Select **New project runner**. -1. Select a platform. -1. Optional. Enter configurations for the runner. -1. Select **Submit**. -1. Follow the on-screen instructions to register the runner from the command line. - -You can also [create a runner](../../api/users.md#create-a-runner) with the API to generate an authentication token. +1. Select the operating system where GitLab Runner is installed. +1. In the **Tags** section, in the **Tags** field, enter the job tags to specify jobs the runner can run. + If there are no job tags for this runner, select **Run untagged**. +1. Optional. In the **Runner description** field, add a description for the runner + that displays in GitLab. +1. Optional. In the **Configuration** section, add additional configurations. +1. Select **Create runner**. +1. Follow the on-screen instructions to register the runner from the command line. When prompted by the command line: + - For the `GitLab instance URL`, use the URL for your GitLab instance. For example, if your project + is hosted on `gitlab.example.com/yourname/yourproject`, your GitLab instance URL is `https://gitlab.example.com`. + - For the `executor`, enter the type of [executor](https://docs.gitlab.com/runner/executors/). The executor is the + environment where the runner executes the job. + +You can also [use the API](../../api/users.md#create-a-runner) to create a runner. NOTE: -The authentication token displays in the UI for only a short period of time during registration. +The runner authentication token displays in the UI for only a short period of time during registration. ### Create a project runner with a registration token (deprecated) @@ -441,7 +476,7 @@ Prerequisite: To create a project runner: 1. [Install GitLab Runner](https://docs.gitlab.com/runner/install/). -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to +1. On the left sidebar, select **Search or go to** and find the project where you want to use the runner. 1. Select **Settings > CI/CD**. 1. Expand **Runners**. @@ -459,7 +494,7 @@ Prerequisite: You can pause a project runner so that it does not accept jobs from projects it's assigned to in the GitLab instance. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to +1. On the left sidebar, select **Search or go to** and find the project where you want to enable the runner. 1. Select **Settings > CI/CD**. 1. Expand **Runners**. @@ -479,7 +514,7 @@ When you delete a project runner, it is permanently deleted from the GitLab inst no longer be used by projects. If you want to temporarily stop the runner from accepting jobs, you can [pause](#pause-or-resume-a-project-runner) the runner instead. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to +1. On the left sidebar, select **Search or go to** and find the project where you want to enable the runner. 1. Select **Settings > CI/CD**. 1. Expand **Runners**. @@ -500,7 +535,7 @@ You must have at least the Maintainer role for: To enable a project runner for a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to +1. On the left sidebar, select **Search or go to** and find the project where you want to enable the runner. 1. Select **Settings > CI/CD**. 1. Expand **Runners**. @@ -520,7 +555,7 @@ but can also be changed later. To lock or unlock a project runner: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to +1. On the left sidebar, select **Search or go to** and find the project where you want to enable the runner. 1. Select **Settings > CI/CD**. 1. Expand **Runners**. @@ -553,7 +588,7 @@ runners are considered. queued for longer than the median value, and half of the jobs queued for less than the median value. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **CI/CD > Runners**. 1. Select **View metrics**. @@ -569,10 +604,10 @@ To determine which runners need to be upgraded: 1. View the list of runners: - For a group: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. + 1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Build > Runners**. - For the instance: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **CI/CD > Runners**. @@ -602,7 +637,7 @@ Prerequisite: To determine the IP address of a shared runner: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **CI/CD > Runners**. 1. Find the runner in the table and view the **IP Address** column. diff --git a/doc/ci/runners/saas/linux_saas_runner.md b/doc/ci/runners/saas/linux_saas_runner.md index dd5381f3cd6..c026ccf3d22 100644 --- a/doc/ci/runners/saas/linux_saas_runner.md +++ b/doc/ci/runners/saas/linux_saas_runner.md @@ -29,7 +29,7 @@ The `small` machine type is set as default. If no [tag](../../yaml/index.md#tags the jobs will run on this default runner. All SaaS runners on Linux currently run on -[`n2d-standard`](https://cloud.google.com/compute/docs/general-purpose-machines#n2d_machines) gerneral-purpose compute from GCP. +[`n2d-standard`](https://cloud.google.com/compute/docs/general-purpose-machines#n2d_machines) general-purpose compute from GCP. The machine type and underlying processor type can change. Jobs optimized for a specific processor design could behave inconsistently. ## Container images diff --git a/doc/ci/runners/saas/macos/codesigning.md b/doc/ci/runners/saas/macos/codesigning.md deleted file mode 100644 index 7e70e984c7c..00000000000 --- a/doc/ci/runners/saas/macos/codesigning.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../macos_saas_runner.md' -remove_date: '2023-09-05' ---- - -This document was moved to [another location](../macos_saas_runner.md). - -<!-- This redirect file can be deleted after <2023-09-05>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
\ No newline at end of file diff --git a/doc/ci/runners/saas/macos/environment.md b/doc/ci/runners/saas/macos/environment.md deleted file mode 100644 index 7e70e984c7c..00000000000 --- a/doc/ci/runners/saas/macos/environment.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../macos_saas_runner.md' -remove_date: '2023-09-05' ---- - -This document was moved to [another location](../macos_saas_runner.md). - -<!-- This redirect file can be deleted after <2023-09-05>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
\ No newline at end of file diff --git a/doc/ci/runners/saas/macos_saas_runner.md b/doc/ci/runners/saas/macos_saas_runner.md index 44f99ed6ccc..1445ae58bd4 100644 --- a/doc/ci/runners/saas/macos_saas_runner.md +++ b/doc/ci/runners/saas/macos_saas_runner.md @@ -4,7 +4,7 @@ group: Runner info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# SaaS runners on macOS (Beta) **(PREMIUM SAAS)** +# SaaS runners on macOS **(PREMIUM SAAS BETA)** SaaS runners on macOS are in [Beta](../../../policy/experiment-beta-support.md#beta) for open source programs and customers in Premium and Ultimate plans. @@ -38,28 +38,30 @@ in your `.gitlab-ci.yml` file. Each image runs a specific version of macOS and Xcode. -| VM image | Status | -|---------------------------|---------------| -| `macos-12-xcode-13` | `maintenance` | -| `macos-12-xcode-14` | `maintenance` | -| (none, awaiting macOS 13) | `beta` | +| VM image | Status | +|----------------------------|--------| +| `macos-12-xcode-13` | `GA` | +| `macos-12-xcode-14` | `GA` | +| `macos-13-xcode-14` | `Beta` | -NOTE: -If your job requires tooling or dependencies not available in our available images, those can only be installed in the job execution. +## Image update policy for macOS -## Image update policy +macOS and Xcode follow a yearly release cadence, during which GitLab increments its versions synchronously. GitLab typically supports multiple versions of preinstalled tools. For more information, see +a [full list of preinstalled software](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/job-images/-/tree/main/toolchain). -GitLab expects to release new images based on this cadence: +GitLab provides `stable` and `latest` macOS images that follow different update patterns: -macOS updates: +- **Stable image:** The `stable` images and installed components are updated every release. Images without the `:latest` prefix are considered stable images. +- **Latest image:** The `latest` images are typically updated on a weekly cadence and use a `:latest` prefix in the image name. Using the `latest` image results in more regularly updated components and shorter update times for Homebrew or asdf. The `latest` images are used to test software components before releasing the components to the `stable` images. +By definition, the `latest` images are always Beta. +A `latest` image is not available. -- **For new OS versions:** When Apple releases a new macOS version to developers (like macOS `12`), GitLab will plan to release an image based on the OS within the next 30 business days. The image is considered `beta` and the contents of the image (including tool versions) are subject to change until the first patch release (`12.1`). The long-term name will not include `beta` (for example, `macos-12-xcode-13`), so customers are moved automatically out of beta over time. GitLab will try to minimize breaking changes between the first two minor versions but makes no guarantees. Tooling often gets critical bug fixes after the first public release of an OS version. +### Image release process -- **After the first patch release (`12.1`):** - - The image moves to `maintenance` mode. The tools GitLab builds into the image with Homebrew and asdf are frozen. GitLab continues making Xcode updates, security updates, and any non-breaking changes deemed necessary. - - The image for the previous OS version (`11`) moves to `frozen` mode. GitLab then does only unavoidable changes: security updates, runner version upgrades, and setting the production password. +When Apple releases a new macOS version, GitLab releases both `stable` and `latest` images based on the OS in the next release. Both images are Beta. -Both macOS and Xcode follow a yearly release cadence. As time goes on, GitLab increments their versions synchronously (meaning we build macOS 11 with Xcode 12, macOS 12 with Xcode 13, and so on). +With the release of the first patch to macOS, the `stable` image becomes Generally Available (GA). +As only two GA images are supported at a time, the prior OS version becomes deprecated and is deleted after three months in accordance with the [supported image lifecycle](../index.md#supported-image-lifecycle). ## Example `.gitlab-ci.yml` file diff --git a/doc/ci/runners/saas/windows_saas_runner.md b/doc/ci/runners/saas/windows_saas_runner.md index 8ec44b8c275..108388f22c8 100644 --- a/doc/ci/runners/saas/windows_saas_runner.md +++ b/doc/ci/runners/saas/windows_saas_runner.md @@ -4,7 +4,7 @@ group: Runner info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# SaaS runners on Windows (Beta) **(FREE SAAS)** +# SaaS runners on Windows **(FREE SAAS BETA)** SaaS runner on Windows autoscale by launching virtual machines on the Google Cloud Platform. This solution uses an @@ -35,7 +35,7 @@ You can execute your job in one of the following Windows versions: | Version tag | Status | |----------------|---------------| -| `windows-1809` | `maintenance` | +| `windows-1809` | `Beta` | You can find a full list of available pre-installed software in the [pre-installed software documentation](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/gcp/windows-containers/blob/main/cookbooks/preinstalled-software/README.md). @@ -81,16 +81,15 @@ test: - echo "running scripts in the test job" ``` -## Limitations and known issues +## Known issues -- All the limitations mentioned in our [Beta definition](../../../policy/experiment-beta-support.md#beta). -- The average provisioning time for a new Windows VM is 5 minutes. - This means that you may notice slower build start times - on the Windows runner fleet during the beta. In a future - release we intend to update the autoscaler to enable - the pre-provisioning of virtual machines. This is intended to significantly reduce - the time it takes to provision a VM on the Windows fleet. You can - follow along in the [related issue](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/-/issues/32). +- For more information about support for Beta features, see [Beta](../../../policy/experiment-beta-support.md#beta). +- The average provisioning time for a new Windows virtual machine (VM) is five minutes, so + you might notice slower start times for builds on the Windows runner + fleet during the Beta. Updating the autoscaler to enable the pre-provisioning + of virtual machines is proposed in a future release. This update is intended to + significantly reduce the time it takes to provision a VM on the Windows fleet. + For more information, see [issue 32](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/-/issues/32). - The Windows runner fleet may be unavailable occasionally for maintenance or updates. - The job may stay in a pending state for longer than the diff --git a/doc/ci/secrets/id_token_authentication.md b/doc/ci/secrets/id_token_authentication.md index 22a260e4bb6..697346474f8 100644 --- a/doc/ci/secrets/id_token_authentication.md +++ b/doc/ci/secrets/id_token_authentication.md @@ -193,7 +193,7 @@ ID token authentication is now always available, and JSON Web Token access is al To enable automatic ID token authentication: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Token Access**. 1. Turn on the **Limit JSON Web Token (JWT) access** toggle. diff --git a/doc/ci/secure_files/index.md b/doc/ci/secure_files/index.md index 37c453a5b9d..a666e0aca7b 100644 --- a/doc/ci/secure_files/index.md +++ b/doc/ci/secure_files/index.md @@ -29,7 +29,7 @@ tool. To add a secure file to a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand the **Secure Files** section. 1. Select **Upload File**. diff --git a/doc/ci/services/index.md b/doc/ci/services/index.md index f92c50fcf3f..a2ffd095de3 100644 --- a/doc/ci/services/index.md +++ b/doc/ci/services/index.md @@ -493,5 +493,10 @@ Docker privileged mode applies to services. This means that the service image co ## Shared /builds directory -Services can access files from the build because all services have the job -directory mounted as a volume under `/builds`. +The build directory is mounted as a volume under `/builds` and is shared +between the job and services. The job checks the project out into +`/builds/$CI_PROJECT_PATH` after the services are running. As a result, if your +service needs files from the project or, for example, wants to put files there +to serve as artifacts, it may need to wait for that directory to exist and +have `$CI_COMMIT_SHA` checked out. Any changes made before the job finishes its +checkout process are removed by the checkout process. diff --git a/doc/ci/test_cases/index.md b/doc/ci/test_cases/index.md index 0bc9ae7776e..d9dcbca0825 100644 --- a/doc/ci/test_cases/index.md +++ b/doc/ci/test_cases/index.md @@ -27,7 +27,7 @@ Prerequisite: To create a test case in a GitLab project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Build > Test cases**. 1. Select **New test case**. You are taken to the new test case form. Here you can enter the new case's title, [description](../../user/markdown.md), attach a file, and assign [labels](../../user/project/labels.md). @@ -45,7 +45,7 @@ Prerequisites: To view a test case: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Build > Test cases**. 1. Select the title of the test case you want to view. You are taken to the test case page. @@ -80,7 +80,7 @@ To archive a test case, on the test case's page, select **Archive test case**. To view archived test cases: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Build > Test cases**. 1. Select **Archived**. diff --git a/doc/ci/testing/code_coverage.md b/doc/ci/testing/code_coverage.md index 90a07314083..fb846f52a72 100644 --- a/doc/ci/testing/code_coverage.md +++ b/doc/ci/testing/code_coverage.md @@ -72,7 +72,7 @@ Use this regex for commonly used test tools. To see the evolution of your project code coverage over time, you can view a graph or download a CSV file with this data. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Analyze > Repository analytics**. The historic data for each job is listed in the dropdown list above the graph. diff --git a/doc/ci/testing/load_performance_testing.md b/doc/ci/testing/load_performance_testing.md index 549aa10287e..6b16f15a9d9 100644 --- a/doc/ci/testing/load_performance_testing.md +++ b/doc/ci/testing/load_performance_testing.md @@ -59,7 +59,7 @@ Configuring your Load Performance Testing job can be broken down into several di ### Determine the test parameters -The first thing you need to do is determine the [type of load test](https://k6.io/docs/test-types/introduction) +The first thing you need to do is determine the [type of load test](https://k6.io/docs/test-types/load-test-types/) you want to run, and how you want it to run (for example, the number of users, throughput, and so on). Refer to the [k6 docs](https://k6.io/docs/), especially the [k6 testing guides](https://k6.io/docs/testing-guides), @@ -79,7 +79,7 @@ We strongly recommend [not running these tests against a production environment] ### Write the load performance test After the environment is prepared, you can write the k6 test itself. k6 is a flexible -tool and can be used to run [many kinds of performance tests](https://k6.io/docs/test-types/introduction). +tool and can be used to run [many kinds of performance tests](https://k6.io/docs/test-types/load-test-types/). Refer to the [k6 documentation](https://k6.io/docs/) for detailed information on how to write tests. ### Configure the test in GitLab CI/CD @@ -151,7 +151,7 @@ The CI/CD YAML configuration example above works for testing against static envi but it can be extended to work with [review apps](../review_apps/index.md) or [dynamic environments](../environments/index.md) with a few extra steps. -The best approach is to capture the dynamic URL in a [`.env` file](https://docs.docker.com/compose/env-file/) +The best approach is to capture the dynamic URL in a [`.env` file](https://docs.docker.com/compose/environment-variables/env-file/) as a job artifact to be shared, then use a custom CI/CD variable we've provided named `K6_DOCKER_OPTIONS` to configure the k6 Docker container to use the file. With this, k6 can then use any environment variables from the `.env` file in scripts using standard JavaScript, diff --git a/doc/ci/triggers/index.md b/doc/ci/triggers/index.md index 506f6fb2106..698118f457f 100644 --- a/doc/ci/triggers/index.md +++ b/doc/ci/triggers/index.md @@ -26,7 +26,7 @@ Prerequisite: To create a trigger token: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Pipeline trigger tokens**. 1. Select **Add new token** @@ -154,7 +154,7 @@ users with the Owner and Maintainer role can view the values. To revoke a pipeline trigger token: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Pipeline triggers**. 1. To the left of the trigger token you want to revoke, select **Revoke** (**{remove}**). diff --git a/doc/ci/troubleshooting.md b/doc/ci/troubleshooting.md index 3d5bcc64889..e0b8c6213de 100644 --- a/doc/ci/troubleshooting.md +++ b/doc/ci/troubleshooting.md @@ -210,6 +210,42 @@ To illustrate its life cycle: 1. The runner fetches the persistent pipeline ref and gets source code from the checkout-SHA. 1. When the pipeline finishes, its persistent ref is cleaned up in a background process. +### `get_sources` job section fails because of an HTTP/2 problem + +Sometimes, jobs fail with the following cURL error: + +```plaintext +++ git -c 'http.userAgent=gitlab-runner <version>' fetch origin +refs/pipelines/<id>:refs/pipelines/<id> ... +error: RPC failed; curl 16 HTTP/2 send again with decreased length +fatal: ... +``` + +You can work around this problem by configuring Git and `libcurl` to +[use HTTP/1.1](https://git-scm.com/docs/git-config#Documentation/git-config.txt-httpversion). +The configuration can be added to: + +- A job's [`pre_get_sources_script`](yaml/index.md#hookspre_get_sources_script): + + ```yaml + job_name: + hooks: + pre_get_sources_script: + - git config --local http.version "HTTP/1.1" + ``` + +- The [runner's `config.toml`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html) + with [Git configuration environment variables](https://git-scm.com/docs/git-config#ENVIRONMENT): + + ```toml + [[runners]] + ... + environment = [ + "GIT_CONFIG_COUNT=1", + "GIT_CONFIG_KEY_1=http.version", + "GIT_CONFIG_VALUE_1=HTTP/1.1" + ] + ``` + ### Merge request pipeline messages The merge request pipeline widget shows information about the pipeline status in @@ -440,9 +476,9 @@ This flag reduces system resource usage on the `jobs/request` endpoint. When enabled, jobs created in the last hour can run in projects which are out of quota. Earlier jobs are already canceled by a periodic background worker (`StuckCiJobsWorker`). -## CI/CD troubleshooting rails console commands +## CI/CD troubleshooting Rails console commands -The following commands are run in the [rails console](../administration/operations/rails_console.md#starting-a-rails-console-session). +The following commands are run in the [Rails console](../administration/operations/rails_console.md#starting-a-rails-console-session). WARNING: Any command that changes data directly could be damaging if not run correctly, or under the right conditions. diff --git a/doc/ci/variables/index.md b/doc/ci/variables/index.md index 6280c9080ab..975157ff917 100644 --- a/doc/ci/variables/index.md +++ b/doc/ci/variables/index.md @@ -170,7 +170,7 @@ To add a group variable: - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. - **Value**: No limitations. - **Type**: `Variable` (default) or [`File`](#use-file-type-cicd-variables). - - **Environment scope** Optional. `All`, or specific [environments](../environments/index.md#limit-the-environment-scope-of-a-cicd-variable). **(PREMIUM)** + - **Environment scope** Optional. `All`, or specific [environments](../environments/index.md#limit-the-environment-scope-of-a-cicd-variable). **(PREMIUM ALL)** - **Protect variable** Optional. If selected, the variable is only available in pipelines that run on protected branches or tags. - **Mask variable** Optional. If selected, the variable's **Value** is masked @@ -194,7 +194,7 @@ Prerequisite: To add an instance variable: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > CI/CD** and expand the **Variables** section. 1. Select **Add variable** and fill in the details: @@ -371,7 +371,7 @@ For example: ```yaml variables: - SITE_URL: "https://example.gitlab.com" + SITE_URL: "https://gitlab.example.com" job: script: diff --git a/doc/ci/variables/predefined_variables.md b/doc/ci/variables/predefined_variables.md index 6acb254e76f..b1a0fca9069 100644 --- a/doc/ci/variables/predefined_variables.md +++ b/doc/ci/variables/predefined_variables.md @@ -68,9 +68,9 @@ as it can cause the pipeline to behave unexpectedly. | `CI_HAS_OPEN_REQUIREMENTS` | 13.1 | all | Only available if the pipeline's project has an open [requirement](../../user/project/requirements/index.md). `true` when available. | | `CI_JOB_ID` | 9.0 | all | The internal ID of the job, unique across all jobs in the GitLab instance. | | `CI_JOB_IMAGE` | 12.9 | 12.9 | The name of the Docker image running the job. | -| `CI_JOB_JWT` (Deprecated) | 12.10 | all | A RS256 JSON web token to authenticate with third party systems that support JWT authentication, for example [HashiCorp's Vault](../secrets/index.md). [Deprecated in GitLab 15.9](../../update/deprecations.md#old-versions-of-json-web-tokens-are-deprecated) and scheduled to be removed in GitLab 16.5. Use [ID tokens](../yaml/index.md#id_tokens) instead. | -| `CI_JOB_JWT_V1` (Deprecated) | 14.6 | all | The same value as `CI_JOB_JWT`. [Deprecated in GitLab 15.9](../../update/deprecations.md#old-versions-of-json-web-tokens-are-deprecated) and scheduled to be removed in GitLab 16.5. Use [ID tokens](../yaml/index.md#id_tokens) instead. | -| `CI_JOB_JWT_V2` (Deprecated) | 14.6 | all | A newly formatted RS256 JSON web token to increase compatibility. Similar to `CI_JOB_JWT`, except the issuer (`iss`) claim is changed from `gitlab.com` to `https://gitlab.com`, `sub` has changed from `job_id` to a string that contains the project path, and an `aud` claim is added. The `aud` field is a constant value. Trusting JWTs in multiple relying parties can lead to [one RP sending a JWT to another one and acting maliciously as a job](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/72555#note_769112331). [Deprecated in GitLab 15.9](../../update/deprecations.md#old-versions-of-json-web-tokens-are-deprecated) and scheduled to be removed in GitLab 16.5. Use [ID tokens](../yaml/index.md#id_tokens) instead. | +| `CI_JOB_JWT` (Deprecated) | 12.10 | all | A RS256 JSON web token to authenticate with third party systems that support JWT authentication, for example [HashiCorp's Vault](../secrets/index.md). [Deprecated in GitLab 15.9](../../update/deprecations.md#old-versions-of-json-web-tokens-are-deprecated) and scheduled to be removed in GitLab 17.0. Use [ID tokens](../yaml/index.md#id_tokens) instead. | +| `CI_JOB_JWT_V1` (Deprecated) | 14.6 | all | The same value as `CI_JOB_JWT`. [Deprecated in GitLab 15.9](../../update/deprecations.md#old-versions-of-json-web-tokens-are-deprecated) and scheduled to be removed in GitLab 17.0. Use [ID tokens](../yaml/index.md#id_tokens) instead. | +| `CI_JOB_JWT_V2` (Deprecated) | 14.6 | all | A newly formatted RS256 JSON web token to increase compatibility. Similar to `CI_JOB_JWT`, except the issuer (`iss`) claim is changed from `gitlab.com` to `https://gitlab.com`, `sub` has changed from `job_id` to a string that contains the project path, and an `aud` claim is added. The `aud` field is a constant value. Trusting JWTs in multiple relying parties can lead to [one RP sending a JWT to another one and acting maliciously as a job](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/72555#note_769112331). [Deprecated in GitLab 15.9](../../update/deprecations.md#old-versions-of-json-web-tokens-are-deprecated) and scheduled to be removed in GitLab 17.0. Use [ID tokens](../yaml/index.md#id_tokens) instead. | | `CI_JOB_MANUAL` | 8.12 | all | Only available if the job was started manually. `true` when available. | | `CI_JOB_NAME` | 9.0 | 0.5 | The name of the job. | | `CI_JOB_NAME_SLUG` | 15.4 | all | `CI_JOB_NAME_SLUG` in lowercase, shortened to 63 bytes, and with everything except `0-9` and `a-z` replaced with `-`. No leading / trailing `-`. Use in paths. | @@ -165,13 +165,15 @@ These variables are available when: | `CI_MERGE_REQUEST_PROJECT_PATH` | 11.6 | all | The path of the project of the merge request. For example `namespace/awesome-project`. | | `CI_MERGE_REQUEST_PROJECT_URL` | 11.6 | all | The URL of the project of the merge request. For example, `http://192.168.10.15:3000/namespace/awesome-project`. | | `CI_MERGE_REQUEST_REF_PATH` | 11.6 | all | The ref path of the merge request. For example, `refs/merge-requests/1/head`. | +| `CI_MERGE_REQUEST_SQUASH_ON_MERGE` | 16.4 | all | `true` when the [squash on merge](../../user/project/merge_requests/squash_and_merge.md) option is set. | | `CI_MERGE_REQUEST_SOURCE_BRANCH_NAME` | 11.6 | all | The source branch name of the merge request. | +| `CI_MERGE_REQUEST_SOURCE_BRANCH_PROTECTED` | 16.4 | all | `true` when the source branch of the merge request is [protected](../../user/project/protected_branches.md). | | `CI_MERGE_REQUEST_SOURCE_BRANCH_SHA` | 11.9 | all | The HEAD SHA of the source branch of the merge request. The variable is empty in merge request pipelines. The SHA is present only in [merged results pipelines](../pipelines/merged_results_pipelines.md). | | `CI_MERGE_REQUEST_SOURCE_PROJECT_ID` | 11.6 | all | The ID of the source project of the merge request. | | `CI_MERGE_REQUEST_SOURCE_PROJECT_PATH` | 11.6 | all | The path of the source project of the merge request. | | `CI_MERGE_REQUEST_SOURCE_PROJECT_URL` | 11.6 | all | The URL of the source project of the merge request. | | `CI_MERGE_REQUEST_TARGET_BRANCH_NAME` | 11.6 | all | The target branch name of the merge request. | -| `CI_MERGE_REQUEST_TARGET_BRANCH_PROTECTED` | 15.2 | all | The protection status for the target branch of the merge request. | +| `CI_MERGE_REQUEST_TARGET_BRANCH_PROTECTED` | 15.2 | all | `true` when the target branch of the merge request is [protected](../../user/project/protected_branches.md). | | `CI_MERGE_REQUEST_TARGET_BRANCH_SHA` | 11.9 | all | The HEAD SHA of the target branch of the merge request. The variable is empty in merge request pipelines. The SHA is present only in [merged results pipelines](../pipelines/merged_results_pipelines.md). | | `CI_MERGE_REQUEST_TITLE` | 11.9 | all | The title of the merge request. | | `CI_MERGE_REQUEST_EVENT_TYPE` | 12.3 | all | The event type of the merge request. Can be `detached`, `merged_result` or `merge_train`. | diff --git a/doc/ci/variables/where_variables_can_be_used.md b/doc/ci/variables/where_variables_can_be_used.md index 52edeb67f0e..d25f9801f5b 100644 --- a/doc/ci/variables/where_variables_can_be_used.md +++ b/doc/ci/variables/where_variables_can_be_used.md @@ -22,6 +22,8 @@ There are two places defined variables can be used. On the: ### `.gitlab-ci.yml` file +> Support for `CI_ENVIRONMENT_*` variables except `CI_ENVIRONMENT_SLUG` [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/128694) in GitLab 16.4. + | Definition | Can be expanded? | Expansion place | Description | |:----------------------------------------------------------------------|:-----------------|:-----------------------|:------------| | [`after_script`](../yaml/index.md#after_script) | yes | Script execution shell | The variable expansion is made by the [execution shell environment](#execution-shell-environment). | @@ -32,15 +34,15 @@ There are two places defined variables can be used. On the: | [`environment:name`](../yaml/index.md#environmentname) | yes | GitLab | Similar to `environment:url`, but the variables expansion doesn't support the following:<br/><br/>- `CI_ENVIRONMENT_*` variables.<br/>- [Persisted variables](#persisted-variables). | | [`environment:url`](../yaml/index.md#environmenturl) | yes | GitLab | The variable expansion is made by the [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism) in GitLab.<br/><br/>Supported are all variables defined for a job (project/group variables, variables from `.gitlab-ci.yml`, variables from triggers, variables from pipeline schedules).<br/><br/>Not supported are variables defined in the GitLab Runner `config.toml` and variables created in the job's `script`. | | [`environment:auto_stop_in`](../yaml/index.md#environmentauto_stop_in)| yes | GitLab | The variable expansion is made by the [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism) in GitLab.<br/><br/> The value of the variable being substituted should be a period of time in a human readable natural language form. See [possible inputs](../yaml/index.md#environmentauto_stop_in) for more information.| -| [`except:variables`](../yaml/index.md#onlyvariables--exceptvariables) | no | Not applicable | The variable must be in the form of `$variable`. Not supported are the following:<br/><br/>- `CI_ENVIRONMENT_*` variables, except `CI_ENVIRONMENT_NAME` which is supported.<br/>- [Persisted variables](#persisted-variables). | +| [`except:variables`](../yaml/index.md#onlyvariables--exceptvariables) | no | Not applicable | The variable must be in the form of `$variable`. Not supported are the following:<br/><br/>- `CI_ENVIRONMENT_SLUG` variable.<br/>- [Persisted variables](#persisted-variables). | | [`id_tokens:aud`](../yaml/index.md#id_tokens) | yes | GitLab | The variable expansion is made by the [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism) in GitLab. Variable expansion [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/414293) in GitLab 16.1. | | [`image`](../yaml/index.md#image) | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism). | | [`include`](../yaml/index.md#include) | yes | GitLab | The variable expansion is made by the [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism) in GitLab. <br/><br/>See [Use variables with include](../yaml/includes.md#use-variables-with-include) for more information on supported variables. | -| [`only:variables`](../yaml/index.md#onlyvariables--exceptvariables) | no | Not applicable | The variable must be in the form of `$variable`. Not supported are the following:<br/><br/>- `CI_ENVIRONMENT_*` variables, except `CI_ENVIRONMENT_NAME` which is supported.<br/>- [Persisted variables](#persisted-variables). | +| [`only:variables`](../yaml/index.md#onlyvariables--exceptvariables) | no | Not applicable | The variable must be in the form of `$variable`. Not supported are the following:<br/><br/>- `CI_ENVIRONMENT_SLUG` variable.<br/>- [Persisted variables](#persisted-variables). | | [`resource_group`](../yaml/index.md#resource_group) | yes | GitLab | Similar to `environment:url`, but the variables expansion doesn't support the following:<br/>- `CI_ENVIRONMENT_URL`<br/>- [Persisted variables](#persisted-variables). | | [`rules:changes`](../yaml/index.md#ruleschanges) | yes | GitLab | The variable expansion is made by the [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism) in GitLab. | | [`rules:exists`](../yaml/index.md#rulesexists) | yes | GitLab | The variable expansion is made by the [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism) in GitLab. | -| [`rules:if`](../yaml/index.md#rulesif) | no | Not applicable | The variable must be in the form of `$variable`. Not supported are the following:<br/><br/>- `CI_ENVIRONMENT_*` variables, except `CI_ENVIRONMENT_NAME` which is supported.<br/>- [Persisted variables](#persisted-variables). | +| [`rules:if`](../yaml/index.md#rulesif) | no | Not applicable | The variable must be in the form of `$variable`. Not supported are the following:<br/><br/>- `CI_ENVIRONMENT_SLUG` variable.<br/>- [Persisted variables](#persisted-variables). | | [`script`](../yaml/index.md#script) | yes | Script execution shell | The variable expansion is made by the [execution shell environment](#execution-shell-environment). | | [`services:name`](../yaml/index.md#services) | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism). | | [`services`](../yaml/index.md#services) | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism). | diff --git a/doc/ci/yaml/artifacts_reports.md b/doc/ci/yaml/artifacts_reports.md index fa7e941ffe5..e931a8b3b4e 100644 --- a/doc/ci/yaml/artifacts_reports.md +++ b/doc/ci/yaml/artifacts_reports.md @@ -40,6 +40,55 @@ GitLab can display the results of one or more reports in the merge request For more information, see [Accessibility testing](../testing/accessibility_testing.md). +## `artifacts:reports:annotations` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/38337) in GitLab 16.3. + +The `annotations` report is used to attach auxiliary data to a job. + +An annotations report is a JSON file with annotation sections. Each annotation +section can have any desired name and can have any number of annotations of the +same or differing types. + +Each annotation is a single key (the annotation type), containing the subkeys with +the data for that annotation. + +### Annotation types + +#### `external_link` + +An `external_link` annotation can be attached to a job to add a link to the job +output page. The value of an `external_link` annotation is an object with the +following keys: + +| Key | Description | +|---------|----------------------------------------------------| +| `label` | The human-readable label associated with the link. | +| `url` | The URL pointed to by the link. | + +### Example report + +The following is an example of what a job annotations report might look like: + +```json +{ + "my_annotation_section_1": [ + { + "external_link": { + "label": "URL 1", + "url": "https://url1.example.com/" + } + }, + { + "external_link": { + "label": "URL 2", + "url": "https://url2.example.com/" + } + } + ] +} +``` + ## `artifacts:reports:api_fuzzing` **(ULTIMATE ALL)** > - Introduced in GitLab 13.4. @@ -95,7 +144,7 @@ GitLab can display the results of coverage report in the merge request ## `artifacts:reports:codequality` > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212499) to GitLab Free in 13.2. -> - [Added support for multiple reports in diff annotations and full pipeline report](https://gitlab.com/gitlab-org/gitlab/-/issues/9014) in 15.7. +> - Support for multiple reports in diff annotations and full pipeline report [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9014) in 15.7. The `codequality` report collects [code quality issues](../testing/code_quality.md). The collected code quality report uploads to GitLab as an artifact. @@ -313,7 +362,7 @@ The collected Secret Detection report is uploaded to GitLab. GitLab can display the results of one or more reports in: - The merge request [secret scanning widget](../../user/application_security/secret_detection/index.md). -- The [pipeline **Security** tab](../../user/application_security/index.md#view-security-scan-information-in-the-pipeline-security-tab). +- The [pipeline security tab](../../user/application_security/index.md#pipeline-security-tab). - The [security dashboard](../../user/application_security/security_dashboard/index.md). ## `artifacts:reports:terraform` @@ -325,6 +374,6 @@ The `terraform` report obtains a Terraform `tfplan.json` file. [JQ processing re The collected Terraform plan report uploads to GitLab as an artifact. GitLab can display the results of one or more reports in the merge request -[terraform widget](../../user/infrastructure/iac/mr_integration.md#output-terraform-plan-information-into-a-merge-request). +[Terraform widget](../../user/infrastructure/iac/mr_integration.md#output-terraform-plan-information-into-a-merge-request). For more information, see [Output `terraform plan` information into a merge request](../../user/infrastructure/iac/mr_integration.md). diff --git a/doc/ci/yaml/includes.md b/doc/ci/yaml/includes.md index 79eb42fd781..f0e5a475838 100644 --- a/doc/ci/yaml/includes.md +++ b/doc/ci/yaml/includes.md @@ -501,6 +501,44 @@ In this example, GitLab checks for the existence of `test-file.yml` in `my-group not the current project. Follow [issue 386040](https://gitlab.com/gitlab-org/gitlab/-/issues/386040) for information about work to improve this behavior. +### `include` with `rules:changes` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/342209) in GitLab 16.4. + +Use [`rules:changes`](index.md#ruleschanges) to conditionally include other configuration files +based on changed files. For example: + +```yaml +include: + - local: builds1.yml + rules: + - changes: + - Dockerfile + - local: builds2.yml + rules: + - changes: + paths: + - Dockerfile + compare_to: 'refs/heads/branch1' + when: always + - local: builds3.yml + rules: + - if: $CI_PIPELINE_SOURCE == "merge_request_event" + changes: + paths: + - Dockerfile + +test: + stage: test + script: exit 0 +``` + +In this example: + +- `builds1.yml` is included when `Dockerfile` has changed. +- `builds2.yml` is included when `Dockerfile` has changed relative to `refs/heads/branch1`. +- `builds3.yml` is included when `Dockerfile` has changed and the pipeline source is a merge request event. + ## Use `include:local` with wildcard file paths > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25921) in GitLab 13.11. @@ -528,144 +566,6 @@ When the pipeline runs, GitLab: include: 'configs/**/*.yml' ``` -## Define inputs for configuration added with `include` (Beta) - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/391331) in GitLab 15.11 as a Beta feature. - -FLAG: -`spec` and `with` are experimental [Open Beta features](../../policy/experiment-beta-support.md#beta) -and subject to change without notice. - -### Define input parameters with `spec:inputs` - -Use `spec:inputs` to define input parameters for CI/CD configuration intended to be added -to a pipeline with `include`. Use [`include:inputs`](#set-input-parameter-values-with-includeinputs) -to define the values to use when the pipeline runs. - -The specs must be declared at the top of the configuration file, in a header section. -Separate the header from the rest of the configuration with `---`. - -Use the interpolation format `$[[ input.input-id ]]` to reference the values outside of the header section. -The inputs are evaluated and interpolated once, when the configuration is fetched -during pipeline creation, but before the configuration is merged with the contents of the `.gitlab-ci.yml`. - -```yaml -spec: - inputs: - environment: - job-stage: ---- - -scan-website: - stage: $[[ inputs.job-stage ]] - script: ./scan-website $[[ inputs.environment ]] -``` - -When using `spec:inputs`: - -- Defined inputs are mandatory by default. -- Inputs can be made optional by specifying a `default`. Use `default: null` to have no default value. -- A string containing an interpolation block must not exceed 1 MB. -- The string inside an interpolation block must not exceed 1 KB. - -For example, a `custom_configuration.yml`: - -```yaml -spec: - inputs: - website: - user: - default: 'test-user' - flags: - default: null ---- - -# The pipeline configuration would follow... -``` - -In this example: - -- `website` is mandatory and must be defined. -- `user` is optional. If not defined, the value is `test-user`. -- `flags` is optional. If not defined, it has no value. - -### Specify functions to manipulate input values - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/409462) in GitLab 16.3. - -You can specify predefined functions in the interpolation block to manipulate the input value. -The format supported is the following: - -```yaml -$[[ input.input-id | <function1> | <function2> | ... <functionN> ]] -``` - -Details: - -- Only [predefined interpolation functions](#predefined-interpolation-functions) are permitted. -- A maximum of 3 functions may be specified in a single interpolation block. -- The functions are executed in the sequence they are specified. - -```yaml -spec: - inputs: - test: - default: '0123456789' ---- - -test-job: - script: echo $[[ inputs.test | truncate(1,3) ]] -``` - -In this example: - -- The function [`truncate`](#truncate) applies to the value of `inputs.test`. -- Assuming the value of `inputs.test` is `0123456789`, then the output of `script` would be `echo 123`. - -### Predefined interpolation functions - -#### Truncate - -Use `truncate` to shorten the interpolated value. For example: - -- `truncate(<offset>,<length>)` - -| Name | Type | Description | -| ---- | ---- | ----------- | -| `offset` | Integer | Number of characters to offset by. | -| `length` | Integer | Number of characters to return after the offset. | - -Example: - -```yaml -$[[ inputs.test | truncate(3,5) ]] -``` - -Assuming the value of `inputs.test` is `0123456789`, then the output would be `34567`. - -### Set input parameter values with `include:inputs` - -> `include:with` [renamed to `include:inputs`](https://gitlab.com/gitlab-org/gitlab/-/issues/406780) in GitLab 16.0. - -Use `include:inputs` to set the values for the parameters when the included configuration -is added to the pipeline. - -For example, to include a `custom_configuration.yml` that has the same specs -as the [example above](#define-input-parameters-with-specinputs): - -```yaml -include: - - local: 'custom_configuration.yml' - inputs: - website: "My website" -``` - -In this example: - -- `website` has a value of `My website` for the included configuration. -- `user` has a value of `test-user`, because that is the default when not specified. -- `flags` has no value, because it is optional and has no default when not specified. - ## Troubleshooting ### `Maximum of 150 nested includes are allowed!` error diff --git a/doc/ci/yaml/index.md b/doc/ci/yaml/index.md index c4f7e6d6e01..6275d4293ea 100644 --- a/doc/ci/yaml/index.md +++ b/doc/ci/yaml/index.md @@ -9,9 +9,11 @@ type: reference This document lists the configuration options for your GitLab `.gitlab-ci.yml` file. -- For a quick introduction to GitLab CI/CD, follow the [quick start guide](../quick_start/index.md). -- For a collection of examples, see [GitLab CI/CD Examples](../examples/index.md). -- To view a large `.gitlab-ci.yml` file used in an enterprise, see the [`.gitlab-ci.yml` file for `gitlab`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). +- For a collection of examples, see [GitLab CI/CD examples](../examples/index.md). +- To view a large `.gitlab-ci.yml` file used in an enterprise, see the + [`.gitlab-ci.yml` file for `gitlab`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). +- To create your own `.gitlab-ci.yml` file, try a tutorial that demonstrates a + [simple](../quick_start/index.md) or [complex](../quick_start/tutorial.md) pipeline. When you are editing your `.gitlab-ci.yml` file, you can validate it with the [CI Lint](../lint.md) tool. @@ -75,8 +77,11 @@ or import additional pipeline configuration. ### `default` -You can set global defaults for some keywords. Jobs that do not define one or more -of the listed keywords use the value defined in the `default` section. +> Support for `id_tokens` [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/419750) in GitLab 16.4. + +You can set global defaults for some keywords. Each default keyword is copied to every job +that doesn't already have it defined. If the job already has a keyword defined, that default +is not used. **Keyword type**: Global keyword. @@ -87,6 +92,7 @@ of the listed keywords use the value defined in the `default` section. - [`before_script`](#before_script) - [`cache`](#cache) - [`hooks`](#hooks) +- [`id_tokens`](#id_tokens) - [`image`](#image) - [`interruptible`](#interruptible) - [`retry`](#retry) @@ -99,6 +105,7 @@ of the listed keywords use the value defined in the `default` section. ```yaml default: image: ruby:3.0 + retry: 2 rspec: script: bundle exec rspec @@ -108,16 +115,17 @@ rspec 2.7: script: bundle exec rspec ``` -In this example, `ruby:3.0` is the default `image` value for all jobs in the pipeline. -The `rspec 2.7` job does not use the default, because it overrides the default with -a job-specific `image` section: +In this example: + +- `image: ruby:3.0` and `retry: 2` are the default keywords for all jobs in the pipeline. +- The `rspec` job does not have `image` or `retry` defined, so it uses the defaults of + `image: ruby:3.0` and `retry: 2`. +- The `rspec 2.7` job does not have `retry` defined, but it does have `image` explictly defined. + It uses the default `retry: 2`, but ignores the default `image` and uses the `image: ruby:2.7` + defined in the job. **Additional details**: -- When the pipeline is created, each default is copied to all jobs that don't have - that keyword defined. -- If a job already has one of the keywords configured, the configuration in the job - takes precedence and is not replaced by the default. - Control inheritance of default keywords in jobs with [`inherit:default`](#inheritdefault). ### `include` @@ -267,6 +275,11 @@ include: - When you include a YAML file from another private project, the user running the pipeline must be a member of both projects and have the appropriate permissions to run pipelines. A `not found or access denied` error may be displayed if the user does not have access to any of the included files. +- Be careful when including another project's CI/CD configuration file. No pipelines or notifications trigger when CI/CD configuration files change. + From a security perspective, this is similar to pulling a third-party dependency. For the `ref`, consider: + - Using a specific SHA hash, which should be the most stable option. + - Applying both [protected branch](../../user/project/protected_branches.md) and [protected tag](../../user/project/protected_tags.md#prevent-tag-creation-with-the-same-name-as-branches) rules to + the `ref` in the other project. Protected tags and branches are more likely to pass through change management before changing. #### `include:remote` @@ -293,9 +306,11 @@ include: - All [nested includes](includes.md#use-nested-includes) are executed without context as a public user, so you can only include public projects or templates. No variables are available in the `include` section of nested includes. -- Be careful when including a remote CI/CD configuration file. No pipelines or notifications - trigger when external CI/CD configuration files change. From a security perspective, - this is similar to pulling a third-party dependency. +- Be careful when including another project's CI/CD configuration file. No pipelines or notifications trigger + when the other project's files change. From a security perspective, this is similar to + pulling a third-party dependency. If you link to another GitLab project you own, consider the use of both + [protected branches](../../user/project/protected_branches.md) and [protected tags](../../user/project/protected_tags.md#prevent-tag-creation-with-the-same-name-as-branches) + to enforce change management rules. #### `include:template` @@ -1518,27 +1533,28 @@ is extracted from the job output. The coverage is shown in the UI if at least on line in the job output matches the regular expression. To extract the code coverage value from the match, GitLab uses -this smaller regular expression: `\d+(\.\d+)?`. +this smaller regular expression: `\d+(?:\.\d+)?`. **Possible inputs**: -- A regular expression. Must start and end with `/`. Must match the coverage number. +- An RE2 regular expression. Must start and end with `/`. Must match the coverage number. May match surrounding text as well, so you don't need to use a regular expression character group to capture the exact number. + Because it uses RE2 syntax, all groups must be non-capturing. **Example of `coverage`**: ```yaml job1: script: rspec - coverage: '/Code coverage: \d+\.\d+/' + coverage: '/Code coverage: \d+(?:\.\d+)?/' ``` In this example: 1. GitLab checks the job log for a match with the regular expression. A line like `Code coverage: 67.89% of lines covered` would match. -1. GitLab then checks the matched fragment to find a match to `\d+(\.\d+)?`. +1. GitLab then checks the matched fragment to find a match to `\d+(?:\.\d+)?`. The sample matching line above gives a code coverage of `67.89`. **Additional details**: @@ -1997,12 +2013,19 @@ at certain stages of job execution, like before retrieving the Git repository. > - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/381840) in GitLab 15.10. Feature flag `ci_hooks_pre_get_sources_script` removed. Use `hooks:pre_get_sources_script` to specify a list of commands to execute on the runner -before retrieving the Git repository and any submodules. You can use it -to adjust the Git client configuration first, for example. +before cloning the Git repository and any submodules. +You can use it for example to: -**Related topics**: +- Adjust the [Git configuration](../troubleshooting.md#get_sources-job-section-fails-because-of-an-http2-problem). +- Export [tracing variables](../../topics/git/useful_git_commands.md). -- [GitLab Runner configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) +**Possible inputs**: An array including: + +- Single line commands. +- Long commands [split over multiple lines](script.md#split-long-commands). +- [YAML anchors](yaml_optimization.md#yaml-anchors-for-scripts). + +CI/CD variables [are supported](../variables/where_variables_can_be_used.md#gitlab-ciyml-file). **Example of `hooks:pre_get_sources_script`**: @@ -2014,6 +2037,10 @@ job1: script: echo 'hello job1 script' ``` +**Related topics**: + +- [GitLab Runner configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) + ### `id_tokens` > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/356986) in GitLab 15.7. @@ -2717,7 +2744,7 @@ linux:rspec: parallel: matrix: - PROVIDER: aws - - STACK: app1 + STACK: app1 script: echo "Running rspec on linux..." ``` @@ -3379,6 +3406,8 @@ only one of the jobs starts. The other jobs wait until the `resource_group` is f Resource groups behave similar to semaphores in other programming languages. +You can choose a [process mode](../resource_groups/index.md#process-modes) to strategically control the job concurrency for your deployment preferences. The default process mode is `unordered`. To change the process mode of a resource group, use the [API](../../api/resource_groups.md#edit-an-existing-resource-group) to send a request to edit an existing resource group. + You can define multiple resource groups per environment. For example, when deploying to physical devices, you might have multiple physical devices. Each device can be deployed to, but only one deployment can occur per device at any given time. @@ -3598,10 +3627,10 @@ to specific files. WARNING: You should use `rules: changes` only with **branch pipelines** or **merge request pipelines**. You can use `rules: changes` with other pipeline types, but `rules: changes` always -evaluates to true when there is no Git `push` event. Tag pipelines, scheduled pipelines, manual pipelines, -and so on do **not** have a Git `push` event associated with them. A `rules: changes` job -is **always** added to those pipelines if there is no `if` that limits the job to -branch or merge request pipelines. +evaluates to true for new branch pipelines or when there is no Git `push` event. Pipelines like tag pipelines, +scheduled pipelines, and manual pipelines, all do **not** +have a Git `push` event associated with them. In these cases, use [`rules: changes: compare_to`](#ruleschangescompare_to) +to specify the branch to compare against. **Keyword type**: Job keyword. You can use it only as part of a job. @@ -4368,6 +4397,9 @@ test: ### `trigger` +> - Support for `resource_group` [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/39057) support for `resource_group` in GitLab 13.9. +> - Support for `environment` [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/369061) in GitLab 16.4. + Use `trigger` to declare that a job is a "trigger job" which starts a [downstream pipeline](../pipelines/downstream_pipelines.md) that is either: @@ -4386,6 +4418,8 @@ The keywords available for use in trigger jobs are: - [`trigger`](#trigger). - [`variables`](#variables). - [`when`](#when) (only with a value of `on_success`, `on_failure`, or `always`). +- [`resource_group`](#resource_group). +- [`environment`](#environment). **Keyword type**: Job keyword. You can use it only as part of a job. @@ -4417,6 +4451,7 @@ trigger-multi-project-pipeline: to forward these variables to downstream pipelines. - [Job-level persisted variables](../variables/where_variables_can_be_used.md#persisted-variables) are not available in trigger jobs. +- Environment variables [defined in the runner's `config.toml`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) are not available to trigger jobs and are not passed to downstream pipelines. **Related topics**: diff --git a/doc/ci/yaml/inputs.md b/doc/ci/yaml/inputs.md new file mode 100644 index 00000000000..1af53d666ce --- /dev/null +++ b/doc/ci/yaml/inputs.md @@ -0,0 +1,174 @@ +--- +stage: Verify +group: Pipeline Authoring +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Define inputs for configuration added with `include` **(FREE ALL BETA)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/391331) in GitLab 15.11 as a Beta feature. + +FLAG: +`spec` and `inputs` are experimental [Open Beta features](../../policy/experiment-beta-support.md#beta) +and subject to change without notice. + +## Define input parameters with `spec:inputs` + +Use `spec:inputs` to define input parameters for CI/CD configuration intended to be added +to a pipeline with `include`. Use [`include:inputs`](#set-input-parameter-values-with-includeinputs) +to define the values to use when the pipeline runs. + +The specs must be declared at the top of the configuration file, in a header section. +Separate the header from the rest of the configuration with `---`. + +Use the interpolation format `$[[ input.input-id ]]` to reference the values outside of the header section. +The inputs are evaluated and interpolated once, when the configuration is fetched +during pipeline creation, but before the configuration is merged with the contents of the `.gitlab-ci.yml`. + +```yaml +spec: + inputs: + environment: + job-stage: +--- + +scan-website: + stage: $[[ inputs.job-stage ]] + script: ./scan-website $[[ inputs.environment ]] +``` + +When using `spec:inputs`: + +- Defined inputs are mandatory by default. +- Inputs can be made optional by specifying a `default`. Use `default: null` to have no default value. +- A string containing an interpolation block must not exceed 1 MB. +- The string inside an interpolation block must not exceed 1 KB. + +For example, a `custom_configuration.yml`: + +```yaml +spec: + inputs: + website: + user: + default: 'test-user' + flags: + default: null +--- + +# The pipeline configuration would follow... +``` + +In this example: + +- `website` is mandatory and must be defined. +- `user` is optional. If not defined, the value is `test-user`. +- `flags` is optional. If not defined, it has no value. + +## Set input parameter values with `include:inputs` + +> `include:with` [renamed to `include:inputs`](https://gitlab.com/gitlab-org/gitlab/-/issues/406780) in GitLab 16.0. + +Use `include:inputs` to set the values for the parameters when the included configuration +is added to the pipeline. + +For example, to include a `custom_configuration.yml` that has the same specs +as the [example above](#define-input-parameters-with-specinputs): + +```yaml +include: + - local: 'custom_configuration.yml' + inputs: + website: "My website" +``` + +In this example: + +- `website` has a value of `My website` for the included configuration. +- `user` has a value of `test-user`, because that is the default when not specified. +- `flags` has no value, because it is optional and has no default when not specified. + +### Use `include:inputs` with multiple files + +`inputs` must be specified separately for each included file. For example: + +```yaml +include: + - component: gitlab.com/org/my-component@1.0 + inputs: + stage: my-stage + - local: path/to/file.yml + inputs: + stage: my-stage +``` + +You can also include the same file multiple times, with different inputs. +For example: + +```yaml +include: + - local: path/to/my-super-linter.yml + inputs: + type: docs + job-name: lint-docs + lint-path: "doc/" + - local: path/to/my-super-linter.yml + inputs: + type: yaml + job-name: lint-yaml + lint-path: "data/yaml/" +``` + +## Specify functions to manipulate input values + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/409462) in GitLab 16.3. + +You can specify predefined functions in the interpolation block to manipulate the input value. +The format supported is the following: + +```yaml +$[[ input.input-id | <function1> | <function2> | ... <functionN> ]] +``` + +Details: + +- Only [predefined interpolation functions](#predefined-interpolation-functions) are permitted. +- A maximum of 3 functions may be specified in a single interpolation block. +- The functions are executed in the sequence they are specified. + +```yaml +spec: + inputs: + test: + default: '0123456789' +--- + +test-job: + script: echo $[[ inputs.test | truncate(1,3) ]] +``` + +In this example: + +- The function [`truncate`](#truncate) applies to the value of `inputs.test`. +- Assuming the value of `inputs.test` is `0123456789`, then the output of `script` would be `echo 123`. + +### Predefined interpolation functions + +#### `truncate` + +Use `truncate` to shorten the interpolated value. For example: + +- `truncate(<offset>,<length>)` + +| Name | Type | Description | +| ---- | ---- | ----------- | +| `offset` | Integer | Number of characters to offset by. | +| `length` | Integer | Number of characters to return after the offset. | + +Example: + +```yaml +$[[ inputs.test | truncate(3,5) ]] +``` + +Assuming the value of `inputs.test` is `0123456789`, then the output would be `34567`. diff --git a/doc/ci/yaml/signing_examples.md b/doc/ci/yaml/signing_examples.md index 72e007a749f..e97ade891c4 100644 --- a/doc/ci/yaml/signing_examples.md +++ b/doc/ci/yaml/signing_examples.md @@ -7,12 +7,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Use Sigstore for keyless signing and verification **(FREE SAAS)** The [Sigstore](https://www.sigstore.dev/) project provides a CLI called -[Cosign](https://docs.sigstore.dev/cosign/overview/) which can be used for keyless signing of container images built +[Cosign](https://docs.sigstore.dev/signing/quickstart/) which can be used for keyless signing of container images built with GitLab CI/CD. Keyless signing has many advantages, including eliminating the need to manage, safeguard, and rotate a private key. Cosign requests a short-lived key pair to use for signing, records it on a certificate transparency log, and then discards it. The key is generated through a token obtained from the GitLab server using the OIDC identity of the user who ran the pipeline. This token includes unique claims that certify the token was generated by a CI/CD pipeline. To learn more, -see Cosign [documentation](https://docs.sigstore.dev/cosign/overview/#example-working-with-containers) on keyless signatures. +see Cosign [documentation](https://docs.sigstore.dev/signing/quickstart/#example-working-with-containers) on keyless signatures. For details on the mapping between GitLab OIDC claims and Fulcio certificate extensions, see the GitLab column of [Mapping OIDC token claims to Fulcio OIDs](https://github.com/sigstore/fulcio/blob/main/docs/oid-info.md#mapping-oidc-token-claims-to-fulcio-oids). @@ -30,17 +30,21 @@ You can use Cosign to sign and verify container images and build artifacts. - You must use a version of Cosign that is `>= 2.0.1`. +**Limitations** + +- The `id_tokens` portion of the CI/CD config file must be located in the project that is being built and signed. AutoDevOps, CI files included from another repository, and child pipelines are not supported. Work to remove this limitation is being tracked in [issue 411317](https://gitlab.com/gitlab-org/gitlab/-/issues/411317). + **Best practices**: - Build and sign an image/artifact in the same job to prevent it from being tampered with before it is signed. - When signing container images, sign the digest (which is immutable) instead of the tag. GitLab [ID tokens](../secrets/id_token_authentication.md#id-tokens) can be used by Cosign for -[keyless signing](https://docs.sigstore.dev/cosign/overview/). The token must have +[keyless signing](https://docs.sigstore.dev/signing/quickstart/). The token must have `sigstore` set as the [`aud`](../secrets/id_token_authentication.md#token-payload) claim. The token can be used by Cosign automatically when it is set in the `SIGSTORE_ID_TOKEN` environment variable. -To learn more about how to install Cosign, see [Cosign Installation documentation](https://docs.sigstore.dev/cosign/installation/). +To learn more about how to install Cosign, see [Cosign Installation documentation](https://docs.sigstore.dev/system_config/installation/). ### Signing @@ -49,7 +53,7 @@ To learn more about how to install Cosign, see [Cosign Installation documentatio The example below demonstrates how to sign a container image in GitLab CI. The signature is automatically stored in the same container repository as the image. -To learn more about signing containers, see [Cosign Signing Containers documentation](https://docs.sigstore.dev/cosign/signing_with_containers/). +To learn more about signing containers, see [Cosign Signing Containers documentation](https://docs.sigstore.dev/signing/signing_with_containers/). ```yaml build_and_sign_image: @@ -77,7 +81,7 @@ build_and_sign_image: The example below demonstrates how to sign a build artifact in GitLab CI. You should save the `cosign.bundle` file produced by `cosign sign-blob`, which is used for signature verification. -To learn more about signing artifacts, see [Cosign Signing Blobs documentation](https://docs.sigstore.dev/cosign/signing_with_blobs/#keyless-signing-of-blobs-and-files). +To learn more about signing artifacts, see [Cosign Signing Blobs documentation](https://docs.sigstore.dev/signing/signing_with_blobs/). ```yaml build_and_sign_artifact: @@ -109,7 +113,7 @@ build_and_sign_artifact: | `--certificate-oidc-issuer` | The GitLab instance URL where the image/artifact was signed. For example, `https://gitlab.com`. | | `--bundle` | The `bundle` file produced by `cosign sign-blob`. Only used for verifying build artifacts. | -To learn more about verifying signed images/artifacts, see [Cosign Verifying documentation](https://docs.sigstore.dev/cosign/verify/#keyless-verification-using-openid-connect). +To learn more about verifying signed images/artifacts, see [Cosign Verifying documentation](https://docs.sigstore.dev/verifying/verify/). #### Container images @@ -149,7 +153,7 @@ You can use Sigstore and npm, together with GitLab CI/CD, to digitally sign buil ### About npm provenance -[npm CLI](https://docs.npmjs.com/cli) allows package maintainers to provide users with provenance attestations. Using npm +[npm CLI](https://docs.npmjs.com/cli/) allows package maintainers to provide users with provenance attestations. Using npm CLI provenance generation allows users to trust and verify that the package they are downloading and using is from you and the build system that built it. |
