From 82b9539d93d7fd3d9d7ebccf4d4472d451e888da Mon Sep 17 00:00:00 2001 From: GitLab Bot Date: Wed, 13 Sep 2023 15:12:03 +0000 Subject: Add latest changes from gitlab-org/gitlab@master --- doc/ci/components/catalog.md | 28 +-- doc/ci/components/index.md | 414 +++++++++++++++++++++++-------------------- 2 files changed, 235 insertions(+), 207 deletions(-) (limited to 'doc/ci/components') diff --git a/doc/ci/components/catalog.md b/doc/ci/components/catalog.md index 3a3f63365bb..12e795073b7 100644 --- a/doc/ci/components/catalog.md +++ b/doc/ci/components/catalog.md @@ -6,17 +6,18 @@ info: To determine the technical writer assigned to the Stage/Group associated w # CI/CD Catalog **(PREMIUM ALL EXPERIMENT)** +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/407249) in GitLab 16.1. + The CI/CD Catalog is a list of [components repositories](index.md#components-repository), each containing resources that you can add to your CI/CD pipelines. -## Mark the project as a catalog resource - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/407249) in GitLab 16.1. +## Mark a components repository as a catalog resource -After components are added to a components repository, they can immediately be [used](index.md#use-a-component-in-a-cicd-configuration) to build pipelines in other projects. +After components are added to a components repository, they can immediately be [used](index.md#use-a-component-in-a-cicd-configuration) +to build pipelines in other projects. -However, this repository is not discoverable. You must mark this project as a catalog resource to allow it to be visible in the CI Catalog -so other users can discover it. +However, this repository is not discoverable. You must mark this project as a catalog resource +to allow it to be visible in the CI/CD Catalog so other users can discover it. To mark a project as a catalog resource: @@ -25,19 +26,8 @@ To mark a project as a catalog resource: 1. Expand **Visibility, project features, permissions**. 1. Scroll down to **CI/CD Catalog resource** and select the toggle to mark the project as a catalog resource. -On the left sidebar, select **Search or go to** and find your project. +Ensure the project has a clear [description](../../user/project/settings/index.md#edit-project-name-and-description), +as the project description is displayed in the component list in the catalog. NOTE: This action is not reversible. - -## Convert a CI template to component - -Any existing CI template, that you share with other projects via `include:` syntax, can be converted to a CI component. - -1. Decide whether you want the component to be part of an existing [components repository](index.md#components-repository), - if you want to logically group components together. Create and setup a [components repository](index.md#components-repository) otherwise. -1. Create a YAML file in the components repository according to the expected [directory structure](index.md#directory-structure). -1. Copy the content of the template YAML file into the new component YAML file. -1. Refactor the component YAML to follow the [best practices](index.md#best-practices) for components. -1. Leverage the `.gitlab-ci.yml` in the components repository to [test changes to the component](index.md#test-a-component). -1. Tag and [release the component](index.md#release-a-component). diff --git a/doc/ci/components/index.md b/doc/ci/components/index.md index d4ed98893af..377a9a2147d 100644 --- a/doc/ci/components/index.md +++ b/doc/ci/components/index.md @@ -15,15 +15,19 @@ to track future work. Tell us about your use case by leaving comments in the epi ## Components Repository -A components repository is a GitLab project with a repository that hosts one or more pipeline components. A pipeline component is a reusable single pipeline configuration unit. You can use them to compose an entire pipeline configuration or a small part of a larger pipeline. It can optionally take [input parameters](../yaml/includes.md#define-input-parameters-with-specinputs). +A components repository is a GitLab project with a repository that hosts one or more pipeline components. +A pipeline component is a reusable single pipeline configuration unit. You can use them to compose +an entire pipeline configuration or a small part of a larger pipeline. -### Create a components repository +A component can optionally take [input parameters](../yaml/includes.md#define-input-parameters-with-specinputs). + +## Create a components repository To create a components repository, you must: 1. [Create a new project](../../user/project/index.md#create-a-blank-project) with a `README.md` file. - -1. Create a `template.yml` file inside the project's root directory that contains the configuration you want to provide as a component. For example: +1. Create a `template.yml` file inside the project's root directory that contains the configuration you want to provide as a component. + For example: ```yaml spec: @@ -116,8 +120,8 @@ namespace named `my-username`: └── .gitlab-ci.yml ``` - The `.gitlab-ci.yml` file is not required for a CI/CD component to work, but [testing the component](#test-a-component) - in a pipeline in the project is recommended. + The `.gitlab-ci.yml` file is not required for a CI/CD component to work, but + [testing the component](#test-the-component) in a pipeline in the project is recommended. This component is referenced with the path `gitlab.com/my-username/my-component@`. @@ -154,248 +158,282 @@ Nesting of components is not possible. For example: │ └── nested_template.yml ``` -### Test a component +## Release a component + +To create a release for a CI/CD component, you can use: + +- The [`release`](../yaml/index.md#release) keyword in a CI/CD pipeline. Like in the + [component testing example](#test-the-component), you can set a component to automatically + be released after all tests pass in pipelines for new tags. +- The [UI for creating a release](../../user/project/releases/index.md#create-a-release). + +All released versions of the components are displayed in the CI/CD Catalog +page for the given resource, providing users with information about official releases. -Testing CI/CD components as part of the development workflow is strongly recommended and helps ensure consistent behavior. +Components [can be used](#use-a-component-in-a-cicd-configuration) without being released, +but only with a commit SHA or a branch name. To enable the use of tags or the `~latest` version keyword, +you must create a release. -Test changes in a CI/CD pipeline like any other project, by creating a `.gitlab-ci.yml` file in the root directory of the component repository. +## Use a component in a CI/CD configuration +You can add a component to a CI/CD configuration with the `include: component` keyword. For example: ```yaml include: - # include the component located in the current project from the current SHA - - component: gitlab.com/$CI_PROJECT_PATH@$CI_COMMIT_SHA + - component: gitlab.example.com/my-namespace/my-component@1.0 inputs: stage: build +``` -stages: [build, test, release] - -# Expect `component-job` is added. -# This is an example of testing that the included component works as expected. -# You can leverage GitLab API endpoints or 3rd party tools to inspect data generated by the component. -ensure-job-added: - stage: test - image: badouralix/curl-jq - script: - - | - route="https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/pipelines/$CI_PIPELINE_ID/jobs" - count=`curl --silent --header "PRIVATE-TOKEN: $API_TOKEN" $route | jq 'map(select(.name | contains("component-job"))) | length'` - if [ "$count" != "1" ]; then - exit 1 - fi +The component is identified by a unique address in the form `/@`, +where: + +- `` matches the GitLab host. +- `` is the component project's full path and directory where the + component YAML file is located. +- `` is the version of the component. In order of highest priority first, + the version can be: + - A commit SHA, for example `e3262fdd0914fa823210cdb79a8c421e2cef79d8`. + - A tag, for example: `1.0`. + - `~latest`, which is a special version that always points to the most recent released tag. + Only available if the component has been [released](#release-a-component). + - A branch name, for example `main`. + +For example, for a component repository located at `gitlab-org/dast` on `gitlab.com`, +the path: + +- `gitlab.com/gitlab-org/dast@main` targets the `template.yml` in the root directory + on the `main` branch. +- `gitlab.com/gitlab-org/dast@e3262fdd0914fa823210cdb79a8c421e2cef79d8` targets the same file + for the specified commit SHA. +- `gitlab.com/gitlab-org/dast@1.0` targets the same file for the `1.0` tag. +- `gitlab.com/gitlab-org/dast@~latest` targets the same file for the latest release. +- `gitlab.com/gitlab-org/dast/api-scan@main` targets a different file, the `template.yml` + in the `/api-scan` directory in the component repository, for the `main` branch. + +**Additional details**: -# If we are tagging a release with a specific convention ("v" + number) and all -# previous checks succeeded, we proceed with creating a release automatically. -create-release: - stage: release - image: registry.gitlab.com/gitlab-org/release-cli:latest - rules: - - if: $CI_COMMIT_TAG =~ /^v\d+/ - script: echo "Creating release $CI_COMMIT_TAG" - release: - tag_name: $CI_COMMIT_TAG - description: "Release $CI_COMMIT_TAG of components repository $CI_PROJECT_PATH" -``` +- You can only reference components in the same GitLab instance as your project. +- If a tag and branch exist with the same name, the tag takes precedence over the branch. +- If a tag is named the same as a commit SHA that exists, like `e3262fdd0914fa823210cdb79a8c421e2cef79d8`, + the commit SHA takes precedence over the tag. -After committing and pushing changes, the pipeline tests the component then releases it if the test passes. +## Best practices -### Release a component +### Avoid using global keywords -To create a release for a CI/CD component, you can use: +You should try to avoid [global keywords](../yaml/index.md#global-keywords) in a component. +Using these keywords in a component affects all jobs in a pipeline, including jobs +directly defined in the main `.gitlab-ci.yml` or in other included components. -- The [`release`](../yaml/index.md#release) keyword in a CI/CD pipeline. -- The [UI for creating a release](../../user/project/releases/index.md#create-a-release). +As an alternative to global keywords, you should instead: -Like in the [example above](#test-a-component), after all tests pass in a pipeline running for a tag ref, we can release a new version of the components repository. +- Add the configuration directly to each job, even if it creates some duplication + in the component configuration. +- Use the [`extends`](../yaml/index.md#extends) keyword in the component. -All released versions of the components repository are displayed in the Components Catalog page for the given resource, providing users with information about official releases. +For example, using the `default` keyword is not recommended: -Components [can be used](#use-a-component-in-a-cicd-configuration) without being released, but only with a commit SHA or a branch name. To enable the use of tags or the `~latest` version keyword, you must create a release. +```yaml +# Not recommended +default: + image: ruby:3.0 -### Use a component in a CI/CD configuration +rspec-1: + script: bundle exec rspec dir1/ -A pipeline component is identified by a unique address in the form `/@` -containing: +rspec-2: + script: bundle exec rspec dir2/ +``` -- **A fully qualified domain name (FQDN)**: The FQDN must match the GitLab host. -- **A specific version**: The version of the component can be (in order of highest priority first): - - A commit SHA, for example `gitlab.com/gitlab-org/dast@e3262fdd0914fa823210cdb79a8c421e2cef79d8`. - - A tag. for example: `gitlab.com/gitlab-org/dast@1.0`. - - `~latest`, which is a special version that always points to the most recent released tag, - for example `gitlab.com/gitlab-org/dast@~latest`. - - A branch name, for example `gitlab.com/gitlab-org/dast@main`. -- **A component path**: Contains the project's full path and the directory where the component YAML file `template.yml` is located. +Instead, you can: -For example, for a component repository located at `gitlab-org/dast` on `gitlab.com`: +- Add the configuration to each job: -- The path `gitlab.com/gitlab-org/dast` tries to load the `template.yml` from the root directory. -- The path `gitlab.com/gitlab-org/dast/api-scan` tries to load the `template.yml` from the `/api-scan` directory. + ```yaml + rspec-1: + image: ruby:3.0 + script: bundle exec rspec dir1/ -**Additional notes:** + rspec-2: + image: ruby:3.0 + script: bundle exec rspec dir2/ + ``` -- You can only reference components in the same GitLab instance as your project. -- If a tag and branch exist with the same name, the tag takes precedence over the branch. -- If a tag is named the same as a commit SHA that exists, like `e3262fdd0914fa823210cdb79a8c421e2cef79d8`, - the commit SHA takes precedence over the tag. +- Use `extends` to reuse configuration: -### Best practices + ```yaml + .rspec-image: + image: ruby:3.0 -#### Avoid using global keywords + rspec-1: + extends: + - .rspec-image + script: bundle exec rspec dir1/ -When using [global keywords](../yaml/index.md#global-keywords) all jobs in the -pipeline are affected. Using these keywords in a component affects all jobs in a -pipeline, whether they are directly defined in the main `.gitlab-ci.yml` or -in any included components. + rspec-2: + extends: + - .rspec-image + script: bundle exec rspec dir2/ + ``` -To make the composition of pipelines more deterministic, either: +### Replace hard-coded values with inputs -- Duplicate the default configuration for each job. -- Use [`extends`](../yaml/index.md#extends) feature within the component. +You should avoid hard-coding values in a CI/CD components. Hard-coded values might force +component users to need to review the component's internal details and adapt their pipeline +to work with the component. -```yaml -## -# BAD -default: - image: ruby:3.0 +A common keyword with problematic hard-coded values is `stage`. If a component job's +stage is set to a specific value, the pipeline using the component **must** define +the exact same stage. Additionally, if the component user wants to use a different stage, +they must [override](../yaml/includes.md#override-included-configuration-values) the configuration. -rspec: - script: bundle exec rspec -``` +The preferred method is to use the [`input` keyword](../yaml/includes.md#define-input-parameters-with-specinputs). +The component user can specify the exact value they need. -```yaml -## -# GOOD -rspec: - image: ruby:3.0 - script: bundle exec rspec -``` +For example: -#### Replace hard-coded values with inputs +- In the component configuration: -A typical hard-coded value found in CI templates is `stage:` value. Such hard coded values may force the user -of the component to know and adapt the pipeline to such implementation details. + ```yaml + spec: + inputs: + stage: + default: test + --- + unit-test: + stage: $[[ inputs.stage ]] + script: echo unit tests + + integration-test: + stage: $[[ inputs.stage ]] + script: echo integration tests + ``` -For example, if `stage: test` is hard-coded for a job in a component, the pipeline using the component must -define the `test` stage. Additionally, if the user of the component want to customize the stage value it has -to override the configuration: +- In the project using the component: -```yaml -## -# BAD: In order to use different stage name you need to override all the jobs -# included by the component. -include: - - component: gitlab.com/gitlab-org/ruby-test@1.0 + ```yaml + include: + - component: gitlab.com/gitlab-org/ruby-test@1.0 + inputs: + stage: verify -stages: [verify, deploy] + stages: [verify, deploy] + ``` -unit-test: - stage: verify +### Replace custom CI/CD variables with inputs -integration-test: - stage: verify -``` +When using CI/CD variables in a component, you should evaluate if the `inputs` keyword +should be used instead. Requiring a user to define custom variables to change a component's +behavior should be avoided. You should try to use `inputs` for any component customization. -```yaml -## -# BAD: In order to use the component correctly you need to define the stage -# that is hard-coded in it. -include: - - component: gitlab.com/gitlab-org/ruby-test@1.0 +Inputs are explicitly defined in the component's specs and are better validated than variables. +For example, if a required input is not passed to the component, GitLab returns a pipeline error. +By contrast, if a variable is not defined, it's value is empty and there is no error. -stages: [test, deploy] -``` +For example, use `inputs` instead of variables to let users change a scanner's output format: -To improve this we can use [input parameters](../yaml/includes.md#define-input-parameters-with-specinputs) -allowing the user of a component to inject values that can be customized: +- In the component configuration: -```yaml -## -# GOOD: We don't need to know the implementation details of a component and instead we can -# rely on the inputs. -include: - - component: gitlab.com/gitlab-org/ruby-test@1.0 + ```yaml + spec: inputs: - stage: verify + scanner-output: + default: json + --- + my-scanner: + script: my-scan --output $[[ inputs.scanner-output ]] + ``` -stages: [verify, deploy] +- In the project using the component: -## -# inside the component's template.yml file: -spec: - inputs: - stage: - default: test ---- -unit-test: - stage: $[[ inputs.stage ]] - script: echo unit tests + ```yaml + include: + - component: gitlab.example.com/my-scanner@1.0 + inputs: + scanner-output: yaml + ``` -integration-test: - stage: $[[ inputs.stage ]] - script: echo integration tests -``` +In other cases, CI/CD variables are still preferred, including: -#### Prefer inputs over variables +- Using [predefined variables](../variables/predefined_variables.md) to automatically configure + a component to match a user's project. +- Requiring tokens or other sensitive values to be stored as [masked or protected variables in project settings](../variables/index.md#define-a-cicd-variable-in-the-ui). -If variables are only used for YAML evaluation (for example `rules`) and not by the Runner -execution, it's advised to use inputs instead. -Inputs are explicitly defined in the component's contract and they are better validated -than variables. +### Use semantic versioning -For example, if a required input is not passed an error is returned as soon as the component -is being used. By contrast, if a variable is not defined, it's value is empty. +When tagging and releasing new versions of components, you should use [semantic versioning](https://semver.org). +Semantic versioning is the standard for communicating that a change is a major, minor, patch +or other kind of change. -```yaml -## -# BAD: you need to configure an environment variable for a custom value that doesn't need -# to be used on the Runner -unit-test: - image: $MY_COMPONENT_X_IMAGE - script: echo unit tests - -integration-test: - image: $MY_COMPONENT_X_IMAGE - script: echo integration tests - -## -# Usage: -include: - - component: gitlab.com/gitlab-org/ruby-test@1.0 +You should use at least the `major.minor` format, as this is widely understood, for example +`2.0` or `2.1`. -variables: - MY_COMPONENT_X_IMAGE: ruby:3.2 -``` +Other examples of semantic versioning: -```yaml -## -# GOOD: we define a customizable value and accept it as input -spec: - inputs: - image: - default: ruby:3.0 ---- -unit-test: - image: $[[ inputs.image ]] - script: echo unit tests +- `1.0.0` +- `2.1.3` +- `1.0.0-alpha` +- `3.0.0-rc1` -integration-test: - image: $[[ inputs.image ]] - script: echo integration tests +### Test the component -## -# Usage: +Testing CI/CD components as part of the development workflow is strongly recommended +and helps ensure consistent behavior. + +You can test changes in a CI/CD pipeline (like any other project) by creating a `.gitlab-ci.yml` +in the root directory. + +For example: + +```yaml include: - - component: gitlab.com/gitlab-org/ruby-test@1.0 + # include the component located in the current project from the current SHA + - component: gitlab.com/$CI_PROJECT_PATH@$CI_COMMIT_SHA inputs: - image: ruby:3.2 + stage: build + +stages: [build, test, release] + +# Expect `component-job` is added. +# This is an example of testing that the included component works as expected. +# You can leverage GitLab API endpoints or 3rd party tools to inspect data generated by the component. +ensure-job-added: + stage: test + image: badouralix/curl-jq + script: + - | + route="https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/pipelines/$CI_PIPELINE_ID/jobs" + count=`curl --silent --header "PRIVATE-TOKEN: $API_TOKEN" $route | jq 'map(select(.name | contains("component-job"))) | length'` + if [ "$count" != "1" ]; then + exit 1 + fi + +# If we are tagging a release with a specific convention ("v" + number) and all +# previous checks succeeded, we proceed with creating a release automatically. +create-release: + stage: release + image: registry.gitlab.com/gitlab-org/release-cli:latest + rules: + - if: $CI_COMMIT_TAG =~ /^v\d+/ + script: echo "Creating release $CI_COMMIT_TAG" + release: + tag_name: $CI_COMMIT_TAG + description: "Release $CI_COMMIT_TAG of components repository $CI_PROJECT_PATH" ``` -#### Use semantic versioning +After committing and pushing changes, the pipeline tests the component then releases it if the test passes. -When tagging and releasing new versions of components we recommend using [semantic versioning](https://semver.org) -which is the standard for communicating bugfixes, minor and major or breaking changes. +## Convert a CI/CD template to a component -We recommend adopting at least the `MAJOR.MINOR` format. +Any existing CI/CD template that you use in projects by using the `include:` syntax +can be converted to a CI/CD component: -For example: `2.1`, `1.0.0`, `1.0.0-alpha`, `2.1.3`, `3.0.0-rc.1`. +1. Decide if you want the component to be part of an existing [components repository](index.md#components-repository) + to be grouped with other components, or create and set up a new components repository. +1. Create a YAML file in the components repository according to the expected [directory structure](index.md#directory-structure). +1. Copy the content of the original template YAML file into the new component YAML file. +1. Refactor the new component's configuration to follow the [best practices](index.md#best-practices) for components. +1. Leverage the `.gitlab-ci.yml` in the components repository to [test changes to the component](index.md#test-the-component). +1. Tag and [release the component](index.md#release-a-component). -- cgit v1.2.3