diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2022-12-20 17:22:11 +0300 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2022-12-20 17:22:11 +0300 |
commit | 0c872e02b2c822e3397515ec324051ff540f0cd5 (patch) | |
tree | ce2fb6ce7030e4dad0f4118d21ab6453e5938cdd /doc/architecture/blueprints/ci_pipeline_components/index.md | |
parent | f7e05a6853b12f02911494c4b3fe53d9540d74fc (diff) |
Add latest changes from gitlab-org/gitlab@15-7-stable-eev15.7.0-rc42
Diffstat (limited to 'doc/architecture/blueprints/ci_pipeline_components/index.md')
-rw-r--r-- | doc/architecture/blueprints/ci_pipeline_components/index.md | 362 |
1 files changed, 343 insertions, 19 deletions
diff --git a/doc/architecture/blueprints/ci_pipeline_components/index.md b/doc/architecture/blueprints/ci_pipeline_components/index.md index a3c72227f3e..100c9e67fda 100644 --- a/doc/architecture/blueprints/ci_pipeline_components/index.md +++ b/doc/architecture/blueprints/ci_pipeline_components/index.md @@ -3,7 +3,7 @@ status: proposed creation-date: "2022-09-14" authors: [ "@fabio", "@grzesiek" ] coach: "@kamil" -approvers: [ "@dov" ] +approvers: [ "@dhershkovitch", "@marknuzzo" ] owning-stage: "~devops::verify" participating-stages: [] --- @@ -110,7 +110,8 @@ identifying abstract concepts and are subject to changes as we refine the design ## Definition of pipeline component -A pipeline component is a reusable single-purpose building block that abstracts away a single pipeline configuration unit. Components are used to compose a part or entire pipeline configuration. +A pipeline component is a reusable single-purpose building block that abstracts away a single pipeline configuration unit. +Components are used to compose a part or entire pipeline configuration. It can optionally take input parameters and set output data to be adaptable and reusable in different pipeline contexts, while encapsulating and isolating implementation details. @@ -133,27 +134,319 @@ For best experience with any systems made of components it's fundamental that co The version identifies the exact interface and behavior of the component. - **Resolvable**: when a component depends on another component, this dependency must be explicit and trackable. -## Proposal +## Structure of a component -Prerequisites to create a component: +A pipeline component is identified by the path to a repository or directory that defines it +and a specific version: `<component-path>@<version>`. -- Create a project. Description and avatar are highly recommended to improve discoverability. -- Add a `README.md` in the top level directory that documents the component. - What it does, how to use it, how to contribute, etc. - This file is mandatory. -- Add a `.gitlab-ci.yml` in the top level directory to test that the components works as expected. - This file is highly recommended. +For example: `gitlab-org/dast@1.0`. -Characteristics of a component: +### The component path -- It must have a **name** to be referenced to and **description** for extra details. -- It must specify its **type** which defines how it can be used (raw configuration to be `include`d, child pipeline workflow, job step). -- It must define its **content** based on the type. -- It must specify **input parameters** that it accepts. Components should depend on input parameters for dynamic values and not environment variables. -- It can optionally define **output data** that it returns. -- Its YAML specification should be **validated statically** (for example: using JSON schema validators). -- It should be possible to use specific **versions** of a component by referencing official releases and SHA. -- It should be possible to use components defined locally in the same repository. +A component path must contain at least the metadata YAML and optionally a related `README.md` documentation file. + +The component path can be: + +- A path to a project: `gitlab-org/dast`. In this case the 2 files are defined in the root directory of the repository. +- A path to a project subdirectory: `gitlab-org/dast/api-scan`. In this case the 2 files are defined in the `api-scan` directory. +- A path to a local directory: `/path/to/component`. This path must contain the metadata YAML that defines the component. + The path must start with `/` to indicate a full path in the repository. + +The metadata YAML file follows the filename convention `gitlab-<component-type>.yml` where component type is one of: + +| Component type | Context | +| -------------- | ------- | +| `template` | For components used under `include:` keyword | +| `step` | For components used under `steps:` keyword | +| `workflow` | For components used under `trigger:` keyword | + +Based on the context where the component is used we fetch the correct YAML file. +For example, if we are including a component `gitlab-org/dast@1.0` we expect a YAML file named `gitlab-template.yml` in the +top level directory of `gitlab-org/dast` repository. + +A `gitlab-<component-type>.yml` file: + +- Must have a **name** to be referenced to and **description** for extra details. +- Must specify its **type** in the filename, which defines how it can be used (raw configuration to be `include`d, child pipeline workflow, job step). +- Must define its **content** based on the type. +- Must specify **input parameters** that it accepts. Components should depend on input parameters for dynamic values and not environment variables. +- Can optionally define **output data** that it returns. +- Should be **validated statically** (for example: using JSON schema validators). + +```yaml +spec: + inputs: + website: + environment: + default: test + test_run: + options: + - unit + - integration + - system +content: { ... } +``` + +Components that are released in the catalog must have a `README.md` file in the same directory as the +metadata YAML file. The `README.md` represents the documentation for the specific component, hence it's recommended +even when not releasing versions in the catalog. + +### The component version + +The version of the component can be (in order of highest priority first): + +1. A commit SHA - For example: `gitlab-org/dast@e3262fdd0914fa823210cdb79a8c421e2cef79d8` +1. A released tag - For example: `gitlab-org/dast@1.0` +1. A special moving target version that points to the most recent released tag - For example: `gitlab-org/dast@~latest` +1. An unreleased tag - For example: `gitlab-org/dast@rc-1.0` +1. A branch name - For example: `gitlab-org/dast@master` + +If a tag and branch exist with the same name, the tag takes precedence over the branch. +Similarly, if a tag is named `e3262fdd0914fa823210cdb79a8c421e2cef79d8`, a commit SHA (if exists) +takes precedence over the tag. + +As we want to be able to reference any revisions (even those not released), a component must be defined in a Git repository. + +NOTE: +When referencing a component by local path (for example `./path/to/component`), its version is implicit and matches +the commit SHA of the current pipeline context. + +## Components project + +A components project is a GitLab project/repository that exclusively hosts one or more pipeline components. + +For components projects it's highly recommended to set an appropriate avatar and project description +to improve discoverability in the catalog. + +### Structure of a components project + +A project can host one or more components depending on whether the author wants to define a single component +per project or include multiple cohesive components under the same project. + +Let's imagine we are developing a component that runs RSpec tests for a Rails app. We create a component project +called `myorg/rails-rspec`. + +The following directory structure would support 1 component per project: + +```plaintext +. +├── gitlab-<type>.yml +├── README.md +└── .gitlab-ci.yml +``` + +The `.gitlab-ci.yml` is recommended for the project to ensure changes are verified accordingly. + +The component is now identified by the path `myorg/rails-rspec`. In other words, this means that +the `gitlab-<type>.yml` and `README.md` are located in the root directory of the repository. + +The following directory structure would support multiple components per project: + +```plaintext +. +├── .gitlab-ci.yml +├── unit/ +│ ├── gitlab-workflow.yml +│ └── README.md +├── integration/ +│ ├── gitlab-workflow.yml +│ └── README.md +└── feature/ + ├── gitlab-workflow.yml + └── README.md +``` + +In this example we are defining multiple test profiles that are executed with RSpec. +The user could choose to use one or more of these. + +Each of these components are identified by their path `myorg/rails-rspec/unit`, `myorg/rails-rspec/integration` +and `myorg/rails-rspec/feature`. + +This directory structure could also support both strategies: + +```plaintext +. +├── gitlab-template.yml # myorg/rails-rspec +├── README.md +├── .gitlab-ci.yml +├── unit/ +│ ├── gitlab-workflow.yml # myorg/rails-rspec/unit +│ └── README.md +├── integration/ +│ ├── gitlab-workflow.yml # myorg/rails-rspec/integration +│ └── README.md +└── feature/ + ├── gitlab-workflow.yml # myorg/rails-rspec/feature + └── README.md +``` + +With the above structure we could have a top-level component that can be used as the +default component. For example, `myorg/rails-rspec` could run all the test profiles together. +However, more specific test profiles could be used separately (for example `myorg/rails-rspec/integration`). + +NOTE: +Any nesting more than 1 level is initially not permitted. +This limitation encourages cohesion at project level and keeps complexity low. + +## Input parameters `spec:inputs:` parameters + +If the component takes any input parameters they must be specified according to the following schema: + +```yaml +spec: + inputs: + website: # by default all declared inputs are mandatory. + environment: + default: test # apply default if not provided. This makes the input optional. + test_run: + options: # a choice must be made from the list since there is no default value. + - unit + - integration + - system +``` + +When using the component we pass the input parameters as follows: + +```yaml +include: + - component: org/my-component@1.0 + with: + website: ${MY_WEBSITE} # variables expansion + test_run: system + environment: $[[ inputs.environment ]] # interpolation of upstream inputs +``` + +Variables expansion must be supported for `with:` syntax as well as interpolation of +possible [inputs provided upstream](#input-parameters-for-pipelines). + +Input parameters are validated as soon as possible: + +1. Read the file `gitlab-template.yml` inside `org/my-component`. +1. Parse `spec:inputs` and validate the parameters against this schema. +1. If successfully validated, proceed with parsing `content:`. Return an error otherwise. +1. Interpolate input parameters inside the component's `content:`. + +```yaml +spec: + inputs: + environment: + options: [test, staging, production] +content: + "run-tests-$[[ inputs.environment ]]": + script: ./run-test + + scan-website: + script: ./scan-website $[[ inputs.environment ]] + rules: + - if: $[[ inputs.environment ]] == 'staging' + - if: $[[ inputs.environment ]] == 'production' +``` + +With `$[[ inputs.XXX ]]` inputs are interpolated immediately after parsing the `content:`. + +### Why input parameters and not environment variables? + +Until today we have been leveraging environment variables to pass information around. +For example, we use environment variables to pass information from an upstream pipeline to a +downstream pipeline. + +Using environment variables for passing information to a component is like declaring global +variables in programming languages. The more variables we declare the more we risk variable +conflicts and increase variables scope. + +Input parameters are like variables passed to the component which exist inside a specific +scope and they don't leak to the outside. +Inputs are not inherited from upstream `include`s. They must be passed explicitly. + +This paradigm allows to build more robust and isolated components as well as declare and +enforce contracts. + +### Input parameters for existing `include:` syntax + +Because we are adding input parameters to components used via `include:component` we have an opportunity to +extend it to other `include:` types support inputs via `with:` syntax: + +```yaml +include: + - component: org/my-component@1.0 + with: + foo: bar + - local: path/to/file.yml + with: + foo: bar + - project: org/another + file: .gitlab-ci.yml + with: + foo: bar + - remote: http://example.com/ci/config + with: + foo: bar + - template: Auto-DevOps.gitlab-ci.yml + with: + foo: bar +``` + +Then the configuration being included must specify the inputs: + +```yaml +spec: + inputs: + foo: + +# rest of the configuration +``` + +If a YAML includes content using `with:` but the including YAML doesn't specify `inputs:`, an error should be raised. + +|`with:`| `inputs:` | result | +| --- | --- | --- | +| specified | | raise error | +| specified | specified | validate inputs | +| | specified | use defaults | +| | | legacy `include:` without input passing | + +### Input parameters for pipelines + +Inputs can also be used to pass parameters to a pipeline when triggered and benefit from immediate validation. + +Today we have different use cases where using explicit input parameters would be beneficial: + +1. `Run Pipeline` UI form. + - **Problem today**: We are using top-level variables with `variables:*:description` to surface environment variables to the UI. + The problem with this is the mix of responsibilities as well as the jump in [precedence](../../../ci/variables/index.md#cicd-variable-precedence) + that a variable gets (from a YAML variable to a pipeline variable). + Building validation and features on top of this solution is challenging and complex. +1. Trigger a pipeline via API. For example `POST /projects/:id/pipelines/trigger` with `{ inputs: { provider: 'aws' } }` +1. Trigger a pipeline via `trigger:` syntax. + +```yaml +deploy-app: + trigger: + project: org/deployer + with: + provider: aws + deploy_environment: staging +``` + +To solve the problem of `Run Pipeline` UI form we could fully leverage the `spec:inputs` schema: + +```yaml +spec: + inputs: + concurrency: + default: 10 # displayed as default value in the input box + provider: # can enforce `required` in the form validation + description: Deployment provider # optional: render as input label. + deploy_environment: + options: # render a selectbox with options in order of how they are defined below + - staging # 1st option + - canary # 2nd option + - production # 3rd option + default: staging # selected by default in the UI. + # if `default:` is not specified, the user must explicitly select + # an option. + description: Deployment environment # optional: render as input label. +``` ## Limits @@ -188,3 +481,34 @@ Some limits we could consider adding: - Allow self-managed administrators to populate their self-managed catalog by importing/updating components from GitLab.com or from repository exports. - Iterate on feedback. + +## Who + +Proposal: + +<!-- vale gitlab.Spelling = NO --> + +| Role | Who +|--------------------------------|-------------------------| +| Author | Fabio Pitino | +| Engineering Leaders | Cheryl Li, Mark Nuzzo | +| Product Manager | Dov Hershkovitch | +| Architecture Evolution Coaches | Kamil Trzciński, Grzegorz Bizon | + +DRIs: + +| Role | Who +|------------------------------|------------------------| +| Leadership | Mark Nuzzo | +| Product | Dov Hershkovitch | +| Engineering | Fabio Pitino | +| UX | Kevin Comoli (interim), Sunjung Park | + +Domain experts: + +| Area | Who +|------------------------------|------------------------| +| Verify / Pipeline authoring | Avielle Wolfe | +| Verify / Pipeline authoring | Laura Montemayor-Rodriguez | + +<!-- vale gitlab.Spelling = YES --> |