diff options
Diffstat (limited to 'doc/ci/migration/jenkins.md')
| -rw-r--r-- | doc/ci/migration/jenkins.md | 345 |
1 files changed, 173 insertions, 172 deletions
diff --git a/doc/ci/migration/jenkins.md b/doc/ci/migration/jenkins.md index 54ea3812029..d02c2f9c54e 100644 --- a/doc/ci/migration/jenkins.md +++ b/doc/ci/migration/jenkins.md @@ -7,215 +7,175 @@ type: index, howto # Migrating from Jenkins **(FREE ALL)** -A lot of GitLab users have successfully migrated to GitLab CI/CD from Jenkins. -We've collected several resources here that you might find informative if you're just getting started. -Think of this page as a "GitLab CI/CD for Jenkins Users" guide. +If you're migrating from Jenkins to GitLab CI/CD, you should be able +to create CI/CD pipelines that do everything you need. + +You can start by watching the [Migrating from Jenkins to GitLab](https://www.youtube.com/watch?v=RlEVGOpYF5Y) +video for examples of: + +- Converting a Jenkins pipeline into a GitLab CI/CD pipeline. +- Using Auto DevOps to test your code automatically. + +## Get started The following list of recommended steps was created after observing organizations -that were able to quickly complete this migration: - -1. Start by reading the GitLab CI/CD [Quick Start Guide](../quick_start/index.md) and [important product differences](#important-product-differences). -1. Learn the importance of [managing the organizational transition](#manage-organizational-transition). -1. [Add runners](../runners/index.md) to your GitLab instance. -1. Educate and enable your developers to independently perform the following steps in their projects: - 1. Review the [Quick Start Guide](../quick_start/index.md) and [Pipeline Configuration Reference](../yaml/index.md). - 1. Use the [Jenkins Wrapper](#jenkinsfile-wrapper) to temporarily maintain fragile Jenkins jobs. - 1. Migrate the build and CI jobs and configure them to show results directly in your merge requests. They can use [Auto DevOps](../../topics/autodevops/index.md) as a starting point, and [customize](../../topics/autodevops/customize.md) or [decompose](../../topics/autodevops/customize.md#use-individual-components-of-auto-devops) the configuration as needed. - 1. Add [Review Apps](../review_apps/index.md). - 1. Migrate the deployment jobs using [cloud deployment templates](../cloud_deployment/index.md), adding [environments](../environments/index.md), and [deploy boards](../../user/project/deploy_boards.md). - 1. Work to unwrap any jobs still running with the use of the Jenkins wrapper. -1. Take stock of any common CI/CD job definitions then create and share [templates](#templates) for them. -1. Check the [pipeline efficiency documentation](../pipelines/pipeline_efficiency.md) - to learn how to make your GitLab CI/CD pipelines faster and more efficient. - -Watch the [Migrating from Jenkins to GitLab](https://www.youtube.com/watch?v=RlEVGOpYF5Y) video for examples of how to: - -- Convert a Jenkins pipeline into a GitLab CI/CD pipeline. -- Use Auto DevOps to test your code automatically. - -Otherwise, read on for important information that helps you get the ball rolling. Welcome -to GitLab! +that were able to quickly complete this migration. + +Before doing any migration work, you should [start with a migration plan](plan_a_migration.md). + +Engineers that need to migrate projects to GitLab CI/CD should: + +- Read about some [key GitLab CI/CD features](#key-gitlab-cicd-features). +- Follow tutorials to create: + - [Your first GitLab pipeline](../quick_start/index.md). + - [A more complex pipeline](../quick_start/tutorial.md) that builds, tests, + and deploys a static site. +- Review the [`.gitlab-ci.yml` keyword reference](../yaml/index.md). +- Ensure [runners](../runners/index.md) are available, either by using shared GitLab.com runners + or installing new runners. +- Migrate build and CI jobs and configure them to show results directly in merge requests. + You can use [Auto DevOps](../../topics/autodevops/index.md) as a starting point, + and [customize](../../topics/autodevops/customize.md) or [decompose](../../topics/autodevops/customize.md#use-individual-components-of-auto-devops) + the configuration as needed. +- Migrate deployment jobs by using [cloud deployment templates](../cloud_deployment/index.md), + [environments](../environments/index.md), and the [GitLab agent for Kubernetes](../../user/clusters/agent/index.md). +- Check if any CI/CD configuration can be reused across different projects, then create + and share [templates](#templates). +- Check the [pipeline efficiency documentation](../pipelines/pipeline_efficiency.md) + to learn how to make your GitLab CI/CD pipelines faster and more efficient. If you have questions that are not answered here, the [GitLab community forum](https://forum.gitlab.com/) can be a great resource. -## Manage organizational transition - -An important part of transitioning from Jenkins to GitLab is the cultural and organizational -changes that come with the move, and successfully managing them. A few -things we have found that help this are: - -- Setting and communicating a clear vision of what your migration goals are helps - your users understand why the effort is worth it. The value is clear when - the work is done, but people need to be aware while it's in progress too. -- Sponsorship and alignment from the relevant leadership team helps with the point above. -- Spending time educating your users on what's different and sharing this document - with them helps ensure you are successful. -- Finding ways to sequence or delay parts of the migration can help a lot, but you - don't want to leave things in a non-migrated (or partially-migrated) state for too - long. To gain all the benefits of GitLab, moving your existing Jenkins setup over - as-is, including any current problems, isn't enough. You need to take advantage - of the improvements that GitLab offers, and this requires (eventually) updating - your implementation as part of the transition. - -## JenkinsFile Wrapper - -We are building a [JenkinsFile Wrapper](https://gitlab.com/gitlab-org/jfr-container-builder/) which -you can use to run a complete Jenkins instance inside of a GitLab job, including plugins. This can help ease the process -of transition, by letting you delay the migration of less urgent pipelines for a period of time. - -If you are interested in helping GitLab test the wrapper, join our [public testing issue](https://gitlab.com/gitlab-org/gitlab/-/issues/215675) for instructions and to provide your feedback. - -NOTE: -If you have a paid GitLab subscription, the JenkinsFile Wrapper is not packaged with GitLab and falls outside of the scope of support. For more information, see the [Statement of Support](https://about.gitlab.com/support/statement-of-support/). - -## Important product differences - -Some high level differences between the products worth mentioning are: - -- With GitLab you don't need a root `pipeline` keyword to wrap everything. -- The way pipelines are triggered and [trigger other pipelines](../yaml/index.md#trigger) - is different than Jenkins. GitLab pipelines can be triggered: - - - on push - - on [schedule](../pipelines/schedules.md) - - from the [GitLab UI](../pipelines/index.md#run-a-pipeline-manually) - - by [API call](../triggers/index.md) - - by [webhook](../triggers/index.md#use-a-webhook) - - by [ChatOps](../chatops/index.md) - -- You can control which jobs run in which cases, depending on how they are triggered, - with the [`rules` syntax](../yaml/index.md#rules). -- GitLab [pipeline scheduling concepts](../pipelines/schedules.md) are also different from Jenkins. -- You can reuse pipeline configurations using the [`include` keyword](../yaml/index.md#include) - and [templates](#templates). Your templates can be kept in a central repository (with different - permissions), and then any project can use them. This central project could also - contain scripts or other reusable code. -- You can also use the [`extends` keyword](../yaml/index.md#extends) to reuse configuration - in a single pipeline configuration. -- All jobs in a single stage always run in parallel, and all stages run in sequence. - Certain jobs might break this sequencing as needed with our [directed acyclic graph](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/47063) - feature. +### Key GitLab CI/CD features + +GitLab CI/CD key features might be different or not exist in Jenkins. For example, +in GitLab: + +- Pipelines can be triggered with: + - A Git push + - A [Schedule](../pipelines/schedules.md) + - The [GitLab UI](../pipelines/index.md#run-a-pipeline-manually) + - An [API call](../triggers/index.md) + - A [webhook](../triggers/index.md#use-a-webhook) +- You can control which jobs run in which cases with the [`rules` syntax](../yaml/index.md#rules). +- You can reuse pipeline configurations: + - Use the [`extends` keyword](../yaml/index.md#extends) to reuse configuration + in a single pipeline configuration. + - Use the [`include` keyword](../yaml/index.md#include) to reuse configuration across + multiple pipelines and projects. +- Jobs are grouped into stages, and jobs in the same stage can run at the same time. + Stages run in sequence. Jobs can be configured to run outside of the stage ordering with the + [`needs` keyword](../yaml/index.md#needs). - The [`parallel`](../yaml/index.md#parallel) keyword can automatically parallelize tasks, - like tests that support parallelization. -- Usually all jobs in a single stage run in parallel, and all stages run in sequence. - Different [pipeline architectures](../pipelines/pipeline_architectures.md) allow you to change this behavior. -- The new [`rules` syntax](../yaml/index.md#rules) is the recommended method of - controlling when different jobs run. It is more powerful than the `only/except` syntax. -- One important difference is that jobs run independently of each other and have a - fresh environment in each job. Passing artifacts between jobs is controlled using the - [`artifacts`](../yaml/index.md#artifacts) and [`dependencies`](../yaml/index.md#dependencies) - keywords. When finished, use the planned [Workspaces](https://gitlab.com/gitlab-org/gitlab/-/issues/29265) - feature to persist a common workspace between serial jobs. -- The `.gitlab-ci.yml` file is checked in to the root of your repository, much like a Jenkinsfile, but - is in the YAML format (see [complete reference](../yaml/index.md)) instead of a Groovy DSL. It's most - analogous to the declarative Jenkinsfile format. -- Manual approvals or gates can be set up as [`when:manual` jobs](../jobs/job_control.md#create-a-job-that-must-be-run-manually). These can - also leverage [`protected environments`](../jobs/job_control.md#run-a-job-after-a-delay) - to control who is able to approve them. -- GitLab comes with a [container registry](../../user/packages/container_registry/index.md), so you can use - container images to set up your build environment. For example, set up one pipeline that builds your build environment - itself and publish that to the container registry. Then, have your pipelines use this instead of each building their - own environment, which is slower and may be less consistent. We have extensive documentation on [how to use the Container Registry](../../user/packages/container_registry/index.md). -- A central utilities repository can be a great place to put assorted scheduled jobs - or other manual jobs that function like utilities. Jenkins installations tend to - have a few of these. - -## Agents vs. runners - -Both Jenkins agents and GitLab runners are the hosts that run jobs. To convert the -Jenkins agent, uninstall it and then [install and register the runner](../runners/index.md). -Runners do not require much overhead, so you can size them similarly to the Jenkins -agents you were using. - -Some important differences in the way runners work in comparison to agents are: - -- Runners can be set up as [shared across an instance, be added at the group level, or set up at the project level](../runners/runners_scope.md). - They self-select jobs from the scopes you've defined automatically. -- You can also [use tags](../runners/configure_runners.md#use-tags-to-control-which-jobs-a-runner-can-run) for finer control, and - associate runners with specific jobs. For example, you can use a tag for jobs that + especially tests that support parallelization. +- Jobs run independently of each other and have a fresh environment for each job. + Passing artifacts between jobs is controlled using the [`artifacts`](../yaml/index.md#artifacts) + and [`dependencies`](../yaml/index.md#dependencies) keywords. +- The `.gitlab-ci.yml` configuration file exists in your Git repository, like a `Jenkinsfile`, + but is [a YAML file](#yaml-configuration-file), not Groovy. +- GitLab comes with a [container registry](../../user/packages/container_registry/index.md). + You can build and store custom container images to run your jobs in. + +## Runners + +Like Jenkins agents, GitLab runners are the hosts that run jobs. If you are using GitLab.com, +you can use the [shared runner fleet](../runners/index.md) to run jobs without provisioning +your own runners. + +To convert a Jenkins agent for use with GitLab CI/CD, uninstall the agent and then +[install and register a runner](../runners/index.md). Runners do not require much overhead, +so you might be able to use similar provisioning as the Jenkins agents you were using. + +Some key details about runners: + +- Runners can be [configured](../runners/runners_scope.md) to be shared across an instance, + a group, or dedicated to a single project. +- You can use the [`tags` keyword](../runners/configure_runners.md#use-tags-to-control-which-jobs-a-runner-can-run) + for finer control, and associate runners with specific jobs. For example, you can use a tag for jobs that require dedicated, more powerful, or specific hardware. - GitLab has [autoscaling for runners](https://docs.gitlab.com/runner/configuration/autoscale.html). Use autoscaling to provision runners only when needed and scale down when not needed, similar to ephemeral agents in Jenkins. -If you are using `gitlab.com`, you can take advantage of our [shared runner fleet](../runners/index.md) -to run jobs without provisioning your own runners. We are investigating making them -[available for self-managed instances](https://gitlab.com/groups/gitlab-org/-/epics/835) -as well. +## YAML configuration file -## Groovy vs. YAML +GitLab pipeline configuration files use the [YAML](https://yaml.org/) format instead of +the [Groovy](https://groovy-lang.org/) format that Jenkins uses. -Jenkins Pipelines are based on [Groovy](https://groovy-lang.org/), so the pipeline specification is written as code. -GitLab works a bit differently, using the more highly structured [YAML](https://yaml.org/) format. -The scripting elements are in `script` blocks separate from the pipeline specification itself. - -Using YAML is a strength of GitLab, in that it helps keep the learning curve much simpler to get up and running. -It also avoids some of the problem of unconstrained complexity which can make your Jenkinsfile hard to understand -and manage. - -We do of course still value DRY (don't repeat yourself) principles. We want to ensure that -behaviors of your jobs can be codified once and applied as needed. You can use the `extends` syntax to -[reuse configuration in your jobs](../yaml/index.md#extends), and `include` can -be used to [reuse pipeline configurations](../yaml/index.md#include) in pipelines -in different projects: +Using YAML is a strength of GitLab CI/CD, as it is a simple format to understand +and start using. For example, a small configuration file with two jobs and some +shared configuration in a hidden job: ```yaml -.in-docker: +.test-config: tags: - - docker - image: alpine + - docker-runners + stage: test -rspec: +test-job: extends: - - .in-docker + - .docker-config script: - - rake rspec + - bundle exec rake rspec + +lint-job: + extends: + - .docker-config + script: + - yarn run prettier ``` -## Artifact publishing +In this example: + +- The commands to run in jobs are added with the [`script` keyword](../yaml/index.md#script). +- The [`extends` keyword](../yaml/index.md#extends) reduces duplication in the configuration + by adding the same `tags` and `stage` configuration defined in `.test-config` to both jobs. + +### Artifacts + +In GitLab, any job can use the [`artifacts` keyword](../yaml/index.md#artifacts) +to define a set of [artifacts](../jobs/job_artifacts.md) to be stored when a job completes. +Artifacts are files that can be used in later jobs, for example for testing or deployment. -Artifacts may work a bit differently than you've used them with Jenkins. In GitLab, any job can define -a set of artifacts to be saved by using the `artifacts` keyword. This can be configured to point to a file -or set of files that can then be persisted from job to job. Read more on our detailed -[artifacts documentation](../jobs/job_artifacts.md): +For example: ```yaml pdf: script: xelatex mycv.tex artifacts: paths: - - ./mycv.pdf - - ./output/ + - mycv.pdf + - output/ expire_in: 1 week ``` -Additionally, we have package management features like built-in container and package registries that you -can leverage. You can see the complete list of packaging features in the -[Packages and registries](../../user/packages/index.md) documentation. +In this example: -## Integrated features +- The `mycv.pdf` file and all the files in `output/` are stored and could be used + in later jobs. +- To save resources, the artifacts expire and are deleted after one week. -You may have used plugins to get things like code quality, unit tests, and security scanning working in Jenkins. -GitLab takes advantage of our connected ecosystem to automatically pull these kinds of results into -your merge requests, pipeline details pages, and other locations. You may find that you actually don't -need to configure anything to have these appear. +### Scanning features -Our [CI/CD feature index](../index.md#features) has the full list of bundled features and links to the documentation for each. -Refer to this index if these features aren't working as expected, or if you'd like to see what's available. +You might have used plugins for things like code quality, security, or static application scanning +in Jenkins. Tools like these are already available in GitLab and can be used in your +pipeline. + +GitLab features including [code quality](../testing/code_quality.md), [security scanning](../../user/application_security/index.md), +[SAST](../../user/application_security/sast/index.md), and many others generate reports +when they complete. These reports can be displayed in merge requests and pipeline details pages. ### Templates -For advanced CI/CD teams, project templates can enable the reuse of pipeline configurations, -as well as encourage inner sourcing. +For organizations with many CI/CD pipelines, you can use project templates to configure +custom CI/CD configuration templates and reuse them across projects. + +Group maintainers can configure a group to use as the source for [custom project templates](../../administration/custom_project_templates.md). +These templates can be used by all projects in the group. -In self-managed GitLab instances, you can build an [Instance Template Repository](../../administration/settings/instance_template_repository.md). -Development teams across the whole organization can select templates from a dropdown list. -A group maintainer or a group owner is able to set a group to use as the source for the -[custom project templates](../../administration/custom_project_templates.md). This can -be used by all projects in the group. An instance administrator can set a group as -the source for [instance project templates](../../user/group/custom_project_templates.md), -which can be used by projects in that instance. +An instance administrator can set a group as the source for [instance project templates](../../user/group/custom_project_templates.md), +which can be used by all projects in that instance. ## Convert a declarative Jenkinsfile @@ -361,7 +321,48 @@ my_job: - if: $CI_COMMIT_BRANCH ``` +## Secrets Management + +Privileged information, often referred to as "secrets", is sensitive information +or credentials you need in your CI/CD workflow. You might use secrets to unlock protected resources +or sensitive information in tools, applications, containers, and cloud-native environments. + +Secrets management in Jenkins is usually handled with the `Secret` type field or the +Credentials Plugin. Credentials stored in the Jenkins settings can be exposed to +jobs as environment variables by using the Credentials Binding plugin. + +For secrets management in GitLab, you can use one of the supported integrations +for an external service. These services securely store secrets outside of your GitLab project, +though you must have a subscription for the service: + +- [HashiCorp Vault](../secrets/id_token_authentication.md#automatic-id-token-authentication-with-hashicorp-vault) +- [Azure Key Vault](../secrets/azure_key_vault.md). + +GitLab also supports [OIDC authentication](../secrets/id_token_authentication.md) +for other third party services that support OIDC. + +Additionally, you can make credentials available to jobs by storing them in CI/CD variables, though secrets +stored in plain text are susceptible to accidental exposure, [the same as in Jenkins](https://www.jenkins.io/doc/developer/security/secrets/#storing-secrets). +You should always store sensitive information in [masked](../variables/index.md#mask-a-cicd-variable) +and [protected](../variables/index.md#protect-a-cicd-variable) variables, which mitigates +some of the risk. + +Also, never store secrets as variables in your `.gitlab-ci.yml` file, which is public to all +users with access to the project. Storing sensitive information in variables should +only be done in [the project, group, or instance settings](../variables/index.md#define-a-cicd-variable-in-the-ui). + +Review the [security guidelines](../variables/index.md#cicd-variable-security) to improve +the safety of your CI/CD variables. + ## Additional resources -For help making your pipelines faster and more efficient, see the -[pipeline efficiency documentation](../pipelines/pipeline_efficiency.md). +- You can use the [JenkinsFile Wrapper](https://gitlab.com/gitlab-org/jfr-container-builder/) + to run a complete Jenkins instance inside of a GitLab CI/CD job, including plugins. Use this tool to + help ease the transition to GitLab CI/CD, by delaying the migration of less urgent pipelines. + + NOTE: + The JenkinsFile Wrapper is not packaged with GitLab and falls outside of the scope of support. + For more information, see the [Statement of Support](https://about.gitlab.com/support/statement-of-support/). +- If your tooling outputs packages that you want to make accessible, you can store them + in a [package registry](../../user/packages/index.md). +- Use [review Apps](../review_apps/index.md) to preview changes before merging them. |