Add latest changes from gitlab-org/gitlab@13-4-stable-ee

2 files changed, 112 insertions, 49 deletions

diff --git a/doc/development/cicd/index.md b/doc/development/cicd/index.md
index 5b598a19a6e..30ccc52ec5e 100644
--- a/doc/development/cicd/index.md
+++ b/doc/development/cicd/index.md
@@ -25,7 +25,7 @@ On the left side we have the events that can trigger a pipeline based on various
 - The [Web API](../../api/pipelines.md#create-a-new-pipeline).
 - A user clicking the "Run Pipeline" button in the UI.
 - When a [merge request is created or updated](../../ci/merge_request_pipelines/index.md#pipelines-for-merge-requests).
-- When an MR is added to a [Merge Train](../../ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md#merge-trains-premium).
+- When an MR is added to a [Merge Train](../../ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md#merge-trains).
 - A [scheduled pipeline](../../ci/pipelines/schedules.md#pipeline-schedules).
 - When project is [subscribed to an upstream project](../../ci/multi_project_pipelines.md#trigger-a-pipeline-when-an-upstream-project-is-rebuilt).
 - When [Auto DevOps](../../topics/autodevops/index.md) is enabled.
@@ -53,28 +53,28 @@ The component that processes a pipeline is [`ProcessPipelineService`](https://gi
 which is responsible for moving all the pipeline's jobs to a completed state. When a pipeline is created, all its
 jobs are initially in `created` state. This services looks at what jobs in `created` stage are eligible
 to be processed based on the pipeline structure. Then it moves them into the `pending` state, which means
-they can now [be picked up by a Runner](#job-scheduling). After a job has been executed it can complete
+they can now [be picked up by a runner](#job-scheduling). After a job has been executed it can complete
 successfully or fail. Each status transition for job within a pipeline triggers this service again, which
 looks for the next jobs to be transitioned towards completion. While doing that, `ProcessPipelineService`
 updates the status of jobs, stages and the overall pipeline.
 
-On the right side of the diagram we have a list of [Runners](../../ci/runners/README.md#configuring-gitlab-runners)
-connected to the GitLab instance. These can be Shared Runners, Group Runners or Project-specific Runners.
-The communication between Runners and the Rails server occurs through a set of API endpoints, grouped as
+On the right side of the diagram we have a list of [runners](../../ci/runners/README.md)
+connected to the GitLab instance. These can be shared runners, group runners, or project-specific runners.
+The communication between runners and the Rails server occurs through a set of API endpoints, grouped as
 the `Runner API Gateway`.
 
-We can register, delete and verify Runners, which also causes read/write queries to the database. After a Runner is connected,
+We can register, delete, and verify runners, which also causes read/write queries to the database. After a runner is connected,
 it keeps asking for the next job to execute. This invokes the [`RegisterJobService`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/services/ci/register_job_service.rb)
-which will pick the next job and assign it to the Runner. At this point the job will transition to a
+which will pick the next job and assign it to the runner. At this point the job will transition to a
 `running` state, which again triggers `ProcessPipelineService` due to the status change.
 For more details read [Job scheduling](#job-scheduling)).
 
-While a job is being executed, the Runner sends logs back to the server as well any possible artifacts
+While a job is being executed, the runner sends logs back to the server as well any possible artifacts
 that need to be stored. Also, a job may depend on artifacts from previous jobs in order to run. In this
-case the Runner will download them using a dedicated API endpoint.
+case the runner will download them using a dedicated API endpoint.
 
 Artifacts are stored in object storage, while metadata is kept in the database. An important example of artifacts
-is reports (JUnit, SAST, DAST, etc.) which are parsed and rendered in the merge request.
+are reports (JUnit, SAST, DAST, etc.) which are parsed and rendered in the merge request.
 
 Job status transitions are not all automated. A user may run [manual jobs](../../ci/yaml/README.md#whenmanual), cancel a pipeline, retry
 specific failed jobs or the entire pipeline. Anything that
@@ -90,25 +90,25 @@ from the `CreatePipelineService` every time a downstream pipeline is triggered.
 
 When a Pipeline is created all its jobs are created at once for all stages, with an initial state of `created`. This makes it possible to visualize the full content of a pipeline.
 
-A job with the `created` state won't be seen by the Runner yet. To make it possible to assign a job to a Runner, the job must transition first into the `pending` state, which can happen if:
+A job with the `created` state won't be seen by the runner yet. To make it possible to assign a job to a runner, the job must transition first into the `pending` state, which can happen if:
 
 1. The job is created in the very first stage of the pipeline.
 1. The job required a manual start and it has been triggered.
 1. All jobs from the previous stage have completed successfully. In this case we transition all jobs from the next stage to `pending`.
 1. The job specifies DAG dependencies using `needs:` and all the dependent jobs are completed.
 
-When the Runner is connected, it requests the next `pending` job to run by polling the server continuously.
+When the runner is connected, it requests the next `pending` job to run by polling the server continuously.
 
 NOTE: **Note:**
-API endpoints used by the Runner to interact with GitLab are defined in [`lib/api/runner.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/api/runner.rb)
+API endpoints used by the runner to interact with GitLab are defined in [`lib/api/ci/runner.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/api/ci/runner.rb)
 
-After the server receives the request it selects a `pending` job based on the [`Ci::RegisterJobService` algorithm](#ciregisterjobservice), then assigns and sends the job to the Runner.
+After the server receives the request it selects a `pending` job based on the [`Ci::RegisterJobService` algorithm](#ciregisterjobservice), then assigns and sends the job to the runner.
 
-Once all jobs are completed for the current stage, the server "unlocks" all the jobs from the next stage by changing their state to `pending`. These can now be picked by the scheduling algorithm when the Runner requests new jobs, and continues like this until all stages are completed.
+Once all jobs are completed for the current stage, the server "unlocks" all the jobs from the next stage by changing their state to `pending`. These can now be picked by the scheduling algorithm when the runner requests new jobs, and continues like this until all stages are completed.
 
-### Communication between Runner and GitLab server
+### Communication between runner and GitLab server
 
-Once the Runner is [registered](https://docs.gitlab.com/runner/register/) using the registration token, the server knows what type of jobs it can execute. This depends on:
+Once the runner is [registered](https://docs.gitlab.com/runner/register/) using the registration token, the server knows what type of jobs it can execute. This depends on:
 
 - The type of runner it is registered as:
   - a shared runner
@@ -116,30 +116,30 @@ Once the Runner is [registered](https://docs.gitlab.com/runner/register/) using
   - a project specific runner
 - Any associated tags.
 
-The Runner initiates the communication by requesting jobs to execute with `POST /api/v4/jobs/request`. Although this polling generally happens every few seconds we leverage caching via HTTP headers to reduce the server-side work load if the job queue doesn't change.
+The runner initiates the communication by requesting jobs to execute with `POST /api/v4/jobs/request`. Although this polling generally happens every few seconds we leverage caching via HTTP headers to reduce the server-side work load if the job queue doesn't change.
 
 This API endpoint runs [`Ci::RegisterJobService`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/services/ci/register_job_service.rb), which:
 
 1. Picks the next job to run from the pool of `pending` jobs
-1. Assigns it to the Runner
-1. Presents it to the Runner via the API response
+1. Assigns it to the runner
+1. Presents it to the runner via the API response
 
 ### `Ci::RegisterJobService`
 
-There are 3 top level queries that this service uses to gather the majority of the jobs and they are selected based on the level where the Runner is registered to:
+There are 3 top level queries that this service uses to gather the majority of the jobs and they are selected based on the level where the runner is registered to:
 
-- Select jobs for shared Runner (instance level)
-- Select jobs for group level Runner
-- Select jobs for project Runner
+- Select jobs for shared runner (instance level)
+- Select jobs for group runner
+- Select jobs for project runner
 
-This list of jobs is then filtered further by matching tags between job and Runner tags.
+This list of jobs is then filtered further by matching tags between job and runner tags.
 
 NOTE: **Note:**
-If a job contains tags, the Runner will not pick the job if it does not match **all** the tags.
-The Runner may have more tags than defined for the job, but not vice-versa.
+If a job contains tags, the runner will not pick the job if it does not match **all** the tags.
+The runner may have more tags than defined for the job, but not vice-versa.
 
-Finally if the Runner can only pick jobs that are tagged, all untagged jobs are filtered out.
+Finally if the runner can only pick jobs that are tagged, all untagged jobs are filtered out.
 
-At this point we loop through remaining `pending` jobs and we try to assign the first job that the Runner "can pick" based on additional policies. For example, Runners marked as `protected` can only pick jobs that run against protected branches (such as production deployments).
+At this point we loop through remaining `pending` jobs and we try to assign the first job that the runner "can pick" based on additional policies. For example, runners marked as `protected` can only pick jobs that run against protected branches (such as production deployments).
 
-As we increase the number of Runners in the pool we also increase the chances of conflicts which would arise if assigning the same job to different Runners. To prevent that we gracefully rescue conflict errors and assign the next job in the list.
+As we increase the number of runners in the pool we also increase the chances of conflicts which would arise if assigning the same job to different runners. To prevent that we gracefully rescue conflict errors and assign the next job in the list.
diff --git a/doc/development/cicd/templates.md b/doc/development/cicd/templates.md
index 0169ca42ac6..77cedc9814e 100644
--- a/doc/development/cicd/templates.md
+++ b/doc/development/cicd/templates.md
@@ -13,15 +13,15 @@ This document explains how to develop [GitLab CI/CD templates](../../ci/examples
 
 All template files reside in the `lib/gitlab/ci/templates` directory, and are categorized by the following sub-directories:
 
-| Sub-directroy | Content                                                      | [Selectable in UI](#make-sure-the-new-template-can-be-selected-in-ui) |
-|---------------|--------------------------------------------------------------|-----------------------------------------------------------------------|
-| `/AWS/*`      | Cloud Deployment (AWS) related jobs                          | No                                                                    |
-| `/Jobs/*`     | Auto DevOps related jobs                                     | Yes                                                                   |
-| `/Pages/*`    | Static site generators for GitLab Pages (for example Jekyll) | Yes                                                                   |
-| `/Security/*` | Security related jobs                                        | Yes                                                                   |
-| `/Verify/*`   | Verify/testing related jobs                                  | Yes                                                                   |
-| `/Worklows/*` | Common uses of the `workflow:` keyword                       | No                                                                    |
-| `/*` (root)   | General templates                                            | Yes                                                                   |
+| Sub-directory  | Content                                                      | [Selectable in UI](#make-sure-the-new-template-can-be-selected-in-ui) |
+|----------------|--------------------------------------------------------------|-----------------------------------------------------------------------|
+| `/AWS/*`       | Cloud Deployment (AWS) related jobs                          | No                                                                    |
+| `/Jobs/*`      | Auto DevOps related jobs                                     | No                                                                    |
+| `/Pages/*`     | Static site generators for GitLab Pages (for example Jekyll) | Yes                                                                   |
+| `/Security/*`  | Security related jobs                                        | Yes                                                                   |
+| `/Verify/*`    | Verify/testing related jobs                                  | Yes                                                                   |
+| `/Workflows/*` | Common uses of the `workflow:` keyword                       | No                                                                    |
+| `/*` (root)    | General templates                                            | Yes                                                                   |
 
 ## Criteria
 
@@ -64,6 +64,67 @@ users have to fix their `.gitlab-ci.yml` that could annoy their workflow.
 
 Please read [versioning](#versioning) section for introducing breaking change safely.
 
+## Versioning
+
+Versioning allows you to introduce a new template without modifying the existing
+one. This process is useful when we need to introduce a breaking change,
+but don't want to affect the existing projects that depends on the current template.
+
+### Stable version
+
+A stable CI/CD template is a template that only introduces breaking changes in major
+release milestones. Name the stable version of a template as `<template-name>.gitlab-ci.yml`,
+for example `Jobs/Deploy.gitlab-ci.yml`.
+
+You can make a new stable template by copying [the latest template](#latest-version)
+available in a major milestone release of GitLab like `13.0`. All breaking changes
+must be announced in a blog post before the official release, for example
+[GitLab.com is moving to 13.0, with narrow breaking changes](https://about.gitlab.com/releases/2020/05/06/gitlab-com-13-0-breaking-changes/)
+
+You can change a stable template version in a minor GitLab release like `13.1` if:
+
+- The change is not a [breaking change](#backward-compatibility).
+- The change is ported to [the latest template](#latest-version), if one exists.
+
+### Latest version
+
+Templates marked as `latest` can be updated in any release, even with
+[breaking changes](#backward-compatibility). Add `.latest` to the template name if
+it's considered the latest version, for example `Jobs/Deploy.latest.gitlab-ci.yml`.
+
+When you introduce [a breaking change](#backward-compatibility),
+you **must** test and document [the upgrade path](#verify-breaking-changes).
+In general, we should not promote the latest template as the best option, as it could surprise users with unexpected problems.
+
+If the `latest` template does not exist yet, you can copy [the stable template](#stable-version).
+
+### How to include an older stable template
+
+Users may want to use an older [stable template](#stable-version) that is not bundled
+in the current GitLab package. For example, the stable templates in GitLab v13.0 and
+GitLab v14.0 could be so different that a user will want to continue using the v13.0 template even
+after upgrading to GitLab 14.0.
+
+You can add a note in the template or in documentation explaining how to use `include:remote`
+to include older template versions. If other templates are included with `include: template`,
+they can be combined with the `include: remote`:
+
+```yaml
+# To use the v13 stable template, which is not included in v14, fetch the specifc
+# template from the remote template repository with the `include:remote:` keyword.
+# If you fetch from the GitLab canonical project, use the following URL format:
+# https://gitlab.com/gitlab-org/gitlab/-/raw/<version>/lib/gitlab/ci/templates/<template-name>
+include:
+  - template: Auto-DevOps.gitlab-ci.yml
+  - remote: https://gitlab.com/gitlab-org/gitlab/-/raw/v13.0.1-ee/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml
+```
+
+### Further reading
+
+There is an [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/17716) about
+introducing versioning concepts in GitLab CI Templates. You can check that issue to
+follow the progress.
+
 ## Testing
 
 Each CI/CD template must be tested in order to make sure that it's safe to be published.
@@ -95,18 +156,20 @@ You should write an RSpec test to make sure that pipeline jobs will be generated
 1. Add a test file at `spec/lib/gitlab/ci/templates/<template-category>/<template-name>_spec.rb`
 1. Test that pipeline jobs are properly created via `Ci::CreatePipelineService`.
 
+### Verify breaking changes
+
+When you introduce a breaking change to [a `latest` template](#latest-version),
+you must:
+
+1. Test the upgrade path from [the stable template](#stable-version).
+1. Verify what kind of errors users will encounter.
+1. Document it as a troubleshooting guide.
+
+This information will be important for users when [a stable template](#stable-version)
+is updated in a major version GitLab release.
+
 ## Security
 
 A template could contain malicious code. For example, a template that contains the `export` shell command in a job
 might accidentally expose project secret variables in a job log.
 If you're unsure if it's secure or not, you need to ask security experts for cross-validation.
-
-## Versioning
-
-Versioning allows you to introduce a new template without modifying the existing
-one. This is useful process especially when we need to introduce a breaking change,
-but don't want to affect the existing projects that depends on the current template.
-
-There is an [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/17716) for
-introducing versioning concept in GitLab Ci Template. Please follow the issue for
-checking the progress.