Add latest changes from gitlab-org/gitlab@14-9-stable-eev14.9.0-rc42

3 files changed, 185 insertions, 4 deletions

diff --git a/doc/development/cicd/index.md b/doc/development/cicd/index.md
index 2779b457fb9..8677d5b08e3 100644
--- a/doc/development/cicd/index.md
+++ b/doc/development/cicd/index.md
@@ -7,9 +7,10 @@ type: index, concepts, howto
 
 # CI/CD development documentation **(FREE)**
 
-Development guides that are specific to CI/CD are listed here.
+Development guides that are specific to CI/CD are listed here:
 
-If you are creating new CI/CD templates, please read [the development guide for GitLab CI/CD templates](templates.md).
+- If you are creating new CI/CD templates, please read [the development guide for GitLab CI/CD templates](templates.md).
+- If you are adding a new keyword or changing the CI schema, check the [CI schema guide](schema.md)
 
 See the [CI/CD YAML reference documentation guide](cicd_reference_documentation_guide.md)
 to learn how to update the [reference page](../../ci/yaml/index.md).
@@ -29,7 +30,7 @@ On the left side we have the events that can trigger a pipeline based on various
 - A user clicking the "Run pipeline" button in the UI.
 - When a [merge request is created or updated](../../ci/pipelines/merge_request_pipelines.md).
 - When an MR is added to a [Merge Train](../../ci/pipelines/merge_trains.md#merge-trains).
-- A [scheduled pipeline](../../ci/pipelines/schedules.md#pipeline-schedules).
+- A [scheduled pipeline](../../ci/pipelines/schedules.md).
 - When project is [subscribed to an upstream project](../../ci/pipelines/multi_project_pipelines.md#trigger-a-pipeline-when-an-upstream-project-is-rebuilt).
 - When [Auto DevOps](../../topics/autodevops/index.md) is enabled.
 - When GitHub integration is used with [external pull requests](../../ci/ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests).
diff --git a/doc/development/cicd/schema.md b/doc/development/cicd/schema.md
new file mode 100644
index 00000000000..b63d951b881
--- /dev/null
+++ b/doc/development/cicd/schema.md
@@ -0,0 +1,146 @@
+---
+stage: Verify
+group: Pipeline Authoring
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
+type: index, howto
+---
+
+# Contribute to the CI Schema **(FREE)**
+
+The [pipeline editor](../../ci/pipeline_editor/index.md) uses a CI schema to enhance
+the authoring experience of our CI configuration files. With the CI schema, the editor can:
+
+- Validate the content of the CI configuration file as it is being written in the editor.
+- Provide autocomplete functionality and suggest available keywords.
+- Provide definitions of keywords through annotations.
+
+As the rules and keywords for configuring our CI configuration files change, so too
+should our CI schema.
+
+This feature is behind the [`schema_linting`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/feature_flags/development/schema_linting.yml)
+feature flag for self-managed instances, and is enabled for GitLab.com.
+
+## JSON Schemas
+
+The CI schema follows the [JSON Schema Draft-07](https://json-schema.org/draft-07/json-schema-release-notes.html)
+specification. Although the CI configuration file is written in YAML, it is converted
+into JSON by using `monaco-yaml` before it is validated by the CI schema.
+
+If you're new to JSON schemas, consider checking out
+[this guide](https://json-schema.org/learn/getting-started-step-by-step) for
+a step-by-step introduction on how to work with JSON schemas.
+
+## Update Keywords
+
+The CI schema is at [`app/assets/javascripts/editor/schema/ci.json`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/editor/schema/ci.json).
+It contains all the keywords available for authoring CI configuration files.
+Check the [keyword reference](../../ci/yaml/index.md) for a comprehensive list of
+all available keywords.
+
+All keywords are defined under `definitions`. We use these definitions as
+[references](https://json-schema.org/learn/getting-started-step-by-step#references)
+to share common data structures across the schema.
+
+For example, this defines the `retry` keyword:
+
+```json
+{
+  "definitions": {
+    "retry": {
+      "description": "Retry a job if it fails. Can be a simple integer or object definition.",
+      "oneOf": [
+        {
+          "$ref": "#/definitions/retry_max"
+        },
+        {
+          "type": "object",
+          "additionalProperties": false,
+          "properties": {
+            "max": {
+              "$ref": "#/definitions/retry_max"
+            },
+            "when": {
+              "description": "Either a single or array of error types to trigger job retry.",
+              "oneOf": [
+                {
+                  "$ref": "#/definitions/retry_errors"
+                },
+                {
+                  "type": "array",
+                  "items": {
+                    "$ref": "#/definitions/retry_errors"
+                  }
+                }
+              ]
+            }
+          }
+        }
+      ]
+    }
+  } 
+}
+```
+
+With this definition, the `retry` keyword is both a property of
+the `job_template` definition and the `default` global keyword. Global keywords
+that configure pipeline behavior (such as `workflow` and `stages`) are defined
+under the topmost **properties** key.
+
+```json
+{
+  "properties": {
+    "default": {
+      "type": "object",
+      "properties": {
+        "retry": {
+          "$ref": "#/definitions/retry"
+        },
+      }
+    }
+  },
+  "definitions": {
+    "job_template": {
+      "properties": {
+        "retry": {
+          "$ref": "#/definitions/retry"
+        }
+      },
+    }
+  }  
+}
+```
+
+## Guidelines for updating the schema
+
+- Keep definitions atomic when possible, to be flexible with
+  referencing keywords. For example, `workflow:rules` uses only a subset of
+  properties in the `rules` definition. The `rules` properties have their
+  own definitions, so we can reference them individually.
+- When adding new keywords, consider adding a `description` with a link to the
+  keyword definition in the documentation. This information shows up in the annotations
+  when the user hovers over the keyword.
+- For each property, consider if a `minimum`, `maximum`, or
+  `default` values are required. Some values might be required, and in others we can set
+  blank. In the blank case, we can add the following to the definition:
+
+```json
+{
+  "keyword": {
+    "oneOf": [
+      {
+        "type": "null"
+      },
+      ...
+    ]
+  }
+}
+```
+
+## Test the schema
+
+For now, the CI schema can only be tested manually. To verify the behavior is correct:
+
+1. Enable the `schema_linting` feature flag.
+1. Go to **CI/CD** > **Editor**.
+1. Write your CI/CD configuration in the editor and verify that the schema validates
+   it correctly.
diff --git a/doc/development/cicd/templates.md b/doc/development/cicd/templates.md
index d7edad842b8..c6f59a7e452 100644
--- a/doc/development/cicd/templates.md
+++ b/doc/development/cicd/templates.md
@@ -96,7 +96,7 @@ Additional points to keep in mind when authoring templates:
 |------------------------------------------------------|--------------------|---------------|
 | Can use global keywords, including `stages`.         | Yes                | No            |
 | Can define jobs.                                     | Yes                | Yes           |
-| Can be selected in the new file UI                   | Yes                | Yes           |
+| Can be selected in the new file UI                   | Yes                | No            |
 | Can include other job templates with `include`       | Yes                | No            |
 | Can include other pipeline templates with `include`. | No                 | No            |
 
@@ -105,6 +105,16 @@ Additional points to keep in mind when authoring templates:
 To make templates easier to follow, templates should all use clear syntax styles,
 with a consistent format.
 
+The `before_script`, `script`, and `after_script` keywords of every job are linted
+using [ShellCheck](https://www.shellcheck.net/) and should follow the
+[Shell scripting standards and style guidelines](../shell_scripting_guide/index.md)
+as much as possible.
+
+ShellCheck assumes that the script is designed to run using [Bash](https://www.gnu.org/software/bash/).
+Templates which use scripts for shells that aren't compatible with the Bash ShellCheck
+rules can be excluded from ShellCheck linting. To exclude a script, add it to the
+`EXCLUDED_TEMPLATES` list in [`scripts/lint_templates_bash.rb`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/scripts/lint_templates_bash.rb).
+
 #### Do not hardcode the default branch
 
 Use [`$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH`](../../ci/variables/predefined_variables.md)
@@ -219,6 +229,30 @@ job1:
     - echo ${ERROR_MESSAGE}
 ```
 
+#### Use all-caps naming for non-local variables
+
+If you are expecting a variable to be provided via the CI/CD settings, or via the
+`variables` keyword, that variable must use all-caps naming with underscores (`_`)
+separating words.
+
+```yaml
+.with_login:
+  before_script:
+    # SECRET_TOKEN should be provided via the project settings
+    - docker login -u my-user -p "$SECRET_TOKEN my-registry
+```
+
+Lower-case naming can optionally be used for variables which are defined locally in
+one of the `script` keywords:
+
+```yaml
+job1:
+  script:
+    - response="$(curl "https://example.com/json")"
+    - message="$(echo "$response" | jq -r .message)"
+    - 'echo "Server responded with: $message"'
+```
+
 ### Backward compatibility
 
 A template might be dynamically included with the `include:template:` keyword. If