diff options
Diffstat (limited to 'doc')
666 files changed, 13737 insertions, 7058 deletions
diff --git a/doc/.vale/gitlab/BadPlurals.yml b/doc/.vale/gitlab/BadPlurals.yml new file mode 100644 index 00000000000..81e2a0e563d --- /dev/null +++ b/doc/.vale/gitlab/BadPlurals.yml @@ -0,0 +1,14 @@ +--- +# Suggestion: gitlab.BadPlurals +# +# Don't write plural words with the '(s)' construction. "HTTP(S)" is acceptable. +# +# For a list of all options, see https://docs.errata.ai/vale/styles +extends: existence +message: 'Rewrite "%s" to be plural, without parentheses.' +link: https://docs.gitlab.com/ee/development/documentation/styleguide/word_list.html +level: warning +scope: raw +ignorecase: true +raw: + - '\w*\(s\)(?<!http\(s\))' diff --git a/doc/.vale/gitlab/SubstitutionSuggestions.yml b/doc/.vale/gitlab/SubstitutionSuggestions.yml index e7c0cc04244..df132d89637 100644 --- a/doc/.vale/gitlab/SubstitutionSuggestions.yml +++ b/doc/.vale/gitlab/SubstitutionSuggestions.yml @@ -17,6 +17,7 @@ swap: e-mail: '"email"' GFM: '"GitLab Flavored Markdown"' it is recommended: '"we recommend"' + navigate: go OAuth2: '"OAuth 2.0"' once that: '"after that"' once the: '"after the"' diff --git a/doc/.vale/gitlab/SubstitutionWarning.yml b/doc/.vale/gitlab/SubstitutionWarning.yml index 8000328a20c..7ee32e41ade 100644 --- a/doc/.vale/gitlab/SubstitutionWarning.yml +++ b/doc/.vale/gitlab/SubstitutionWarning.yml @@ -22,6 +22,3 @@ swap: repo: repository timezone: time zone utilize: use - administrator permission: the administrator access level - administrator permissions: the administrator access level - administrator role: the administrator access level diff --git a/doc/.vale/gitlab/Substitutions.yml b/doc/.vale/gitlab/Substitutions.yml index dde05b993ec..1cc5a60ac91 100644 --- a/doc/.vale/gitlab/Substitutions.yml +++ b/doc/.vale/gitlab/Substitutions.yml @@ -38,6 +38,12 @@ swap: can sign-in: can sign in x509: X.509 yaml: YAML + admin user: administrator + admin users: administrators + administrator permission: administrator access + administrator permissions: administrator access + administrator role: administrator access + the administrator access level: administrator access developer access: the Developer role developer permission: the Developer role developer permissions: the Developer role diff --git a/doc/administration/audit_event_streaming.md b/doc/administration/audit_event_streaming.md index eac54416924..3bdc67e5a69 100644 --- a/doc/administration/audit_event_streaming.md +++ b/doc/administration/audit_event_streaming.md @@ -8,12 +8,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332747) in GitLab 14.5 [with a flag](../administration/feature_flags.md) named `ff_external_audit_events_namespace`. Disabled by default. > - [Enabled on GitLab.com and by default on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/338939) in GitLab 14.7. - -FLAG: -On self-managed GitLab, by default this feature is available. To hide the feature per group, ask an administrator to [disable the feature flag](../administration/feature_flags.md) named `ff_external_audit_events_namespace`. On GitLab.com, this feature is available. +> - [Feature flag `ff_external_audit_events_namespace`](https://gitlab.com/gitlab-org/gitlab/-/issues/349588) removed in GitLab 14.8. Event streaming allows owners of top-level groups to set an HTTP endpoint to receive **all** audit events about the group, and its -subgroups and projects. +subgroups and projects as structured JSON. Top-level group owners can manage their audit logs in third-party systems such as Splunk, using the Splunk [HTTP Event Collector](https://docs.splunk.com/Documentation/Splunk/8.2.2/Data/UsetheHTTPEventCollector). Any service that can receive @@ -37,6 +35,7 @@ mutation { externalAuditEventDestination { destinationUrl group { + verificationToken name } } @@ -60,6 +59,7 @@ query { externalAuditEventDestinations { nodes { destinationUrl + verificationToken id } } @@ -68,3 +68,13 @@ query { ``` If the resulting list is empty, then audit event streaming is not enabled for that group. + +## Verify event authenticity + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345424) in GitLab 14.8. + +Each streaming destination has a unique verification token (`verificationToken`) that can be used to verify the authenticity of the event. This +token is generated when the event destination is created and cannot be changed. + +Each streamed event contains a random alphanumeric identifier for the `X-Gitlab-Event-Streaming-Token` HTTP header that can be verified against +the destination's value when [listing streaming destinations](#list-currently-enabled-streaming-destinations). diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index 06ad16bbcba..d4902a18cac 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -51,7 +51,10 @@ There are two kinds of events logged: When a user is being [impersonated](../user/admin_area/index.md#user-impersonation), their actions are logged as audit events as usual, with two additional details: 1. Usual audit events include information about the impersonating administrator. These are visible in their respective Audit Event pages depending on their type (Group/Project/User). -1. Extra audit events are recorded for the start and stop of the administrator's impersonation session. These are visible in the instance Audit Events. +1. Extra audit events are recorded for the start and stop of the administrator's impersonation session. These are visible in + the: + - Instance audit events. + - Group audit events for all groups the user belongs to (GitLab 14.8 and later). This is limited to 20 groups for performance reasons.  @@ -103,6 +106,7 @@ From there, you can see the following actions: - Group CI/CD variable added, removed, or protected status changed. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30857) in GitLab 13.3. - Compliance framework created, updated, or deleted. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340649) in GitLab 14.5. - Event streaming destination created, updated, or deleted. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/344664) in GitLab 14.6. +- Instance administrator started or stopped impersonation of a group member. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/300961) in GitLab 14.8. Group events can also be accessed via the [Group Audit Events API](../api/audit_events.md#group-audit-events) @@ -236,10 +240,13 @@ Don't see the event you want in any of the epics linked above? You can either: ### Disabled events -#### Repository push +#### Repository push (DEPRECATED) > [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/337993) in GitLab 14.3. +WARNING: +This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/337993) in GitLab 14.3. + The current architecture of audit events is not prepared to receive a very high amount of records. It may make the user interface for your project or audit events very busy, and the disk space consumed by the `audit_events` PostgreSQL table may increase considerably. It's disabled by default diff --git a/doc/administration/auditor_users.md b/doc/administration/auditor_users.md index 5498ea9d4be..12e222290e7 100644 --- a/doc/administration/auditor_users.md +++ b/doc/administration/auditor_users.md @@ -64,7 +64,7 @@ To create an Auditor user: 1. Select **Create user** or **Save changes** if you created a new user or edited an existing one respectively. -To revoke Auditor permissions from a user, make them a regular user by +To revoke Auditor permissions from a user, make them a Regular user by following the previous steps. Additionally users can be set as an Auditor using [SAML groups](../integration/saml.md#auditor-groups). diff --git a/doc/administration/auth/atlassian.md b/doc/administration/auth/atlassian.md index 08d07b13e22..7c6aee2c2da 100644 --- a/doc/administration/auth/atlassian.md +++ b/doc/administration/auth/atlassian.md @@ -1,7 +1,7 @@ --- type: reference stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/auth/authentiq.md b/doc/administration/auth/authentiq.md index 23735b75991..a4a085a494c 100644 --- a/doc/administration/auth/authentiq.md +++ b/doc/administration/auth/authentiq.md @@ -1,7 +1,7 @@ --- type: reference stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/auth/cognito.md b/doc/administration/auth/cognito.md index 2509bbb73d7..66f8407894c 100644 --- a/doc/administration/auth/cognito.md +++ b/doc/administration/auth/cognito.md @@ -1,7 +1,7 @@ --- type: concepts, howto stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/auth/crowd.md b/doc/administration/auth/crowd.md index 9a5f6b0a642..8dfa832a103 100644 --- a/doc/administration/auth/crowd.md +++ b/doc/administration/auth/crowd.md @@ -1,7 +1,7 @@ --- type: reference stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/auth/index.md b/doc/administration/auth/index.md index d469988e719..d09344629e2 100644 --- a/doc/administration/auth/index.md +++ b/doc/administration/auth/index.md @@ -2,7 +2,7 @@ comments: false type: index stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -50,3 +50,31 @@ For more information, see the links shown on this page for each external provide | **Authentication** | SAML at top-level group (1 provider) | LDAP (multiple providers)<br>Generic OAuth2<br>SAML (only 1 permitted per unique provider)<br>Kerberos<br>JWT<br>Smartcard<br>OmniAuth Providers (only 1 permitted per unique provider) | | **Provider-to-GitLab Role Sync** | SAML Group Sync | LDAP Group Sync | | **User Removal** | SCIM (remove user from top-level group) | LDAP (Blocking User from Instance) | + +## Change apps or configuration + +When GitLab doesn't support having multiple providers (such as OAuth), GitLab configuration and user identification must be +updated at the same time if the provider or app is changed. + +These instructions apply to all methods of authentication where GitLab stores an `extern_uid` and it is the only data used +for user authentication. + +When changing apps within a provider, if the user `extern_uid` does not change, only the GitLab configuration must be +updated. + +To swap configurations: + +1. Change provider configuration in your `gitlab.rb` file. +1. Update `extern_uid` for all users that have an identity in GitLab for the previous provider. + +To find the `extern_uid`, look at an existing user's current `extern_uid` for an ID that matches the appropriate field in +your current provider for the same user. + +There are two methods to update the `extern_uid`: + +- Using the [Users API](../../api/users.md#user-modification). Pass the provider name and the new `extern_uid`. +- Using the [Rails console](../operations/rails_console.md): + + ```ruby + Identity.where(extern_uid: 'old-id').update!(extern_uid: 'new-id')` + ``` diff --git a/doc/administration/auth/jwt.md b/doc/administration/auth/jwt.md index ac2f699dd5d..98f928fd7ee 100644 --- a/doc/administration/auth/jwt.md +++ b/doc/administration/auth/jwt.md @@ -1,7 +1,7 @@ --- type: reference stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/auth/ldap/google_secure_ldap.md b/doc/administration/auth/ldap/google_secure_ldap.md index 6b8e699e861..10745c5e2bf 100644 --- a/doc/administration/auth/ldap/google_secure_ldap.md +++ b/doc/administration/auth/ldap/google_secure_ldap.md @@ -1,7 +1,7 @@ --- type: reference stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index b773281b216..2a396c4d53a 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -1,7 +1,7 @@ --- type: reference stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -468,7 +468,7 @@ If initially your LDAP configuration looked like: password: '123' ``` -1. Edit `/etc/gitlab/gitlab.rb` and remove the settings for `user_bn` and `password`. +1. Edit `/etc/gitlab/gitlab.rb` and remove the settings for `user_dn` and `password`. 1. [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -502,7 +502,7 @@ If initially your LDAP configuration looked like: password: '123' ``` -1. Edit `config/gitlab.yaml` and remove the settings for `user_bn` and `password`. +1. Edit `config/gitlab.yaml` and remove the settings for `user_dn` and `password`. 1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md index 97a626f4a40..06fe579e101 100644 --- a/doc/administration/auth/ldap/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap/ldap-troubleshooting.md @@ -1,7 +1,7 @@ --- type: reference stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -382,7 +382,7 @@ the following are true: - The configured `admin_group` in the `gitlab.rb` is a CN, rather than a DN or an array. - This CN falls under the scope of the configured `group_base`. - The members of the `admin_group` have already signed into GitLab with their LDAP - credentials. GitLab only grants the Administrator role to the users whose + credentials. GitLab only grants administrator access to the users whose accounts are already connected to LDAP. If all the above are true and the users are still not getting access, diff --git a/doc/administration/auth/ldap/ldap_synchronization.md b/doc/administration/auth/ldap/ldap_synchronization.md index 3329ea8e9cc..d7ce54a47c0 100644 --- a/doc/administration/auth/ldap/ldap_synchronization.md +++ b/doc/administration/auth/ldap/ldap_synchronization.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -347,3 +347,7 @@ These metrics are meant to provide a baseline and performance may vary based on any number of factors. This benchmark was extreme and most instances don't have near this many users or groups. Disk speed, database performance, network and LDAP server response time affects these metrics. + +## Troubleshooting + +See our [administrator guide to troubleshooting LDAP](ldap-troubleshooting.md). diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md index efe4b7440ee..a099b9c76f8 100644 --- a/doc/administration/auth/oidc.md +++ b/doc/administration/auth/oidc.md @@ -1,7 +1,7 @@ --- type: reference stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -461,6 +461,65 @@ To use symmetric key encryption: If after reconfiguring, you see the error `JSON::JWS::VerificationFailed` error message, this means the incorrect secret was specified. +#### Casdoor + +GitLab works with OpenID providers that use HTTPS. To connect to GitLab using OpenID with Casdoor, use HTTPS instead of HTTP. + +For your app, complete the following steps on Casdoor: + +1. Get a client ID and a client secret. +1. Add your GitLab redirect URL. For example, if your GitLab domain is `gitlab.example.com`, ensure the Casdoor app has the following + `Redirect URI`: `https://gitlab.example.com/users/auth/openid_connect/callback`. + +See the [Casdoor documentation](https://casdoor.org/docs/integration/gitlab) for more details. + +Example Omnibus GitLab configuration (file path: `/etc/gitlab/gitlab.rb`): + +```ruby +gitlab_rails['omniauth_providers'] = [ + { + name: "openid_connect", + label: "Casdoor", # optional label for login button, defaults to "Openid Connect" + args: { + name: "openid_connect", + scope: ["openid", "profile", "email"], + response_type: "code", + issuer: "https://<CASDOOR_HOSTNAME>", + client_auth_method: "query", + discovery: true, + uid_field: "sub", + client_options: { + identifier: "<YOUR CLIENT ID>", + secret: "<YOUR CLIENT SECRET>", + redirect_uri: "https://gitlab.example.com/users/auth/openid_connect/callback" + } + } + } +] +``` + +Example installations from source configuration (file path: `config/gitlab.yml`): + +```yaml + - { name: 'openid_connect', + label: 'Casdoor', # optional label for login button, defaults to "Openid Connect" + args: { + name: 'openid_connect', + scope: ['openid','profile','email'], + response_type: 'code', + issuer: 'https://<CASDOOR_HOSTNAME>', + discovery: true, + client_auth_method: 'query', + uid_field: 'sub', + client_options: { + identifier: '<YOUR CLIENT ID>', + secret: '<YOUR CLIENT SECRET>', + redirect_uri: 'https://gitlab.example.com/users/auth/openid_connect/callback' + } + } + } +``` + ## General troubleshooting If you're having trouble, here are some tips: diff --git a/doc/administration/auth/smartcard.md b/doc/administration/auth/smartcard.md index f77d3a36a7b..3f0d95967bf 100644 --- a/doc/administration/auth/smartcard.md +++ b/doc/administration/auth/smartcard.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- diff --git a/doc/administration/clusters/kas.md b/doc/administration/clusters/kas.md index ba0d7280b74..192b636e246 100644 --- a/doc/administration/clusters/kas.md +++ b/doc/administration/clusters/kas.md @@ -4,13 +4,15 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Install the GitLab Agent Server (KAS) **(FREE SELF)** +# Install the GitLab Agent Server for Kubernetes (KAS) **(FREE SELF)** > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3834) in GitLab 13.10, the GitLab Agent Server (KAS) became available on GitLab.com under `wss://kas.gitlab.com`. -> [Moved](https://gitlab.com/groups/gitlab-org/-/epics/6290) from GitLab Premium to GitLab Free in 14.5. +> - [Moved](https://gitlab.com/groups/gitlab-org/-/epics/6290) from GitLab Premium to GitLab Free in 14.5. -The GitLab Agent Server (KAS) is a GitLab backend service dedicated to -managing the [GitLab Agent](../../user/clusters/agent/index.md). +The GitLab Agent Server for Kubernetes is a GitLab backend service dedicated to +managing the [GitLab Agent for Kubernetes](../../user/clusters/agent/index.md). + +The KAS acronym refers to the former name, Kubernetes Agent Server. The KAS is already installed and available in GitLab.com under `wss://kas.gitlab.com`. This document describes how to install a KAS for GitLab self-managed instances. @@ -94,8 +96,8 @@ For GitLab instances installed through Omnibus packages: ## Troubleshooting -If you face any issues with KAS, you can read the service logs -with the following command: +If you have issues while using the GitLab Agent Server for Kubernetes, view the +service logs by running the following command: ```shell kubectl logs -f -l=app=kas -n <YOUR-GITLAB-NAMESPACE> @@ -103,8 +105,7 @@ kubectl logs -f -l=app=kas -n <YOUR-GITLAB-NAMESPACE> In Omnibus GitLab, find the logs in `/var/log/gitlab/gitlab-kas/`. -See also the [user documentation](../../user/clusters/agent/index.md#troubleshooting) -for troubleshooting problems with individual agents. +You can also [troubleshoot issues with individual Agents](../../user/clusters/agent/troubleshooting.md). ### KAS logs - GitOps: failed to get project information diff --git a/doc/administration/compliance.md b/doc/administration/compliance.md index 843d5d0930a..7c3f36c71c2 100644 --- a/doc/administration/compliance.md +++ b/doc/administration/compliance.md @@ -16,7 +16,7 @@ relevant compliance standards. ## Policy management Organizations have unique policy requirements, either due to organizational -standards or mandates from regulatory bodies. The following features help you +standards or mandates from regulatory bodies. The following features help you define rules and policies to adhere to workflow requirements, separation of duties, and secure supply chain best practices: @@ -31,7 +31,7 @@ and secure supply chain best practices: - [**Merge request approvals**](../user/project/merge_requests/approvals/index.md) (for instances, groups, and projects): Configure approvals required for merge requests. -- [**Push rules**](../push_rules/push_rules.md) (for instances, groups, and +- [**Push rules**](../user/project/repository/push_rules.md) (for instances, groups, and projects): Control pushes to your repositories. - Separation of duties using [**protected branches**](../user/project/protected_branches.md#require-code-owner-approval-on-a-protected-branch) and [**custom CI/CD configuration paths**](../ci/pipelines/settings.md#specify-a-custom-cicd-configuration-file) (for projects): Users can leverage the GitLab cross-project YAML configurations diff --git a/doc/administration/docs_self_host.md b/doc/administration/docs_self_host.md index 007055b5de7..3a4511e5aa4 100644 --- a/doc/administration/docs_self_host.md +++ b/doc/administration/docs_self_host.md @@ -4,77 +4,69 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# How to self-host the docs site **(FREE SELF)** - -If you have a self-managed instance of GitLab, you may not be able to access the -product documentation as hosted on `docs.gitlab.com` from your GitLab instance. - -Be aware of the following items if you self-host the product documentation: - -- You must host the product documentation site under a subdirectory that matches - your installed GitLab version (for example, `14.5/`). The - [Docker images](https://gitlab.com/gitlab-org/gitlab-docs/container_registry/631635) - hosted by the GitLab Docs team provide this by default. We use a - [script](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/2995d1378175803b22fb8806ba77adf63e79f32c/scripts/normalize-links.sh#L28-82) - to normalize the links and prefix them with the respective version. -- The version dropdown will display additional versions that don't exist, selecting - those versions will display a 404 Not Found page. -- Results when using the search box will display results from `docs.gitlab.com` - and not the local documentation. -- When you use the Docker images to serve the product documentation site, by - default the landing page redirects to the respective version (for example, `/14.5/`), - which causes the landing page <https://docs.gitlab.com> to not be displayed. +# How to host the GitLab product documentation **(FREE SELF)** + +If you are not able to access the GitLab product documentation at `docs.gitlab.com`, +you can host the documentation yourself instead. ## Documentation self-hosting options -You can self-host the GitLab product documentation locally using one of these -methods: +To host the GitLab product documentation, you can use: -- Docker +- A Docker container - GitLab Pages -- From your own webserver +- Your own web server + +After you create a website by using one of these methods, you redirect the UI links +in the product to point to your website. -The examples on this page are based on GitLab 14.5. +NOTE: +The website you create must be hosted under a subdirectory that matches +your installed GitLab version (for example, `14.5/`). The +[Docker images](https://gitlab.com/gitlab-org/gitlab-docs/container_registry/631635) +use this version by default. + +The following examples use GitLab 14.5. ### Self-host the product documentation with Docker -The Docker images use a built-in webserver listening on port `4000`, so you need -to expose that. +You can run the GitLab product documentation website in a Docker container: -In the server that you host GitLab, or any other server that your GitLab instance -can talk to, you can use Docker to pull the docs site: +1. Expose port `4000`. The Docker image uses this port for the web server. +1. On the server where you host GitLab, or on any other server that your GitLab instance + can communicate with, pull the docs site: -```shell -docker run -it --rm -p 4000:4000 registry.gitlab.com/gitlab-org/gitlab-docs:14.5 -``` + ```shell + docker run -it --rm -p 4000:4000 registry.gitlab.com/gitlab-org/gitlab-docs:14.5 + ``` -If you use [Docker compose](../install/docker.md#install-gitlab-using-docker-compose) -to host your GitLab instance, add the following to `docker-compose.yaml`: - -```yaml -version: '3.6' -services: - docs: - image: registry.gitlab.com/gitlab-org/gitlab-docs:14.5 - hostname: 'https://gitlab.example.com' - ports: - - '4000:4000' -``` + If you host your GitLab instance using [Docker compose](../install/docker.md#install-gitlab-using-docker-compose), + add the following to `docker-compose.yaml`: + + ```yaml + version: '3.6' + services: + docs: + image: registry.gitlab.com/gitlab-org/gitlab-docs:14.5 + hostname: 'https://gitlab.example.com' + ports: + - '4000:4000' + ``` ### Self-host the product documentation with GitLab Pages -You use GitLab Pages to host the GitLab product documentation locally. +You can use GitLab Pages to host the GitLab product documentation. Prerequisite: -- The Pages site URL must not use a subfolder. Due to the nature of how the docs +- Ensure the Pages site URL does not use a subfolder. Because of how the docs site is pre-compiled, the CSS and JavaScript files are relative to the main domain or subdomain. For example, URLs like `https://example.com/docs/` are not supported. To host the product documentation site with GitLab Pages: -1. [Create a new blank project](../user/project/working_with_projects.md#create-a-blank-project). +1. [Create a blank project](../user/project/working_with_projects.md#create-a-blank-project). 1. Create a new or edit your existing `.gitlab-ci.yml` file, and add the following `pages` job, while ensuring the version is the same as your GitLab installation: @@ -97,13 +89,13 @@ To host the product documentation site with GitLab Pages: | [Project website](../user/project/pages/getting_started_part_one.md#project-website-examples) | Not supported | Supported | | [User or group website](../user/project/pages/getting_started_part_one.md#user-and-group-website-examples) | Supported | Supported | -### Self-host the product documentation on your own webserver +### Self-host the product documentation on your own web server -Because the product documentation site is static, you can grab the directory from -the container (in `/usr/share/nginx/html`) and use your own web server to host -it wherever you want. +Because the product documentation site is static, from the container, you can take the contents +of `/usr/share/nginx/html` and use your own web server to host +the docs wherever you want. -Use the following commands, and replace `<destination>` with the directory where the +Run the following commands, replacing `<destination>` with the directory where the documentation files will be copied to: ```shell @@ -114,18 +106,30 @@ docker rm -f gitlab-docs ## Redirect the `/help` links to the new docs page -After your local product documentation site is running, [redirect the help -links](../user/admin_area/settings/help_page.md#redirect-help-pages) in the GitLab -application to your local site. +After your local product documentation site is running, +[redirect the help links](../user/admin_area/settings/help_page.md#redirect-help-pages) +in the GitLab application to your local site. Be sure to use the fully qualified domain name as the docs URL. For example, if you used the [Docker method](#self-host-the-product-documentation-with-docker), enter `http://0.0.0.0:4000`. -You don't need to append the version, as GitLab will detect it and append it to -any documentation URL requests, as needed. For example, if your GitLab version is -14.5, the GitLab Docs URL becomes `http://0.0.0.0:4000/14.5/`. The link -inside GitLab displays as `<instance_url>/help/user/admin_area/settings/help_page#destination-requirements`, -but when you select it, you are redirected to +You don't need to append the version. GitLab detects it and appends it to +documentation URL requests as needed. For example, if your GitLab version is +14.5: + +- The GitLab Docs URL becomes `http://0.0.0.0:4000/14.5/`. +- The link in GitLab displays as `<instance_url>/help/user/admin_area/settings/help_page#destination-requirements`. +- When you select the link, you are redirected to `http://0.0.0.0:4000/14.5/ee/user/admin_area/settings/help_page/#destination-requirements`. To test the setting, select a **Learn more** link within the GitLab application. + +## Known issues + +If you self-host the product documentation: + +- The version dropdown displays additional versions that don't exist. Selecting + these versions displays a `404 Not Found` page. +- The search displays results from `docs.gitlab.com` and not the local site. +- By default, the landing page redirects to the + respective version (for example, `/14.5/`). This causes the landing page <https://docs.gitlab.com> to not be displayed. diff --git a/doc/administration/feature_flags.md b/doc/administration/feature_flags.md index afbf0759452..85a7304b5e8 100644 --- a/doc/administration/feature_flags.md +++ b/doc/administration/feature_flags.md @@ -36,7 +36,7 @@ For example, data is not recorded and services do not run. If you used a certain feature and identified a bug, a misbehavior, or an error, it's very important that you [**provide feedback**](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issue[title]=Docs%20-%20feature%20flag%20feedback%3A%20Feature%20Name&issue[description]=Describe%20the%20problem%20you%27ve%20encountered.%0A%0A%3C!--%20Don%27t%20edit%20below%20this%20line%20--%3E%0A%0A%2Flabel%20~%22docs%5C-comments%22%20) to GitLab as soon as possible so we can improve or fix it while behind a flag. When you upgrade -GitLab to an earlier version, the feature flag status may change. +GitLab, the feature flag status may change. ## Risks when enabling features still in development @@ -130,11 +130,15 @@ irb(main):001:0> Feature.enable(:my_awesome_feature) => nil ``` -To check if a flag is enabled or disabled you can use `Feature.enabled?` or `Feature.disabled?`. For example, for a fictional feature flag named `my_awesome_feature`: +When the feature is ready, GitLab removes the feature flag, and the option for +enabling and disabling it no longer exists. The feature becomes available in all instances. + +### Check if a feature flag is enabled + +To check if a flag is enabled or disabled, use `Feature.enabled?` or `Feature.disabled?`. +For example, for a feature flag named `my_awesome_feature` that is already enabled: ```ruby -Feature.enable(:my_awesome_feature) -=> nil Feature.enabled?(:my_awesome_feature) => true Feature.disabled?(:my_awesome_feature) @@ -143,3 +147,21 @@ Feature.disabled?(:my_awesome_feature) When the feature is ready, GitLab removes the feature flag, and the option for enabling and disabling it no longer exists. The feature becomes available in all instances. + +### View set feature flags + +You can view all GitLab administrator set feature flags: + +```ruby +Feature.all +=> [#<Flipper::Feature:198220 name="my_awesome_feature", state=:on, enabled_gate_names=[:boolean], adapter=:memoizable>] +``` + +### Unset feature flag + +You can unset a feature flag so that GitLab will fall back to the current defaults for that flag: + +```ruby +Feature.remove(:my_awesome_feature) +=> true +``` diff --git a/doc/administration/geo/glossary.md b/doc/administration/geo/glossary.md index f8769d31ec7..a18e78a5e01 100644 --- a/doc/administration/geo/glossary.md +++ b/doc/administration/geo/glossary.md @@ -25,8 +25,8 @@ these definitions yet. | Site | One or a collection of nodes running a single GitLab application. A site can be single-node or multi-node. | GitLab | deployment, installation instance | | Single-node site | A specific configuration of GitLab that uses exactly one node. | GitLab | single-server, single-instance | Multi-node site | A specific configuration of GitLab that uses more than one node. | GitLab | multi-server, multi-instance, high availability | -| Primary site | A GitLab site that is configured to be read and writable. There can only be a single primary site. | Geo-specific | Geo deployment, Primary node | -| Secondary site(s) | GitLab site that is configured to be read-only. There can be one or more secondary sites. | Geo-specific | Geo deployment, Secondary node | +| Primary site | A GitLab site whose data is being replicated by at least one secondary site. There can only be a single primary site. | Geo-specific | Geo deployment, Primary node | +| Secondary site(s) | A GitLab site that is configured to replicate the data of a primary site. There can be one or more secondary sites. | Geo-specific | Geo deployment, Secondary node | | Geo deployment | A collection of two or more GitLab sites with exactly one primary site being replicated by one or more secondary sites. | Geo-specific | | | Reference architecture(s) | A [specified configuration of GitLab for a number of users](../reference_architectures/index.md), possibly including multiple nodes and multiple sites. | GitLab | | | Promoting | Changing the role of a site from secondary to primary. | Geo-specific | | diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index b5eb0ba5841..c4164284e97 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -204,6 +204,7 @@ This list of limitations only reflects the latest version of GitLab. If you are - Real-time updates of issues/merge requests (for example, via long polling) doesn't work on the **secondary** site. - GitLab Runners cannot register with a **secondary** site. Support for this is [planned for the future](https://gitlab.com/gitlab-org/gitlab/-/issues/3294). - [Selective synchronization](replication/configuration.md#selective-synchronization) only limits what repositories and files are replicated. The entire PostgreSQL data is still replicated. Selective synchronization is not built to accommodate compliance / export control use cases. +- [Pages access control](../../user/project/pages/pages_access_control.md) doesn't work on secondaries. See [GitLab issue #9336](https://gitlab.com/gitlab-org/gitlab/-/issues/9336) for details. ### Limitations on replication/verification diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index 0fa08dcc9e0..5beb2479c57 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -39,7 +39,7 @@ verification methods: | Blobs | User uploads _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | LFS objects _(file system)_ | Geo with API | SHA256 checksum | | Blobs | LFS objects _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | -| Blobs | CI job artifacts _(file system)_ | Geo with API | _Not implemented_ | +| Blobs | CI job artifacts _(file system)_ | Geo with API | SHA256 checksum | | Blobs | CI job artifacts _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | Archived CI build traces _(file system)_ | Geo with API | _Not implemented_ | | Blobs | Archived CI build traces _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md index ce1bd8a9d3c..70b3eaf5fea 100644 --- a/doc/administration/geo/replication/geo_validation_tests.md +++ b/doc/administration/geo/replication/geo_validation_tests.md @@ -118,7 +118,7 @@ The following are PostgreSQL upgrade validation tests we performed. [Verify Geo installation with PostgreSQL 13](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6131): - Description: With PostgreSQL 13 available as an opt-in version in GitLab 14.1, we tested fresh installations of GitLab with Geo when PostgreSQL 13 is enabled. -- Outcome: Successfully built an environment with Geo and PostgreSQL 13 using [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit) and performed Geo QA tests against the environment without failures. +- Outcome: Successfully built an environment with Geo and PostgreSQL 13 using [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/gitlab-environment-toolkit) and performed Geo QA tests against the environment without failures. ### September 2020 @@ -200,3 +200,12 @@ The following are additional validation tests we performed. - Description: Tested a Geo deployment with Gitaly clusters configured on both the primary and secondary Geo sites. Triggered automatic Gitaly cluster failover on the primary Geo site, and ran end-to-end Geo tests. Then triggered Gitaly cluster failover on the secondary Geo site, and re-ran the end-to-end Geo tests. - Outcome: Successful end-to-end tests before and after Gitaly cluster failover on the primary site, and before and after Gitaly cluster failover on the secondary site. + +### January 2022 + +[Validate Object storage replication using Azure based object storage](https://gitlab.com/gitlab-org/gitlab/-/issues/348804#note_821294631): + +- Description: Tested the average time it takes for a single image to replicate from the primary object storage location to the secondary when using Azure based object storage replication and [GitLab based object storage replication](object_storage.md#enabling-gitlab-managed-object-storage-replication). This was tested by uploading a 1mb image to a project on the primary site every second for 60 seconds. The time was then measured until a image was available on the secondary site. This was achieved using a [Ruby Script](https://gitlab.com/gitlab-org/quality/geo-replication-tester). +- Outcome: When using Azure based replication the average time for an image to replicate from the primary object storage to the secondary was recorded as 40 seconds, the longest replication time was 70 seconds and the quickest was 11 seconds. When using GitLab based replication the average time for replication to complete was 5 seconds, the longest replication time was 10 seconds and the quickest was 3 seconds. +- Follow up issue: + - [Test and validate object storage replication performance on reference architectures](https://gitlab.com/gitlab-org/gitlab/-/issues/347314) diff --git a/doc/administration/geo/replication/location_aware_git_url.md b/doc/administration/geo/replication/location_aware_git_url.md index ce1af27611e..8689fac987f 100644 --- a/doc/administration/geo/replication/location_aware_git_url.md +++ b/doc/administration/geo/replication/location_aware_git_url.md @@ -7,6 +7,12 @@ type: howto # Location-aware Git remote URL with AWS Route53 **(PREMIUM SELF)** +NOTE: +Since GitLab 14.6, +[GitLab Geo supports a location-aware URL including web UI and API traffic.](../secondary_proxy/location_aware_external_url.md) +This configuration is recommended over the location-aware Git remote URL +described in this document. + You can provide GitLab users with a single remote URL that automatically uses the Geo site closest to them. This means users don't need to update their Git configuration to take advantage of closer Geo sites as they move. @@ -18,12 +24,6 @@ Though these instructions use [AWS Route53](https://aws.amazon.com/route53/), other services such as [Cloudflare](https://www.cloudflare.com/) could be used as well. -NOTE: -You can also use a load balancer to distribute web UI or API traffic to -[multiple Geo **secondary** sites](../../../user/admin_area/geo_nodes.md#multiple-secondary-sites-behind-a-load-balancer). -Importantly, the **primary** site cannot yet be included. See the feature request -[Support putting the **primary** behind a Geo node load balancer](https://gitlab.com/gitlab-org/gitlab/-/issues/10888). - ## Prerequisites In this example, we have already set up: diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index a6ea41171a9..1371e5d84c8 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -115,35 +115,39 @@ http://secondary.example.com/ To check if PostgreSQL replication is working, check if: -- [Nodes are pointing to the correct database instance](#are-nodes-pointing-to-the-correct-database-instance). -- [Geo can detect the current node correctly](#can-geo-detect-the-current-node-correctly). +- [Sites are pointing to the correct database node](#are-sites-pointing-to-the-correct-database-node). +- [Geo can detect the current site correctly](#can-geo-detect-the-current-site-correctly). -#### Are nodes pointing to the correct database instance? +#### Are sites pointing to the correct database node? -You should make sure your **primary** Geo node points to the instance with -writing permissions. +You should make sure your **primary** Geo [site](../glossary.md) points to +the database node that has write permissions. -Any **secondary** nodes should point only to read-only instances. +Any **secondary** sites should point only to read-only database nodes. -#### Can Geo detect the current node correctly? +#### Can Geo detect the current site correctly? -Geo finds the current machine's Geo node name in `/etc/gitlab/gitlab.rb` by: +Geo finds the current Puma or Sidekiq node's Geo [site](../glossary.md) name in +`/etc/gitlab/gitlab.rb` with the following logic: -- Using the `gitlab_rails['geo_node_name']` setting. -- If that is not defined, using the `external_url` setting. +1. Get the "Geo node name" (there is + [an issue to rename the settings to "Geo site name"](https://gitlab.com/gitlab-org/gitlab/-/issues/335944)): + - Omnibus GitLab: Get the `gitlab_rails['geo_node_name']` setting. + - GitLab Helm Charts: Get the `global.geo.nodeName` setting (see [Charts with GitLab Geo](https://docs.gitlab.com/charts/advanced/geo/index.html)). +1. If that is not defined, then get the `external_url` setting. -This name is used to look up the node with the same **Name** in the **Geo Nodes** +This name is used to look up the Geo site with the same **Name** in the **Geo Sites** dashboard. -To check if the current machine has a node name that matches a node in the +To check if the current machine has a site name that matches a site in the database, run the check task: ```shell sudo gitlab-rake gitlab:geo:check ``` -It displays the current machine's node name and whether the matching database -record is a **primary** or **secondary** node. +It displays the current machine's site name and whether the matching database +record is a **primary** or **secondary** site. ```plaintext This machine's Geo node name matches a database record ... yes, found a secondary node named "Shanghai" @@ -158,6 +162,9 @@ This machine's Geo node name matches a database record ... no doc/administration/geo/replication/troubleshooting.md#can-geo-detect-the-current-node-correctly ``` +Learn more about recommended site names in the description of the Name field in +[Geo Admin Area Common Settings](../../../user/admin_area/geo_nodes.md#common-settings). + ### Message: `WARNING: oldest xmin is far in the past` and `pg_wal` size growing If a replication slot is inactive, @@ -692,6 +699,8 @@ determine the actual replication status of Design repositories. ### Sync failure message: "Verification failed with: Error during verification: File is not checksummable" +#### Missing files on the Geo primary site + In GitLab 14.5 and earlier, certain data types which were missing on the Geo primary site were marked as "synced" on Geo secondary sites. This was because from the perspective of Geo secondary sites, the state matched the primary site and nothing more could be done on secondary sites. Secondaries would regularly try to sync these files again by using the "verification" feature: @@ -745,6 +754,32 @@ This behavior affects only the following data types through GitLab 14.6: to make Geo visibly surface data loss risks. The sync/verification loop is therefore short-circuited. `last_sync_failure` is now set to `The file is missing on the Geo primary site`. +#### Failed syncs with GitLab-managed object storage replication + +There is [an issue in GitLab 14.2 through 14.7](https://gitlab.com/gitlab-org/gitlab/-/issues/299819#note_822629467) +that affects Geo when the GitLab-managed object storage replication is used, causing blob object types to fail synchronization. + +Since GitLab 14.2, verification failures result in synchronization failures and cause +a re-synchronization of these objects. + +As verification is not implemented for files stored in object storage (see +[issue 13845](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) for more details), this +results in a loop that consistently fails for all objects stored in object storage. + +You can work around this by marking the objects as synced and succeeded verification, however +be aware that can also mark objects that may be +[missing from the primary](#missing-files-on-the-geo-primary-site). + +To do that, enter the [Rails console](../../troubleshooting/navigating_gitlab_via_rails_console.md) +and run: + +```ruby +Gitlab::Geo.verification_enabled_replicator_classes.each do |klass| + updated = klass.registry_class.failed.where(last_sync_failure: "Verification failed with: Error during verification: File is not checksummable").update_all(verification_checksum: '0000000000000000000000000000000000000000', verification_state: 2, verification_failure: nil, verification_retry_at: nil, state: 2, last_sync_failure: nil, retry_at: nil, verification_retry_count: 0, retry_count: 0) + pp "Updated #{updated} #{klass.replicable_name_plural}" +end +``` + ## Fixing errors during a failover or when promoting a secondary to a primary node The following are possible error messages that might be encountered during failover or diff --git a/doc/administration/geo/replication/using_a_geo_server.md b/doc/administration/geo/replication/using_a_geo_server.md index 04c30514a89..62562a1149d 100644 --- a/doc/administration/geo/replication/using_a_geo_server.md +++ b/doc/administration/geo/replication/using_a_geo_server.md @@ -1,9 +1,9 @@ --- redirect_to: '../../geo/replication/usage.md' -remove_date: '2022-06-01' +remove_date: '2022-02-01' --- This document was moved to [another location](../../geo/replication/usage.md). -<!-- This redirect file can be deleted after 2022-06-01 --> +<!-- This redirect file can be deleted after 2022-02-01 --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/geo/replication/version_specific_updates.md b/doc/administration/geo/replication/version_specific_updates.md index d3a132a6666..95833a290dd 100644 --- a/doc/administration/geo/replication/version_specific_updates.md +++ b/doc/administration/geo/replication/version_specific_updates.md @@ -8,7 +8,22 @@ info: To determine the technical writer assigned to the Stage/Group associated w Review this page for update instructions for your version. These steps accompany the [general steps](updating_the_geo_sites.md#general-update-steps) -for updating Geo nodes. +for updating Geo sites. + +## Updating to 14.2 through 14.7 + +There is [an issue in GitLab 14.2 through 14.7](https://gitlab.com/gitlab-org/gitlab/-/issues/299819#note_822629467) +that affects Geo when the GitLab-managed object storage replication is used, causing blob object types to fail synchronization. + +Since GitLab 14.2, verification failures result in synchronization failures and cause +a resynchronization of these objects. + +As verification is not yet implemented for files stored in object storage (see +[issue 13845](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) for more details), this +results in a loop that consistently fails for all objects stored in object storage. + +For information on how to fix this, see +[Troubleshooting - Failed syncs with GitLab-managed object storage replication](troubleshooting.md#failed-syncs-with-gitlab-managed-object-storage-replication). ## Updating to 14.4 @@ -18,7 +33,7 @@ There is [an issue in GitLab 14.4.0 through 14.4.2](../../../update/index.md#144 ### Multi-arch images -We found an [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/336013) where the Container Registry replication wasn't fully working if you used multi-arch images. In case of a multi-arch image, only the primary architecture (for example `amd64`) would be replicated to the secondary node. This has been [fixed in GitLab 14.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67624) and was backported to 14.2 and 14.1, but manual steps are required to force a re-sync. +We found an [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/336013) where the Container Registry replication wasn't fully working if you used multi-arch images. In case of a multi-arch image, only the primary architecture (for example `amd64`) would be replicated to the secondary site. This has been [fixed in GitLab 14.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67624) and was backported to 14.2 and 14.1, but manual steps are required to force a re-sync. You can check if you are affected by running: @@ -26,12 +41,12 @@ You can check if you are affected by running: docker manifest inspect <SECONDARY_IMAGE_LOCATION> | jq '.mediaType' ``` -Where `<SECONDARY_IMAGE_LOCATION>` is a container image on your secondary node. +Where `<SECONDARY_IMAGE_LOCATION>` is a container image on your secondary site. If the output matches `application/vnd.docker.distribution.manifest.list.v2+json` (there can be a `mediaType` entry at several levels, we only care about the top level entry), then you don't need to do anything. -Otherwise, on all your **secondary** nodes, in a [Rails console](../../operations/rails_console.md), run the following: +Otherwise, for each **secondary** site, on a Rails application node, open a [Rails console](../../operations/rails_console.md), and run the following: ```ruby list_type = 'application/vnd.docker.distribution.manifest.list.v2+json' @@ -63,7 +78,7 @@ We found an issue where [Primary sites can not be removed from the UI](https://g This bug only exists in the UI and does not block the removal of Primary sites using any other method. -If you are running an affected version and need to remove your Primary site, you can manually remove the Primary site by using the [Geo Nodes API](../../../api/geo_nodes.md#delete-a-geo-node). +If you are running an affected version and need to remove your Primary site, you can manually remove the Primary site by using the [Geo Sites API](../../../api/geo_nodes.md#delete-a-geo-node). ### Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode @@ -71,9 +86,9 @@ GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab ## Updating to GitLab 13.12 -### Secondary nodes re-download all LFS files upon update +### Secondary sites re-download all LFS files upon update -We found an issue where [secondary nodes re-download all LFS files](https://gitlab.com/gitlab-org/gitlab/-/issues/334550) upon update. This bug: +We found an issue where [secondary sites re-download all LFS files](https://gitlab.com/gitlab-org/gitlab/-/issues/334550) upon update. This bug: - Only applies to Geo secondary sites that have replicated LFS objects. - Is _not_ a data loss risk. @@ -172,7 +187,7 @@ In GitLab 13.3, Geo removed the PostgreSQL [Foreign Data Wrapper](https://www.po dependency for the tracking database. The FDW server, user, and the extension is removed during the upgrade -process on each secondary node. The GitLab settings related to the FDW in the +process on each secondary site. The GitLab settings related to the FDW in the `/etc/gitlab/gitlab.rb` have been deprecated and can be safely removed. There are some scenarios like using an external PostgreSQL instance for the @@ -185,9 +200,9 @@ DROP EXTENSION IF EXISTS postgres_fdw; ``` WARNING: -In GitLab 13.3, promoting a secondary node to a primary while the secondary is +In GitLab 13.3, promoting a secondary site to a primary while the secondary is paused fails. Do not pause replication before promoting a secondary. If the -node is paused, be sure to resume before promoting. To avoid this issue, +site is paused, be sure to resume before promoting. To avoid this issue, upgrade to GitLab 13.4 or later. WARNING: @@ -198,9 +213,9 @@ contain a workaround if you run into errors during the failover. ## Updating to GitLab 13.2 -In GitLab 13.2, promoting a secondary node to a primary while the secondary is +In GitLab 13.2, promoting a secondary site to a primary while the secondary is paused fails. Do not pause replication before promoting a secondary. If the -node is paused, be sure to resume before promoting. To avoid this issue, +site is paused, be sure to resume before promoting. To avoid this issue, upgrade to GitLab 13.4 or later. ## Updating to GitLab 13.0 @@ -375,5 +390,5 @@ For the recommended procedure, see the WARNING: This version is affected by a [bug that results in new LFS objects not being -replicated to Geo secondary nodes](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). +replicated to Geo secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). The issue is fixed in GitLab 12.1. Be sure to upgrade to GitLab 12.1 or later. diff --git a/doc/administration/geo/secondary_proxy/index.md b/doc/administration/geo/secondary_proxy/index.md index ebd71757e91..c8dcbb81312 100644 --- a/doc/administration/geo/secondary_proxy/index.md +++ b/doc/administration/geo/secondary_proxy/index.md @@ -140,6 +140,18 @@ for details. To use TLS certificates with Let's Encrypt, you can manually point the domain to one of the Geo sites, generate the certificate, then copy it to all other sites. +## Behavior of secondary sites when the primary Geo site is down + +Considering that web traffic is proxied to the primary, the behavior of the secondary sites differs when the primary +site is inaccessible: + +- UI and API traffic return the same errors as the primary (or fail if the primary is not accessible at all), since they are proxied. +- For repositories that already exist on the specific secondary site being accessed, Git read operations still work as expected, + including authentication through HTTP(s) or SSH. +- Git operations for repositories that are not replicated to the secondary site return the same errors + as the primary site, since they are proxied. +- All Git write operations return the same errors as the primary site, since they are proxied. + ## Features accelerated by secondary Geo sites Most HTTP traffic sent to a secondary Geo site can be proxied to the primary Geo site. With this architecture, diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index 7f4c771c962..a8bebef8f44 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -40,7 +40,7 @@ instructions on setting up replication with a Patroni cluster. The GitLab **primary** node where the write operations happen connects to the **primary** database server, and **secondary** nodes -connect to their own database servers (which are also read-only). +connect to their own database servers (which are read-only). We recommend using [PostgreSQL replication slots](https://medium.com/@tk512/replication-slots-in-postgresql-b4b03d277c75) to ensure that the **primary** node retains all the data necessary for the **secondary** nodes to @@ -893,6 +893,9 @@ Instead, follow the instructions below. A production-ready and secure setup requires at least three Consul nodes, two Patroni nodes and one PgBouncer node on the secondary site. +Because of [omnibus-6587](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6587), Consul can't track multiple +services, so these need to be different than the nodes used for the Standby Cluster database. + Be sure to use [password credentials](../../postgresql/replication_and_failover.md#database-authorization-for-patroni) and other database best practices. diff --git a/doc/administration/get_started.md b/doc/administration/get_started.md index 3be3c11947e..f3b4400c8c3 100644 --- a/doc/administration/get_started.md +++ b/doc/administration/get_started.md @@ -62,7 +62,7 @@ You may need to import projects from external sources like GitHub, Bitbucket, or ### Popular project imports -- [GitHub Enterprise to self-managed GitLab](../integration/github.md#enabling-github-oauth): Enabling OAuth makes it easier for developers to find and import their projects. +- [GitHub Enterprise to self-managed GitLab](../integration/github.md): Enabling OAuth makes it easier for developers to find and import their projects. - [Bitbucket Server](../user/project/import/bitbucket_server.md#limitations): There are certain data limitations. For assistance with these data types, contact your GitLab account manager or GitLab Support about our professional migration services. diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md index a0c959d5de9..88efd1885db 100644 --- a/doc/administration/gitaly/configure_gitaly.md +++ b/doc/administration/gitaly/configure_gitaly.md @@ -28,6 +28,12 @@ The following configuration options are also available: - Configuring the [number of `gitaly-ruby` workers](#configure-number-of-gitaly-ruby-workers). - Limiting [RPC concurrency](#limit-rpc-concurrency). +## About the Gitaly token + +The token referred to throughout the Gitaly documentation is just an arbitrary password selected by +the administrator. It is unrelated to tokens created for the GitLab API or other similar web API +tokens. + ## Run Gitaly on its own server By default, Gitaly is run on the same server as Gitaly clients and is @@ -116,11 +122,6 @@ We assume your GitLab installation has three repository storages: You can use as few as one server with one repository storage if desired. -NOTE: -The token referred to throughout the Gitaly documentation is just an arbitrary password selected by -the administrator. It is unrelated to tokens created for the GitLab API or other similar web API -tokens. - ### Install Gitaly Install Gitaly on each Gitaly server using either Omnibus GitLab or install it from source: @@ -476,7 +477,7 @@ example: ```ruby git_data_dirs({ 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, - # Address of the GitLab server that has Gitaly running on it + # Address of the GitLab server that also has Gitaly running on it 'storage1' => { 'gitaly_address' => 'tcp://gitlab.internal:8075', 'path' => '/mnt/gitlab/git-data' }, 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, }) @@ -565,12 +566,6 @@ Note the following: - You can configure Gitaly servers with both an unencrypted listening address `listen_addr` and an encrypted listening address `tls_listen_addr` at the same time. This allows you to gradually transition from unencrypted to encrypted traffic if necessary. -- When running Praefect sub-commands such as `dial-nodes` and `list-untracked-repositories` from the command line with Gitaly TLS enabled, you must set - the `SSL_CERT_DIR` or `SSL_CERT_FILE` environment variable so that the Gitaly certificate is trusted. For example: - - ```shell - sudo SSL_CERT_DIR=/etc/gitlab/trusted_certs /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dial-nodes - ``` To configure Gitaly with TLS: diff --git a/doc/administration/gitaly/faq.md b/doc/administration/gitaly/faq.md index f79b9555c10..b0a88e8adc9 100644 --- a/doc/administration/gitaly/faq.md +++ b/doc/administration/gitaly/faq.md @@ -23,10 +23,10 @@ Gitaly Cluster and [Geo](../geo/index.md) both provide redundancy. However the r The following table outlines the major differences between Gitaly Cluster and Geo: -| Tool | Nodes | Locations | Latency tolerance | Failover | Consistency | Provides redundancy for | -|:---------------|:---------|:----------|:-------------------|:----------------------------------------------------------------------------|:-----------------------------------------|:------------------------| -| Gitaly Cluster | Multiple | Single | Approximately 1 ms | [Automatic](praefect.md#automatic-failover-and-primary-election-strategies) | [Strong](index.md#strong-consistency) | Data storage in Git | -| Geo | Multiple | Multiple | Up to one minute | [Manual](../geo/disaster_recovery/index.md) | Eventual | Entire GitLab instance | +| Tool | Nodes | Locations | Latency tolerance | Failover | Consistency | Provides redundancy for | +|:---------------|:---------|:----------|:------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------|:--------------------------------------|:------------------------| +| Gitaly Cluster | Multiple | Single | [Less than 1 second, ideally single-digit milliseconds](praefect.md#network-latency-and-connectivity) | [Automatic](praefect.md#automatic-failover-and-primary-election-strategies) | [Strong](index.md#strong-consistency) | Data storage in Git | +| Geo | Multiple | Multiple | Up to one minute | [Manual](../geo/disaster_recovery/index.md) | Eventual | Entire GitLab instance | For more information, see: diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 4d60832720b..ea24e901943 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -290,7 +290,7 @@ including [horizontally distributing reads](https://gitlab.com/groups/gitlab-org #### Distributed reads -> - Introduced in GitLab 13.1 in [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga) with feature flag `gitaly_distributed_reads` set to disabled. +> - Introduced in GitLab 13.1 in [beta](../../policy/alpha-beta-support.md#beta-features) with feature flag `gitaly_distributed_reads` set to disabled. > - [Made generally available and enabled by default](https://gitlab.com/gitlab-org/gitaly/-/issues/2951) in GitLab 13.3. > - [Disabled by default](https://gitlab.com/gitlab-org/gitaly/-/issues/3178) in GitLab 13.5. > - [Enabled by default](https://gitlab.com/gitlab-org/gitaly/-/issues/3334) in GitLab 13.8. @@ -316,8 +316,8 @@ You can [monitor distribution of reads](#monitor-gitaly-cluster) using Prometheu #### Strong consistency -> - Introduced in GitLab 13.1 in [alpha](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga), disabled by default. -> - Entered [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga) in GitLab 13.2, disabled by default. +> - Introduced in GitLab 13.1 in [alpha](../../policy/alpha-beta-support.md#alpha-features), disabled by default. +> - Entered [beta](../../policy/alpha-beta-support.md#beta-features) in GitLab 13.2, disabled by default. > - In GitLab 13.3, disabled unless primary-wins voting strategy is disabled. > - From GitLab 13.4, enabled by default. > - From GitLab 13.5, you must use Git v2.28.0 or higher on Gitaly nodes to enable strong consistency. @@ -529,6 +529,10 @@ To monitor [strong consistency](#strong-consistency), you can use the following - `gitaly_hook_transaction_voting_delay_seconds`, the client-side delay introduced by waiting for the transaction to be committed. +To monitor the number of repositories that have no healthy, up-to-date replicas: + +- `gitaly_praefect_unavailable_repositories` + You can also monitor the [Praefect logs](../logs.md#praefect-logs). #### Database metrics `/db_metrics` endpoint diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index e2db30b8358..8f17835b8a3 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -39,10 +39,26 @@ NOTE: If not set in GitLab, feature flags are read as false from the console and Praefect uses their default value. The default value depends on the GitLab version. -### Network connectivity +### Network latency and connectivity -Gitaly Cluster [components](index.md#components) need to communicate with each other over many -routes. Your firewall rules must allow the following for Gitaly Cluster to function properly: +Network latency for Gitaly Cluster should ideally be measurable in single-digit milliseconds. Latency is particularly +important for: + +- Gitaly node health checks. Nodes must be able to respond within 1 second. +- Reference transactions that enforce [strong consistency](index.md#strong-consistency). Lower latencies mean Gitaly + nodes can agree on changes faster. + +Achieving acceptable latency between Gitaly nodes: + +- On physical networks generally means high bandwidth, single location connections. +- On the cloud generally means within the same region, including allowing cross availability zone replication. These links + are designed for this type of synchronization. Latency of less than 2ms should be sufficient for Gitaly Cluster. + +If you can't provide low network latencies for replication (for example, between distant locations), consider Geo. For +more information, see [How does Gitaly Cluster compare to Geo](faq.md#how-does-gitaly-cluster-compare-to-geo). + +Gitaly Cluster [components](index.md#components) communicate with each other over many routes. Your firewall rules must +allow the following for Gitaly Cluster to function properly: | From | To | Default port | TLS port | |:-----------------------|:-----------------------|:-------------|:---------| @@ -254,11 +270,11 @@ reads distribution caching is enabled by configuration To reduce PostgreSQL resource consumption, we recommend setting up and configuring [PgBouncer](https://www.pgbouncer.org/) in front of the PostgreSQL instance. To do -this, you must point Praefect to PgBouncer by setting Praefect database parameters: +this, you must point Praefect to PgBouncer by setting database parameters on Praefect configuration: ```ruby praefect['database_host'] = PGBOUNCER_HOST -praefect['database_port'] = 5432 +praefect['database_port'] = 6432 praefect['database_user'] = 'praefect' praefect['database_password'] = PRAEFECT_SQL_PASSWORD praefect['database_dbname'] = 'praefect_production' @@ -273,40 +289,21 @@ Praefect requires an additional connection to the PostgreSQL that supports the this feature is only available with `session` pool mode (`pool_mode = session`). It is not supported in `transaction` pool mode (`pool_mode = transaction`). -For the additional connection, you must either: +To configure the additional connection, you must either: -- Connect Praefect directly to PostgreSQL and bypass PgBouncer. - Configure a new PgBouncer database that uses to the same PostgreSQL database endpoint, but with different pool mode. That is, `pool_mode = session`. +- Connect Praefect directly to PostgreSQL and bypass PgBouncer. -Praefect can be configured to use different connection parameters for direct access -to PostgreSQL. This is the connection that supports the `LISTEN` feature. - -Here is an example of Praefect that bypasses PgBouncer and directly connects to PostgreSQL: - -```ruby -praefect['database_direct_host'] = POSTGRESQL_HOST -praefect['database_direct_port'] = 5432 - -# Use the following to override parameters of direct database connection. -# Comment out where the parameters are the same for both connections. - -praefect['database_direct_user'] = 'praefect' -praefect['database_direct_password'] = PRAEFECT_SQL_PASSWORD -praefect['database_direct_dbname'] = 'praefect_production' -#praefect['database_direct_sslmode'] = '...' -#praefect['database_direct_sslcert'] = '...' -#praefect['database_direct_sslkey'] = '...' -#praefect['database_direct_sslrootcert'] = '...' -``` +#### Configure a new PgBouncer database with `pool_mode = session` -We recommend using PgBouncer with `session` pool mode instead. You can use the +We recommend using PgBouncer with `session` pool mode. You can use the [bundled PgBouncer](../postgresql/pgbouncer.md) or use an external PgBouncer and [configure it manually](https://www.pgbouncer.org/config.html). -The following example uses the bundled PgBouncer and sets up two separate connection pools, +The following example uses the bundled PgBouncer and sets up two separate connection pools on PostgreSQL host, one in `session` pool mode and the other in `transaction` pool mode. For this example to work, -you need to prepare PostgreSQL server with [setup instruction](#manual-database-setup): +you need to prepare PostgreSQL server as documented in [in the setup instructions](#manual-database-setup): ```ruby pgbouncer['databases'] = { @@ -373,6 +370,29 @@ your databases manually and configuring an external PgBouncer, you must include its password in the file used by PgBouncer. For example, `userlist.txt` if the [`auth_file`](https://www.pgbouncer.org/config.html#auth_file) configuration option is set. For more details, consult the PgBouncer documentation. +#### Configure Praefect to connect directly to PostgreSQL + +As an alternative to configuring PgBouncer with `session` pool mode, Praefect can be configured to use different connection parameters for direct access +to PostgreSQL. This is the connection that supports the `LISTEN` feature. + +An example of Praefect configuration that bypasses PgBouncer and directly connects to PostgreSQL: + +```ruby +praefect['database_direct_host'] = POSTGRESQL_HOST +praefect['database_direct_port'] = 5432 + +# Use the following to override parameters of direct database connection. +# Comment out where the parameters are the same for both connections. + +praefect['database_direct_user'] = 'praefect' +praefect['database_direct_password'] = PRAEFECT_SQL_PASSWORD +praefect['database_direct_dbname'] = 'praefect_production' +#praefect['database_direct_sslmode'] = '...' +#praefect['database_direct_sslcert'] = '...' +#praefect['database_direct_sslkey'] = '...' +#praefect['database_direct_sslrootcert'] = '...' +``` + ### Praefect > [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/2634) in GitLab 13.4, Praefect nodes can no longer be designated as `primary`. @@ -570,15 +590,15 @@ On the **Praefect** node: edit `/etc/gitlab/gitlab.rb`, remember to run `sudo gitlab-ctl reconfigure` again before trying the `sql-ping` command. -#### Enabling TLS support +#### Enable TLS support > [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/1698) in GitLab 13.2. Praefect supports TLS encryption. To communicate with a Praefect instance that listens for secure connections, you must: -- Use a `tls://` URL scheme in the `gitaly_address` of the corresponding storage entry - in the GitLab configuration. +- Ensure Gitaly is [configured for TLS](configure_gitaly.md#enable-tls-support) and use a `tls://` URL scheme in the `gitaly_address` + of the corresponding storage entry in the GitLab configuration. - Bring your own certificates because this isn't provided automatically. The certificate corresponding to each Praefect server must be installed on that Praefect server. @@ -594,6 +614,13 @@ Note the following: - Hostname, you can either use the Common Name field for this, or add it as a Subject Alternative Name. - IP address, you must add it as a Subject Alternative Name to the certificate. +- When running Praefect sub-commands such as `dial-nodes` and `list-untracked-repositories` from the command line with + [Gitaly TLS enabled](configure_gitaly.md#enable-tls-support), you must set the `SSL_CERT_DIR` or `SSL_CERT_FILE` + environment variable so that the Gitaly certificate is trusted. For example: + + ```shell + sudo SSL_CERT_DIR=/etc/gitlab/trusted_certs /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dial-nodes + ``` - You can configure Praefect servers with both an unencrypted listening address `listen_addr` and an encrypted listening address `tls_listen_addr` at the same time. @@ -639,7 +666,7 @@ To configure Praefect with TLS: ```ruby git_data_dirs({ "default" => { - "gitaly_address" => 'tls://PRAEFECT_LOADBALANCER_HOST:2305', + "gitaly_address" => 'tls://PRAEFECT_LOADBALANCER_HOST:3305', "gitaly_token" => 'PRAEFECT_EXTERNAL_TOKEN' } }) @@ -957,7 +984,10 @@ Particular attention should be shown to: balancer. - `PRAEFECT_EXTERNAL_TOKEN` with the real secret - If you are using TLS, the `gitaly_address` should begin with `tls://`. + If you are using TLS: + + - The `gitaly_address` should begin with `tls://` instead. + - The port should be changed to `3305`. ```ruby git_data_dirs({ diff --git a/doc/administration/gitaly/recovery.md b/doc/administration/gitaly/recovery.md index e1b9a73908d..9210c8f08d3 100644 --- a/doc/administration/gitaly/recovery.md +++ b/doc/administration/gitaly/recovery.md @@ -28,7 +28,7 @@ To minimize data loss in GitLab 13.0 to 14.0, Gitaly Cluster: ### Read-only mode -> - Introduced in GitLab 13.0 as [generally available](https://about.gitlab.com/handbook/product/gitlab-the-product/#generally-available-ga). +> - Introduced in GitLab 13.0 as [generally available](../../policy/alpha-beta-support.md#generally-available-ga). > - Between GitLab 13.0 and GitLab 13.2, read-only mode applied to the whole virtual storage and occurred whenever failover occurred. > - [In GitLab 13.3 and later](https://gitlab.com/gitlab-org/gitaly/-/issues/2862), read-only mode applies on a per-repository basis and only occurs if a new primary is out of date. new primary. If the failed primary contained unreplicated writes, [data loss can occur](#check-for-data-loss). diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index fdd281c1a90..ff136b44340 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -435,6 +435,15 @@ Here are common errors and potential causes: - Praefect cannot reach one or more of its child Gitaly nodes. Try running the Praefect connection checker to diagnose. +### Praefect database experiencing high CPU load + +Some common reasons for the Praefect database to experience elevated CPU usage include: + +- Prometheus metrics scrapes [running an expensive query](https://gitlab.com/gitlab-org/gitaly/-/issues/3796). If you have GitLab 14.2 + or above, set `praefect['separate_database_metrics'] = true` in `gitlab.rb`. +- [Read distribution caching](praefect.md#reads-distribution-caching) is disabled, increasing the number of queries made to the + database when user traffic is high. Ensure read distribution caching is enabled. + ### Determine primary Gitaly node To determine the primary node of a repository: diff --git a/doc/administration/img/custom_hooks_error_msg.png b/doc/administration/img/custom_hooks_error_msg.png Binary files differdeleted file mode 100644 index e7d5c157d08..00000000000 --- a/doc/administration/img/custom_hooks_error_msg.png +++ /dev/null diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index 84b30724dff..cd1f39b1295 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -62,8 +62,8 @@ can reserve your catch-all mailbox for other purposes. ### Dedicated email address -This solution is relatively simple to set up: you just need to create an email -address dedicated to receive your users' replies to GitLab notifications. However, +To set up this solution, you must create a dedicated email +address to receive your users' replies to GitLab notifications. However, this method only supports replies, and not the other features of [incoming email](#incoming-email). ## Accepted headers @@ -75,7 +75,7 @@ Email is processed correctly when a configured email address is present in one o - `Envelope-To` or `X-Envelope-To` In GitLab 14.6 and later, [Service Desk](../user/project/service_desk.md) -also checks these additional headers. +also checks accepted headers. Usually, the "To" field contains the email address of the primary receiver. However, it might not include the configured GitLab email address if: @@ -84,6 +84,14 @@ However, it might not include the configured GitLab email address if: - The address was included when using "Reply all". - The email was forwarded. +## Rejected headers + +To prevent unwanted issue creation from automatic email systems, GitLab ignores all incoming email +containing the following headers: + +- `Auto-Submitted` with a value other than `no` +- `X-Autoreply` with a value of `yes` + ## Set it up If you want to use Gmail / Google Apps for incoming email, make sure you have diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index 3bedb6b01bd..614ab5278c5 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -214,7 +214,7 @@ or `git push --mirror` is used. This limit does not affect any of the updated merge request pipelines. All updated merge requests have a pipeline created when using -[pipelines for merge requests](../ci/pipelines/merge_request_pipelines.md). +[merge request pipelines](../ci/pipelines/merge_request_pipelines.md). ## Retention of activity history @@ -228,7 +228,11 @@ There is a limit when embedding metrics in GitLab Flavored Markdown (GFM) for pe - **Max limit**: 100 embeds. -## Number of webhooks +## Webhook limits + +Also see [Webhook rate limits](#webhook-rate-limit). + +### Number of webhooks To set the maximum number of group or project webhooks for a self-managed installation, run the following in the [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): @@ -246,11 +250,33 @@ Plan.default.actual_limits.update!(group_hooks: 100) Set the limit to `0` to disable it. -- **Default maximum number of webhooks**: `100` per project, `50` per group -- **Maximum payload size**: 25 MB +The default maximum number of webhooks is `100` per project, `50` per group. For GitLab.com, see the [webhook limits for GitLab.com](../user/gitlab_com/index.md#webhooks). +### Webhook payload size + +The maximum webhook payload size is 25 MB. + +### Recursive webhooks + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/329743) in GitLab 14.8. + +GitLab detects and blocks webhooks that are recursive or that exceed the limit +of webhooks that can be triggered from other webhooks. This enables GitLab to +continue to support workflows that use webhooks to call the API non-recursively, or that +do not trigger an unreasonable number of other webhooks. + +Recursion can happen when a webhook is configured to make a call +to its own GitLab instance (for example, the API). The call then triggers the same +webhook and creates an infinite loop. + +The maximum number of requests to an instance made by a series of webhooks that +trigger other webhooks is 100. When the limit is reached, GitLab blocks any further +webhooks that would be triggered by the series. + +Blocked recursive webhook calls are logged in `auth.log` with the message `"Recursive webhook blocked from executing"`. + ## Pull Mirroring Interval > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/237891) in GitLab 13.7. diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 62c403bfe43..8f824fcefb3 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Testing +group: Pipeline Insights info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -440,10 +440,7 @@ To change the number of projects listed, change the number in `limit(50)`. #### Delete job artifacts from jobs completed before a specific date WARNING: -These commands remove data permanently from the database and from disk. We -highly recommend running them only under the guidance of a Support Engineer, or -running them in a test environment with a backup of the instance ready to be -restored, just in case. +These commands remove data permanently from both the database and from disk. Before running them, we highly recommend seeking guidance from a Support Engineer, or running them in a test environment with a backup of the instance ready to be restored, just in case. If you need to manually remove job artifacts associated with multiple jobs while **retaining their job logs**, this can be done from the Rails console (`sudo gitlab-rails console`): @@ -490,10 +487,7 @@ If you need to manually remove job artifacts associated with multiple jobs while #### Delete job artifacts and logs from jobs completed before a specific date WARNING: -These commands remove data permanently from both the database and from disk. We -highly recommend running them only under the guidance of a Support Engineer, or -running them in a test environment with a backup of the instance ready to be -restored, just in case. +These commands remove data permanently from both the database and from disk. Before running them, we highly recommend seeking guidance from a Support Engineer, or running them in a test environment with a backup of the instance ready to be restored, just in case. If you need to manually remove **all** job artifacts associated with multiple jobs, **including job logs**, this can be done from the Rails console (`sudo gitlab-rails console`): diff --git a/doc/administration/load_balancer.md b/doc/administration/load_balancer.md index 60293e56202..45fd6e65078 100644 --- a/doc/administration/load_balancer.md +++ b/doc/administration/load_balancer.md @@ -2,7 +2,6 @@ stage: Enablement group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference --- # Load Balancer for multi-node GitLab **(FREE SELF)** @@ -21,38 +20,38 @@ How do you want to handle SSL in your multi-node environment? There are several options: - Each application node terminates SSL -- The load balancer(s) terminate SSL and communication is not secure between - the load balancer(s) and the application nodes -- The load balancer(s) terminate SSL and communication is *secure* between the - load balancer(s) and the application nodes +- The load balancers terminate SSL and communication is not secure between + the load balancers and the application nodes +- The load balancers terminate SSL and communication is *secure* between the + load balancers and the application nodes ### Application nodes terminate SSL -Configure your load balancer(s) to pass connections on port 443 as 'TCP' rather +Configure your load balancers to pass connections on port 443 as 'TCP' rather than 'HTTP(S)' protocol. This passes the connection to the application nodes NGINX service untouched. NGINX has the SSL certificate and listen on port 443. See [NGINX HTTPS documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) for details on managing SSL certificates and configuring NGINX. -### Load Balancer(s) terminate SSL without backend SSL +### Load Balancers terminate SSL without backend SSL -Configure your load balancer(s) to use the 'HTTP(S)' protocol rather than 'TCP'. -The load balancer(s) is be responsible for managing SSL certificates and +Configure your load balancers to use the 'HTTP(S)' protocol rather than 'TCP'. +The load balancers is be responsible for managing SSL certificates and terminating SSL. -Since communication between the load balancer(s) and GitLab isn't secure, +Since communication between the load balancers and GitLab isn't secure, there is some additional configuration needed. See [NGINX Proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#supporting-proxied-ssl) for details. -### Load Balancer(s) terminate SSL with backend SSL +### Load Balancers terminate SSL with backend SSL -Configure your load balancer(s) to use the 'HTTP(S)' protocol rather than 'TCP'. -The load balancer(s) is responsible for managing SSL certificates that +Configure your load balancers to use the 'HTTP(S)' protocol rather than 'TCP'. +The load balancers is responsible for managing SSL certificates that end users see. -Traffic is secure between the load balancer(s) and NGINX in this +Traffic is secure between the load balancers and NGINX in this scenario. There is no need to add configuration for proxied SSL since the connection is secure all the way. However, configuration must be added to GitLab to configure SSL certificates. See diff --git a/doc/administration/logs.md b/doc/administration/logs.md index 0d7635405d6..c4e9642d048 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/github_imports.md b/doc/administration/monitoring/github_imports.md index e91483eb79d..e16e9bb0336 100644 --- a/doc/administration/monitoring/github_imports.md +++ b/doc/administration/monitoring/github_imports.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md index 1cf4e5a25ba..911d76a89e3 100644 --- a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md +++ b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md @@ -1,13 +1,17 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Self monitoring project **(FREE SELF)** +# Self-monitoring project (DEPRECATED) **(FREE SELF)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32351) in GitLab 12.7, behind a disabled feature flag (`self_monitoring_project`). -> - The feature flag was removed and the self monitoring project was [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/198511) in GitLab 12.8. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32351) in GitLab 12.7 [with a flag](../../feature_flags.md) named `self_monitoring_project`. Disabled by default. +> - Generally available in GitLab 12.8. [Feature flag `self_monitoring_project`](https://gitlab.com/gitlab-org/gitlab/-/issues/198511) removed. +> - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/348909) in GitLab 14.8. + +WARNING: +This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/348909) in GitLab 14.8. GitLab provides administrators insights into the health of their GitLab instance. diff --git a/doc/administration/monitoring/index.md b/doc/administration/monitoring/index.md index 4c49efd6bd5..df655053723 100644 --- a/doc/administration/monitoring/index.md +++ b/doc/administration/monitoring/index.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/ip_whitelist.md b/doc/administration/monitoring/ip_whitelist.md index 75b09f8a366..b8347ba3f0d 100644 --- a/doc/administration/monitoring/ip_whitelist.md +++ b/doc/administration/monitoring/ip_whitelist.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/performance/gitlab_configuration.md b/doc/administration/monitoring/performance/gitlab_configuration.md index f316a75a868..128ddad6555 100644 --- a/doc/administration/monitoring/performance/gitlab_configuration.md +++ b/doc/administration/monitoring/performance/gitlab_configuration.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md index c37a264938e..79612145327 100644 --- a/doc/administration/monitoring/performance/grafana_configuration.md +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/performance/index.md b/doc/administration/monitoring/performance/index.md index f3db6ac9f03..20fad8baf91 100644 --- a/doc/administration/monitoring/performance/index.md +++ b/doc/administration/monitoring/performance/index.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index 14a560223f9..e93dbd3a2d6 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -108,7 +108,7 @@ appears next to requests with warnings. The performance bar is disabled by default for non-administrators. To enable it for a given group: -1. Sign in as a user with Administrator [role](../../../user/permissions.md). +1. Sign in as a user with administrator access. 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Metrics and profiling** (`admin/application_settings/metrics_and_profiling`), and expand diff --git a/doc/administration/monitoring/performance/request_profiling.md b/doc/administration/monitoring/performance/request_profiling.md index ebdca8d3960..f8a66f6f48f 100644 --- a/doc/administration/monitoring/performance/request_profiling.md +++ b/doc/administration/monitoring/performance/request_profiling.md @@ -1,10 +1,16 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Request Profiling **(FREE SELF)** +# Request profiling (DEPRECATED) **(FREE SELF)** + +> [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/352488) in GitLab 14.8, and planned for removal in GitLab 15.0. + +WARNING: +This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/352488) +for use in GitLab 14.8, and is planned for removal in GitLab 15.0. To profile a request: diff --git a/doc/administration/monitoring/prometheus/gitlab_exporter.md b/doc/administration/monitoring/prometheus/gitlab_exporter.md index d9852524aec..15ec880533e 100644 --- a/doc/administration/monitoring/prometheus/gitlab_exporter.md +++ b/doc/administration/monitoring/prometheus/gitlab_exporter.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index c5b87afd94b..a1bfae678c7 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w To enable the GitLab Prometheus metrics: -1. Log in to GitLab as a user with Administrator [role](../../../user/permissions.md). +1. Log in to GitLab as a user with administrator access. 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Metrics and profiling**. 1. Find the **Metrics - Prometheus** section, and select **Add link to Prometheus**. @@ -48,6 +48,7 @@ The following metrics are available: | `gitlab_database_transaction_seconds` | Histogram | 12.1 | Time spent in database transactions, in seconds | | | `gitlab_method_call_duration_seconds` | Histogram | 10.2 | Method calls real duration | `controller`, `action`, `module`, `method` | | `gitlab_page_out_of_bounds` | Counter | 12.8 | Counter for the PageLimiter pagination limit being hit | `controller`, `action`, `bot` | +| `gitlab_rails_boot_time_seconds` | Gauge | 14.8 | Time elapsed for Rails primary process to finish startup | | | `gitlab_rails_queue_duration_seconds` | Histogram | 9.4 | Measures latency between GitLab Workhorse forwarding a request to Rails | | | `gitlab_sql_duration_seconds` | Histogram | 10.2 | SQL execution time, excluding `SCHEMA` operations and `BEGIN` / `COMMIT` | | | `gitlab_sql_<role>_duration_seconds` | Histogram | 13.10 | SQL execution time, excluding `SCHEMA` operations and `BEGIN` / `COMMIT`, grouped by database roles (primary/replica) | | @@ -269,6 +270,16 @@ configuration option in `gitlab.yml`. These metrics are served from the | `geo_pages_deployments_verification_total` | Gauge | 14.6 | Number of pages deployments verifications tried on secondary | `url` | | `geo_pages_deployments_verified` | Gauge | 14.6 | Number of pages deployments verified on secondary | `url` | | `geo_pages_deployments_verification_failed` | Gauge | 14.6 | Number of pages deployments verifications failed on secondary | `url` | +| `geo_job_artifacts` | Gauge | 14.8 | Number of job artifacts on primary | `url` | +| `geo_job_artifacts_checksum_total` | Gauge | 14.8 | Number of job artifacts tried to checksum on primary | `url` | +| `geo_job_artifacts_checksummed` | Gauge | 14.8 | Number of job artifacts successfully checksummed on primary | `url` | +| `geo_job_artifacts_checksum_failed` | Gauge | 14.8 | Number of job artifacts failed to calculate the checksum on primary | `url` | +| `geo_job_artifacts_synced` | Gauge | 14.8 | Number of syncable job artifacts synced on secondary | `url` | +| `geo_job_artifacts_failed` | Gauge | 14.8 | Number of syncable job artifacts failed to sync on secondary | `url` | +| `geo_job_artifacts_registry` | Gauge | 14.8 | Number of job artifacts in the registry | `url` | +| `geo_job_artifacts_verification_total` | Gauge | 14.8 | Number of job artifacts verifications tried on secondary | `url` | +| `geo_job_artifacts_verified` | Gauge | 14.8 | Number of job artifacts verified on secondary | `url` | +| `geo_job_artifacts_verification_failed` | Gauge | 14.8 | Number of job artifacts verifications failed on secondary | `url` | | `limited_capacity_worker_running_jobs` | Gauge | 13.5 | Number of running jobs | `worker` | | `limited_capacity_worker_max_running_jobs` | Gauge | 13.5 | Maximum number of running jobs | `worker` | | `limited_capacity_worker_remaining_work_count` | Gauge | 13.5 | Number of jobs waiting to be enqueued | `worker` | diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index acdcdb41dca..9d90c4f3cc7 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -295,6 +295,36 @@ To use an external Prometheus server: 1. Reload the Prometheus server. +### Configure the storage retention size + +Prometheus has several custom flags to configure local storage: + +- `storage.tsdb.retention.time`: when to remove old data. Defaults to `15d`. Overrides + `storage.tsdb.retention` if this flag is set to anything other than the default. +- `storage.tsdb.retention.size`: [EXPERIMENTAL] the maximum number of bytes of storage blocks to + retain. The oldest data is removed first. Defaults to `0` (disabled). This flag is experimental + and may change in future releases. Units supported: `B`, `KB`, `MB`, `GB`, `TB`, `PB`, `EB`. For + example, `512MB`. + +To configure the storage retention size: + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + prometheus['flags'] = { + 'storage.tsdb.path' => "/var/opt/gitlab/prometheus/data", + 'storage.tsdb.retention.time' => "7d", + 'storage.tsdb.retention.size' => "2GB", + 'config.file' => "/var/opt/gitlab/prometheus/prometheus.yml" + } + ``` + +1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + ## Viewing performance metrics You can visit `http://localhost:9090` for the dashboard that Prometheus offers by default. @@ -402,3 +432,35 @@ To disable the monitoring of Kubernetes: 1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +## Troubleshooting + +### `/var/opt/gitlab/prometheus` consumes too much disk space + +If you are **not** using Prometheus monitoring: + +1. [Disable Prometheus](index.md#configuring-prometheus). +1. Delete the data under `/var/opt/gitlab/prometheus`. + +If you are using Prometheus monitoring: + +1. Stop Prometheus (deleting data while it's running can cause data corruption): + + ```shell + gitlab-ctl stop prometheus + ``` + +1. Delete the data under `/var/opt/gitlab/prometheus/data`. +1. Start the service again: + + ```shell + gitlab-ctl start prometheus + ``` + +1. Verify the service is up and running: + + ```shell + gitlab-ctl status prometheus + ``` + +1. Optional. [Configure the storage retention size](index.md#configure-the-storage-retention-size). diff --git a/doc/administration/monitoring/prometheus/node_exporter.md b/doc/administration/monitoring/prometheus/node_exporter.md index 68d997d7596..d7a4a96cd9a 100644 --- a/doc/administration/monitoring/prometheus/node_exporter.md +++ b/doc/administration/monitoring/prometheus/node_exporter.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md index aba1561500a..979a6bcd232 100644 --- a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md +++ b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/prometheus/postgres_exporter.md b/doc/administration/monitoring/prometheus/postgres_exporter.md index 8a851afe35b..95a6540bd19 100644 --- a/doc/administration/monitoring/prometheus/postgres_exporter.md +++ b/doc/administration/monitoring/prometheus/postgres_exporter.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/prometheus/puma_exporter.md b/doc/administration/monitoring/prometheus/puma_exporter.md index 804c4243cfa..794e2c10b25 100644 --- a/doc/administration/monitoring/prometheus/puma_exporter.md +++ b/doc/administration/monitoring/prometheus/puma_exporter.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/prometheus/redis_exporter.md b/doc/administration/monitoring/prometheus/redis_exporter.md index 6cc262842a1..a5f12bbc52f 100644 --- a/doc/administration/monitoring/prometheus/redis_exporter.md +++ b/doc/administration/monitoring/prometheus/redis_exporter.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/prometheus/registry_exporter.md b/doc/administration/monitoring/prometheus/registry_exporter.md index 3a2acd47338..f4fa35c206e 100644 --- a/doc/administration/monitoring/prometheus/registry_exporter.md +++ b/doc/administration/monitoring/prometheus/registry_exporter.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md index 87a10a0ec04..50c8102f366 100644 --- a/doc/administration/nfs.md +++ b/doc/administration/nfs.md @@ -24,11 +24,9 @@ file system performance, see Starting with GitLab version 14.0, support for NFS to store Git repository data is deprecated. Technical customer support and engineering support is available for the 14.x releases. Engineering is fixing bugs and security vulnerabilities consistent with our [release and maintenance policy](../policy/maintenance.md#security-releases). -At the end of the 14.12 milestone (tentatively June 22nd, 2022) technical and engineering support for using NFS to store Git repository data will be officially at end-of-life. There will be no product changes or troubleshooting provided via Engineering, Security or Paid Support channels. +Upon the release of GitLab 15.0 (tentatively May 22nd, 2022) technical and engineering support for using NFS to store Git repository data will be officially at end-of-life. There will be no product changes or troubleshooting provided via Engineering, Security or Paid Support channels after the release date of 15.0, regardless of your GitLab version. -For those customers still running earlier versions of GitLab, [our support eligibility and maintenance policy applies](https://about.gitlab.com/support/statement-of-support.html#version-support). - -For the 14.x releases, we continue to help with Git related tickets from customers running one or more Gitaly servers with its data stored on NFS. Examples may include: +Until the release of 15.0, for customers running 14.x releases, we continue to help with Git related tickets from customers running one or more Gitaly servers with its data stored on NFS. Examples may include: - Performance issues or timeouts accessing Git data - Commits or branches vanish diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index 0976fc3684e..420f2191ef2 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -526,7 +526,7 @@ To migrate existing local data to object storage see the following guides: - [Uploads](raketasks/uploads/migrate.md#migrate-to-object-storage) - [Merge request diffs](merge_request_diffs.md#using-object-storage) - [Packages](packages/index.md#migrating-local-packages-to-object-storage) (optional feature) -- Dependency Proxy - [migration not yet supported](https://gitlab.com/gitlab-org/gitlab/-/issues/343064) +- [Dependency Proxy](packages/dependency_proxy.md#migrate-local-dependency-proxy-blobs-and-manifests-to-object-storage) - [Terraform state files](terraform_state.md#migrate-to-object-storage) - [Pages content](pages/index.md#migrate-pages-deployments-to-object-storage) diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md index 1c9b98041dc..fd04efe8473 100644 --- a/doc/administration/operations/extra_sidekiq_processes.md +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -296,7 +296,7 @@ Instead of a queue, a queue namespace can also be provided, to have the process automatically listen on all queues in that namespace without needing to explicitly list all the queue names. For more information about queue namespaces, see the relevant section in the -[Sidekiq style guide](../../development/sidekiq_style_guide.md#queue-namespaces). +[Sidekiq development documentation](../../development/sidekiq/index.md#queue-namespaces). For example, say you want to start 2 extra processes: one to process the `process_commit` queue, and one to process the `post_receive` queue. This can be diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index ddedb3fe76a..dca99879cc3 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -138,7 +138,7 @@ This is a brief overview. Please refer to the above instructions for more contex > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299109) in GitLab 14.5. WARNING: -`gitlab-sshd` is in [**Alpha**](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga). +`gitlab-sshd` is in [**Alpha**](../../policy/alpha-beta-support.md#alpha-features). It is not ready for production use. `gitlab-sshd` is [a standalone SSH server](https://gitlab.com/gitlab-org/gitlab-shell/-/tree/main/internal/sshd) @@ -178,3 +178,22 @@ GitLab supports `authorized_keys` database lookups with [SELinux](https://en.wik Because the SELinux policy is static, GitLab doesn't support the ability to change internal webserver ports at the moment. Administrators would have to create a special `.te` file for the environment, since it isn't generated dynamically. + +## Troubleshooting + +If your SSH traffic is [slow](https://github.com/linux-pam/linux-pam/issues/270) +or causing high CPU load, be sure to check the size of `/var/log/btmp`, and ensure it is rotated on a regular basis. +If this file is very large, GitLab SSH fast lookup can cause the bottleneck to be hit more frequently, thus decreasing performance even further. +If you are able to, you may consider disabling [`UsePAM` in your `sshd_config`](https://linux.die.net/man/5/sshd_config) to avoid reading `/var/log/btmp` altogether. + +Running `strace` and `lsof` on a running `sshd: git` process can return useful debugging information. To get an `strace` on an in-progress Git over SSH connection for IP `x.x.x.x`, run: + +```plaintext +sudo strace -s 10000 -p $(sudo netstat -tp | grep x.x.x.x | egrep 'ssh.*: git' | sed -e 's/.*ESTABLISHED *//' -e 's#/.*##') +``` + +Or get an `lsof` for a running Git over SSH process: + +```plaintext +sudo lsof -p $(sudo netstat -tp | egrep 'ssh.*: git' | head -1 | sed -e 's/.*ESTABLISHED *//' -e 's#/.*##') +``` diff --git a/doc/administration/package_information/supported_os.md b/doc/administration/package_information/supported_os.md index 78f496a3998..71ebc4d3647 100644 --- a/doc/administration/package_information/supported_os.md +++ b/doc/administration/package_information/supported_os.md @@ -23,7 +23,6 @@ The following lists the currently supported OSs and their possible EOL dates. | Debian 9 | GitLab CE / GitLab EE 9.3.0 | amd64 | 2022 | <https://wiki.debian.org/LTS> | | Debian 10 | GitLab CE / GitLab EE 12.2.0 | amd64, arm64 | 2024 | <https://wiki.debian.org/LTS> | | Debian 11 | GitLab CE / GitLab EE 14.6.0 | amd64, arm64 | 2026 | <https://wiki.debian.org/LTS> | -| OpenSUSE 15.2 | GitLab CE / GitLab EE 13.11.0 | x86_64, aarch64 | Dec 2021 | <https://en.opensuse.org/Lifetime> | | OpenSUSE 15.3 | GitLab CE / GitLab EE 14.5.0 | x86_64, aarch64 | Nov 2022 | <https://en.opensuse.org/Lifetime> | | SLES 12 | GitLab EE 9.0.0 | x86_64 | Oct 2027 | <https://www.suse.com/lifecycle/> | | Ubuntu 18.04 | GitLab CE / GitLab EE 10.7.0 | amd64 | April 2023 | <https://wiki.ubuntu.com/Releases> | @@ -81,8 +80,9 @@ release for them can be found below: | Raspbian Stretch | [June 2020](https://downloads.raspberrypi.org/raspbian/images/raspbian-2019-04-09/) | [GitLab CE](https://packages.gitlab.com/app/gitlab/raspberry-pi2/search?q=gitlab-ce_13.2&dist=raspbian%2Fstretch) 13.3 | | Debian Jessie | [June 2020](https://www.debian.org/News/2020/20200709) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_13.2&dist=debian%2Fjessie) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_13.2&dist=debian%2Fjessie) 13.3 | | CentOS 6 | [November 2020](https://wiki.centos.org/About/Product) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=13.6&filter=all&filter=all&dist=el%2F6) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=13.6&filter=all&filter=all&dist=el%2F6) 13.6 | -| OpenSUSE 15.1 | [November 2020](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-13.12&dist=opensuse%2F15.1) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-13.12&dist=opensuse%2F15.2) 13.12 | +| OpenSUSE 15.1 | [November 2020](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-13.12&dist=opensuse%2F15.1) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-13.12&dist=opensuse%2F15.1) 13.12 | | Ubuntu 16.04 | [April 2021](https://ubuntu.com/info/release-end-of-life) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_13.12&dist=ubuntu%2Fxenial) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_13.12&dist=ubuntu%2Fxenial) 13.12 | +| OpenSUSE 15.2 | [December 2021](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-14.7&dist=opensuse%2F15.2) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-14.7&dist=opensuse%2F15.2) 14.7 | NOTE: An exception to this deprecation policy is when we are unable to provide diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index 43293385ed9..33a5311709f 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -662,14 +662,14 @@ configurable in future releases. The Registry server listens on localhost at port `5000` by default, which is the address for which the Registry server should accept connections. -In the examples below we set the Registry's port to `5001`. +In the examples below we set the Registry's port to `5010`. **Omnibus GitLab** 1. Open `/etc/gitlab/gitlab.rb` and set `registry['registry_http_addr']`: ```ruby - registry['registry_http_addr'] = "localhost:5001" + registry['registry_http_addr'] = "localhost:5010" ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -681,7 +681,7 @@ In the examples below we set the Registry's port to `5001`. ```yaml http: - addr: localhost:5001 + addr: localhost:5010 ``` 1. Save the file and restart the Registry server. @@ -1379,7 +1379,7 @@ project or branch name. Special characters can include: To get around this, you can [change the group path](../../user/group/index.md#change-a-groups-path), [change the project path](../../user/project/settings/index.md#renaming-a-repository) or change the -branch name. Another option is to create a [push rule](../../push_rules/push_rules.md) to prevent +branch name. Another option is to create a [push rule](../../user/project/repository/push_rules.md) to prevent this at the instance level. ### Image push errors @@ -1548,6 +1548,46 @@ To fix this you can do one of two things: We use a concrete example to illustrate how to diagnose a problem with the S3 setup. +#### Investigate a cleanup policy + +If you're unsure why your cleanup policy did or didn't delete a tag, execute the policy line by line +by running the below script from the [Rails console](../../administration/operations/rails_console.md). +This can help diagnose problems with the policy. + +```ruby +repo = ContainerRepository.find(<project_id>) +policy = repo.project.container_expiration_policy + +tags = repo.tags +tags.map(&:name) + +tags.reject!(&:latest?) +tags.map(&:name) + +regex_delete = ::Gitlab::UntrustedRegexp.new("\\A#{policy.name_regex}\\z") +regex_retain = ::Gitlab::UntrustedRegexp.new("\\A#{policy.name_regex_keep}\\z") + +tags.select! { |tag| regex_delete.match?(tag.name) && !regex_retain.match?(tag.name) } + +tags.map(&:name) + +now = DateTime.current +tags.sort_by! { |tag| tag.created_at || now }.reverse! # Lengthy operation + +tags = tags.drop(policy.keep_n) +tags.map(&:name) + +older_than_timestamp = ChronicDuration.parse(policy.older_than).seconds.ago + +tags.select! { |tag| tag.created_at && tag.created_at < older_than_timestamp } + +tags.map(&:name) +``` + +- The script builds the list of tags to delete (`tags`). +- `tags.map(&:name)` prints a list of tags to remove. This may be a lengthy operation. +- After each filter, check the list of `tags` to see if it contains the intended tags to destroy. + #### Unexpected 403 error during push A user attempted to enable an S3-backed Registry. The `docker login` step went diff --git a/doc/administration/packages/dependency_proxy.md b/doc/administration/packages/dependency_proxy.md index b3dc6ffc2b2..93c6585230f 100644 --- a/doc/administration/packages/dependency_proxy.md +++ b/doc/administration/packages/dependency_proxy.md @@ -199,6 +199,55 @@ This section describes the earlier configuration format. 1. [Restart GitLab](../restart_gitlab.md#installations-from-source "How to restart GitLab") for the changes to take effect. +#### Migrate local Dependency Proxy blobs and manifests to object storage + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79663) in GitLab 14.8. + +After [configuring object storage](#using-object-storage), +use the following task to migrate existing Dependency Proxy blobs and manifests from local storage +to remote storage. The processing is done in a background worker and requires no downtime. + +For Omnibus GitLab: + +```shell +sudo gitlab-rake "gitlab:dependency_proxy:migrate" +``` + +For installations from source: + +```shell +RAILS_ENV=production sudo -u git -H bundle exec rake gitlab:dependency_proxy:migrate +``` + +You can optionally track progress and verify that all packages migrated successfully using the +[PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database): + +- For Omnibus GitLab instances: `sudo gitlab-rails dbconsole` +- For installations from source: `sudo -u git -H psql -d gitlabhq_production` + +Verify that `objectstg` (where `file_store = '2'`) has the count of all Dependency Proxy blobs and +manifests for each respective query: + +```shell +gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM dependency_proxy_blobs; + +total | filesystem | objectstg +------+------------+----------- + 22 | 0 | 22 + +gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM dependency_proxy_manifests; + +total | filesystem | objectstg +------+------------+----------- + 10 | 0 | 10 +``` + +Verify that there are no files on disk in the `dependency_proxy` folder: + +```shell +sudo find /var/opt/gitlab/gitlab-rails/shared/dependency_proxy -type f | grep -v tmp | wc -l +``` + ## Disabling Authentication Authentication was introduced in 13.7 as part of [enabling private groups to use the diff --git a/doc/administration/packages/index.md b/doc/administration/packages/index.md index eea4964efbe..abf5c46114b 100644 --- a/doc/administration/packages/index.md +++ b/doc/administration/packages/index.md @@ -240,7 +240,7 @@ You can optionally track progress and verify that all packages migrated successf - `sudo gitlab-rails dbconsole` for Omnibus GitLab instances. - `sudo -u git -H psql -d gitlabhq_production` for source-installed instances. -Verify `objectstg` below (where `store=2`) has count of all packages: +Verify `objectstg` below (where `file_store = '2'`) has count of all packages: ```shell gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM packages_package_files; diff --git a/doc/administration/pages/img/pages_deploy_failure_v14_8.png b/doc/administration/pages/img/pages_deploy_failure_v14_8.png Binary files differnew file mode 100644 index 00000000000..ec502b86453 --- /dev/null +++ b/doc/administration/pages/img/pages_deploy_failure_v14_8.png diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 53510688794..f000824711f 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -238,6 +238,7 @@ control over how the Pages daemon runs and serves content in your environment. | `artifacts_server_url` | API URL to proxy artifact requests to. Defaults to GitLab `external URL` + `/api/v4`, for example `https://gitlab.com/api/v4`. When running a [separate Pages server](#running-gitlab-pages-on-a-separate-server), this URL must point to the main GitLab server's API. | | `auth_redirect_uri` | Callback URL for authenticating with GitLab. Defaults to project's subdomain of `pages_external_url` + `/auth`. | | `auth_secret` | Secret key for signing authentication requests. Leave blank to pull automatically from GitLab during OAuth registration. | +| `client_cert_key_pairs` | Client certificates and keys used for mutual TLS with the GitLab API. See [Support mutual TLS when calling the GitLab API](#support-mutual-tls-when-calling-the-gitlab-api) for details. [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/548) in GitLab 14.8. | | `dir` | Working directory for configuration and secrets files. | | `enable` | Enable or disable GitLab Pages on the current system. | | `external_http` | Configure Pages to bind to one or more secondary IP addresses, serving HTTP requests. Multiple addresses can be given as an array, along with exact ports, for example `['1.2.3.4', '1.2.3.5:8063']`. Sets value for `listen_http`. | @@ -511,6 +512,22 @@ Authority (CA) in the system certificate store. For Omnibus, this is fixed by [installing a custom CA in Omnibus GitLab](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). +### Support mutual TLS when calling the GitLab API + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/548) in GitLab 14.8. + +If GitLab has been [configured to require mutual TLS](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-2-way-ssl-client-authentication), you need to add the client certificates to Pages: + +1. Configure in `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_pages['client_cert_key_pairs'] = ['</path/to/cert>:</path/to/key>'] + ``` + + Where `</path/to/cert>` and `</path/to/key>` are the file paths to the client certificate and its respective key file. + Both of these files must be encoded in PEM format. +1. To configure Pages to validate the server certificates, [add the root CA to the system trust store](#using-a-custom-certificate-authority-ca). + ### ZIP serving and cache configuration > [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/392) in GitLab 13.7. @@ -688,7 +705,7 @@ To override the global maximum pages size for a specific group: ## Running GitLab Pages on a separate server You can run the GitLab Pages daemon on a separate server to decrease the load on -your main application server. This configuration does not support mutual TLS (mTLS). See the [corresponding feature proposal](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/548) for more information. +your main application server. To configure GitLab Pages on a separate server: @@ -1021,6 +1038,25 @@ Migrate your existing Pages deployments from local storage to object storage: sudo gitlab-rake gitlab:pages:deployments:migrate_to_object_storage ``` +You can track progress and verify that all Pages deployments migrated successfully using the +[PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database): + +- `sudo gitlab-rails dbconsole` for Omnibus GitLab instances. +- `sudo -u git -H psql -d gitlabhq_production` for source-installed instances. + +Verify `objectstg` below (where `store=2`) has count of all Pages deployments: + +```shell +gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM pages_deployments; + +total | filesystem | objectstg +------+------------+----------- + 10 | 0 | 10 +``` + +After verifying everything is working correctly, +[disable Pages local storage](#disable-pages-local-storage). + ### Rolling Pages deployments back to local storage After the migration to object storage is performed, you can choose to move your Pages deployments back to local storage: @@ -1033,7 +1069,7 @@ sudo gitlab-rake gitlab:pages:deployments:migrate_to_local > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301159) in GitLab 13.11. -If you use [object storage](#using-object-storage), you can disable local storage: +If you use [object storage](#using-object-storage), you can disable local storage to avoid unnecessary disk usage/writes: 1. Edit `/etc/gitlab/gitlab.rb`: @@ -1064,6 +1100,8 @@ is the default starting from GitLab 14.0. Skip this step if you're already runni 1. If you want to store your pages content in [object storage](#using-object-storage), make sure to configure it. If you want to store the pages content locally or continue using an NFS server, skip this step. 1. [Migrate legacy storage to ZIP storage.](#migrate-legacy-storage-to-zip-storage) +1. If you have configured GitLab to store your pages content in [object storage](#using-object-storage), + [migrate Pages deployments to object storage](#migrate-pages-deployments-to-object-storage) 1. Upgrade GitLab to 14.0. ## Backup @@ -1077,8 +1115,6 @@ than GitLab to prevent XSS attacks. ### Rate limits -> [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/631) in GitLab 14.5. - You can enforce rate limits to help minimize the risk of a Denial of Service (DoS) attack. GitLab Pages uses a [token bucket algorithm](https://en.wikipedia.org/wiki/Token_bucket) to enforce rate limiting. By default, requests that exceed the specified limits are reported but not rejected. @@ -1098,6 +1134,8 @@ Rate limits are enforced using the following: #### Enable source-IP rate limits +> [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/631) in GitLab 14.5. + 1. Set rate limits in `/etc/gitlab/gitlab.rb`: ```ruby @@ -1116,6 +1154,8 @@ Rate limits are enforced using the following: #### Enable domain rate limits +> [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/630) in GitLab 14.7. + 1. Set rate limits in `/etc/gitlab/gitlab.rb`: ```ruby @@ -1495,3 +1535,20 @@ behavior: NOTE: `inplace_chroot` option might not work with the other features, such as [Pages Access Control](#access-control). The [GitLab Pages README](https://gitlab.com/gitlab-org/gitlab-pages#caveats) has more information about caveats and workarounds. + +### GitLab Pages deploy job fails with error "is not a recognized provider" + +If the **pages** job succeeds but the **deploy** job gives the error "is not a recognized provider": + + + +The error message `is not a recognized provider` could be coming from the `fog` gem that GitLab uses to connect to cloud providers for object storage. + +To fix that: + +1. Check your `gitlab.rb` file. If you have `gitlab_rails['pages_object_store_enabled']` enabled, but no bucket details have been configured, either: + + - Configure object storage for your Pages deployments, following the [S3-compatible connection settings](#s3-compatible-connection-settings) guide. + - Store your deployments locally, by commenting out that line. + +1. Save the changes you made to your `gitlab.rb` file, then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). diff --git a/doc/administration/postgresql/database_load_balancing.md b/doc/administration/postgresql/database_load_balancing.md index b83820dd0b6..1324666c32b 100644 --- a/doc/administration/postgresql/database_load_balancing.md +++ b/doc/administration/postgresql/database_load_balancing.md @@ -147,9 +147,9 @@ When the list of hosts is updated, it might take a while for the old connections to be terminated. The `disconnect_timeout` setting can be used to enforce an upper limit on the time it takes to terminate all old database connections. -### Handling Stale Reads **(PREMIUM SELF)** +### Handling stale reads -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3526) in GitLab 10.3. +> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/327902) from GitLab Premium to GitLab Free in 14.0. To prevent reading from an outdated secondary the load balancer checks if it is in sync with the primary. If the data is recent enough, the diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index 5777f35bfcf..8c7151606a5 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -1029,7 +1029,7 @@ Considering these, you should carefully plan your PostgreSQL upgrade: ``` NOTE: - `gitlab-ctl pg-upgrade` tries to detect the role of the node. If for any reason the auto-detection does not work or you believe it did not detect the role correctly, you can use the `--leader` or `--replica` arguments to manually override it. + On a Geo secondary site, the Patroni leader node is called `standby leader`. 1. Stop Patroni **only on replicas**. @@ -1049,6 +1049,11 @@ Considering these, you should carefully plan your PostgreSQL upgrade: sudo gitlab-ctl pg-upgrade -V 12 ``` + NOTE: + `gitlab-ctl pg-upgrade` tries to detect the role of the node. If for any reason the auto-detection + does not work or you believe it did not detect the role correctly, you can use the `--leader` or + `--replica` arguments to manually override it. + 1. Check the status of the leader and cluster. You can proceed only if you have a healthy leader: ```shell diff --git a/doc/administration/raketasks/ldap.md b/doc/administration/raketasks/ldap.md index e6adb98a92a..cae2465fad8 100644 --- a/doc/administration/raketasks/ldap.md +++ b/doc/administration/raketasks/ldap.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index a5c60af47b1..eb1127b5e99 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -12,7 +12,7 @@ full list of reference architectures, see > - **Supported users (approximate):** 10,000 > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) -> - **Estimated Costs:** [GCP](https://cloud.google.com/products/calculator#id=e77713f6-dc0b-4bb3-bcef-cea904ac8efd) +> - **Estimated Costs:** [See cost table](index.md#cost-to-run) > - **Cloud Native Hybrid Alternative:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) > - **Performance tested daily with the [GitLab Performance Tool](https://gitlab.com/gitlab-org/quality/performance)**: > - **Test requests per second (RPS) rates:** API: 200 RPS, Web: 20 RPS, Git (Pull): 20 RPS, Git (Push): 4 RPS @@ -274,11 +274,11 @@ for details. ### Load balancer terminates SSL with backend SSL -Configure your load balancer(s) to use the 'HTTP(S)' protocol rather than 'TCP'. -The load balancer(s) will be responsible for managing SSL certificates that +Configure your load balancers to use the 'HTTP(S)' protocol rather than 'TCP'. +The load balancers will be responsible for managing SSL certificates that end users will see. -Traffic will also be secure between the load balancer(s) and NGINX in this +Traffic will also be secure between the load balancers and NGINX in this scenario. There is no need to add configuration for proxied SSL since the connection will be secure all the way. However, configuration will need to be added to GitLab to configure SSL certificates. See @@ -2234,7 +2234,7 @@ future with further specific cloud provider details. | Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | 15.5 vCPU, 50 GB memory | | Supporting services such as NGINX, Prometheus | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | 7.75 vCPU, 25 GB memory | -- For this setup, we **recommend** and regularly [test](index.md#testing-process-and-results) +- For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. - Nodes configuration is shown as it is forced to ensure pod vcpu / memory ratios and avoid scaling during **performance testing**. - In production deployments, there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices. diff --git a/doc/administration/reference_architectures/1k_users.md b/doc/administration/reference_architectures/1k_users.md index ed6fbe84a48..0d0e7681ffd 100644 --- a/doc/administration/reference_architectures/1k_users.md +++ b/doc/administration/reference_architectures/1k_users.md @@ -18,6 +18,7 @@ many organizations. > - **Supported users (approximate):** 1,000 > - **High Availability:** No. For a highly-available environment, you can > follow a modified [3K reference architecture](3k_users.md#supported-modifications-for-lower-user-counts-ha). +> - **Estimated Costs:** [See cost table](index.md#cost-to-run) > - **Cloud Native Hybrid:** No. For a cloud native hybrid environment, you > can follow a [modified hybrid reference architecture](#cloud-native-hybrid-reference-architecture-with-helm-charts). > - **Performance tested daily with the [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance)**: diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 8cc355db951..86819024eeb 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -12,7 +12,7 @@ full list of reference architectures, see > - **Supported users (approximate):** 25,000 > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) -> - **Estimated Costs:** [GCP](https://cloud.google.com/products/calculator#id=925386e1-c01c-4c0a-8d7d-ebde1824b7b0) +> - **Estimated Costs:** [See cost table](index.md#cost-to-run) > - **Cloud Native Hybrid Alternative:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) > - **Performance tested weekly with the [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance)**: > - **Test requests per second (RPS) rates:** API: 500 RPS, Web: 50 RPS, Git (Pull): 50 RPS, Git (Push): 10 RPS @@ -277,11 +277,11 @@ for details. ### Load balancer terminates SSL with backend SSL -Configure your load balancer(s) to use the 'HTTP(S)' protocol rather than 'TCP'. -The load balancer(s) will be responsible for managing SSL certificates that +Configure your load balancers to use the 'HTTP(S)' protocol rather than 'TCP'. +The load balancers will be responsible for managing SSL certificates that end users will see. -Traffic will also be secure between the load balancer(s) and NGINX in this +Traffic will also be secure between the load balancers and NGINX in this scenario. There is no need to add configuration for proxied SSL since the connection will be secure all the way. However, configuration will need to be added to GitLab to configure SSL certificates. See @@ -2232,7 +2232,7 @@ future with further specific cloud provider details. | Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | 15.5 vCPU, 50 GB memory | | Supporting services such as NGINX, Prometheus | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | 7.75 vCPU, 25 GB memory | -- For this setup, we **recommend** and regularly [test](index.md#testing-process-and-results) +- For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. - Nodes configuration is shown as it is forced to ensure pod vcpu / memory ratios and avoid scaling during **performance testing**. - In production deployments, there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices. diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index 467c64b8279..f6c484b08b1 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -13,7 +13,7 @@ For a full list of reference architectures, see > - **Supported users (approximate):** 2,000 > - **High Availability:** No. For a highly-available environment, you can > follow a modified [3K reference architecture](3k_users.md#supported-modifications-for-lower-user-counts-ha). -> - **Estimated Costs:** [GCP](https://cloud.google.com/products/calculator#id=84d11491-d72a-493c-a16e-650931faa658) +> - **Estimated Costs:** [See cost table](index.md#cost-to-run) > - **Cloud Native Hybrid:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) > - **Performance tested daily with the [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance)**: > - **Test requests per second (RPS) rates:** API: 40 RPS, Web: 4 RPS, Git (Pull): 4 RPS, Git (Push): 1 RPS @@ -1022,7 +1022,7 @@ future with further specific cloud provider details. | Sidekiq | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | 3.9 vCPU, 11.8 GB memory | | Supporting services such as NGINX, Prometheus | 2 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | 1.9 vCPU, 5.5 GB memory | -- For this setup, we **recommend** and regularly [test](index.md#testing-process-and-results) +- For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. - Nodes configuration is shown as it is forced to ensure pod vcpu / memory ratios and avoid scaling during **performance testing**. - In production deployments, there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices. diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index 01d9987739b..587303a1f8f 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -22,7 +22,7 @@ For a full list of reference architectures, see > - **Supported users (approximate):** 3,000 > - **High Availability:** Yes, although [Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution -> - **Estimated Costs:** [GCP](https://cloud.google.com/products/calculator/#id=ac4838e6-9c40-4a36-ac43-6d1bc1843e08) +> - **Estimated Costs:** [See cost table](index.md#cost-to-run) > - **Cloud Native Hybrid Alternative:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) > - **Performance tested weekly with the [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance)**: > - **Test requests per second (RPS) rates:** API: 60 RPS, Web: 6 RPS, Git (Pull): 6 RPS, Git (Push): 1 RPS @@ -278,11 +278,11 @@ for details. ### Load balancer terminates SSL with backend SSL -Configure your load balancer(s) to use the 'HTTP(S)' protocol rather than 'TCP'. -The load balancer(s) will be responsible for managing SSL certificates that +Configure your load balancers to use the 'HTTP(S)' protocol rather than 'TCP'. +The load balancers will be responsible for managing SSL certificates that end users will see. -Traffic will also be secure between the load balancer(s) and NGINX in this +Traffic will also be secure between the load balancers and NGINX in this scenario. There is no need to add configuration for proxied SSL since the connection will be secure all the way. However, configuration will need to be added to GitLab to configure SSL certificates. See @@ -2191,7 +2191,7 @@ future with further specific cloud provider details. | Sidekiq | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | 11.8 vCPU, 38.9 GB memory | | Supporting services such as NGINX, Prometheus | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | 3.9 vCPU, 11.8 GB memory | -- For this setup, we **recommend** and regularly [test](index.md#testing-process-and-results) +- For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. - Nodes configuration is shown as it is forced to ensure pod vcpu / memory ratios and avoid scaling during **performance testing**. - In production deployments, there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices. diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index d5bb9c4ad64..f4bf232d548 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -12,7 +12,7 @@ full list of reference architectures, see > - **Supported users (approximate):** 50,000 > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) -> - **Estimated Costs:** [GCP](https://cloud.google.com/products/calculator/#id=8006396b-88ee-40cd-a1c8-77cdefa4d3c8) +> - **Estimated Costs:** [See cost table](index.md#cost-to-run) > - **Cloud Native Hybrid Alternative:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) > - **Performance tested weekly with the [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance)**: > - **Test requests per second (RPS) rates:** API: 1000 RPS, Web: 100 RPS, Git (Pull): 100 RPS, Git (Push): 20 RPS @@ -283,11 +283,11 @@ for details. ### Load balancer terminates SSL with backend SSL -Configure your load balancer(s) to use the 'HTTP(S)' protocol rather than 'TCP'. -The load balancer(s) will be responsible for managing SSL certificates that +Configure your load balancers to use the 'HTTP(S)' protocol rather than 'TCP'. +The load balancers will be responsible for managing SSL certificates that end users will see. -Traffic will also be secure between the load balancer(s) and NGINX in this +Traffic will also be secure between the load balancers and NGINX in this scenario. There is no need to add configuration for proxied SSL since the connection will be secure all the way. However, configuration will need to be added to GitLab to configure SSL certificates. See @@ -2248,7 +2248,7 @@ future with further specific cloud provider details. | Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | 15.5 vCPU, 50 GB memory | | Supporting services such as NGINX, Prometheus | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | 7.75 vCPU, 25 GB memory | -- For this setup, we **recommend** and regularly [test](index.md#testing-process-and-results) +- For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. - Nodes configuration is shown as it is forced to ensure pod vcpu / memory ratios and avoid scaling during **performance testing**. - In production deployments, there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices. diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 33ca4e4899f..4014ec04904 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -19,7 +19,7 @@ costly-to-operate environment by using the > - **Supported users (approximate):** 5,000 > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) -> - **Estimated Costs:** [GCP](https://cloud.google.com/products/calculator/#id=8742e8ea-c08f-4e0a-b058-02f3a1c38a2f) +> - **Estimated Costs:** [See cost table](index.md#cost-to-run) > - **Cloud Native Hybrid Alternative:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) > - **Performance tested weekly with the [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance)**: > - **Test requests per second (RPS) rates:** API: 100 RPS, Web: 10 RPS, Git (Pull): 10 RPS, Git (Push): 2 RPS @@ -276,11 +276,11 @@ for details. ### Load balancer terminates SSL with backend SSL -Configure your load balancer(s) to use the 'HTTP(S)' protocol rather than 'TCP'. -The load balancer(s) will be responsible for managing SSL certificates that +Configure your load balancers to use the 'HTTP(S)' protocol rather than 'TCP'. +The load balancers will be responsible for managing SSL certificates that end users will see. -Traffic will also be secure between the load balancer(s) and NGINX in this +Traffic will also be secure between the load balancers and NGINX in this scenario. There is no need to add configuration for proxied SSL since the connection will be secure all the way. However, configuration will need to be added to GitLab to configure SSL certificates. See @@ -2167,7 +2167,7 @@ future with further specific cloud provider details. | Sidekiq | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | 11.8 vCPU, 38.9 GB memory | | Supporting services such as NGINX, Prometheus | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | 3.9 vCPU, 11.8 GB memory | -- For this setup, we **recommend** and regularly [test](index.md#testing-process-and-results) +- For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. - Nodes configuration is shown as it is forced to ensure pod vcpu / memory ratios and avoid scaling during **performance testing**. - In production deployments, there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices. diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index bd796600564..815155866e8 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -59,7 +59,9 @@ provided architecture. ## Available reference architectures -The following reference architectures are available: +The following reference architectures are available. + +### GitLab package (Omnibus) - [Up to 1,000 users](1k_users.md) - [Up to 2,000 users](2k_users.md) @@ -69,6 +71,8 @@ The following reference architectures are available: - [Up to 25,000 users](25k_users.md) - [Up to 50,000 users](50k_users.md) +### Cloud native hybrid + The following Cloud Native Hybrid reference architectures, where select recommended components can be run in Kubernetes, are available: - [Up to 2,000 users](2k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) @@ -83,6 +87,204 @@ to get assistance from Support with troubleshooting the [2,000 users](2k_users.m and higher reference architectures. [Read more about our definition of scaled architectures](https://about.gitlab.com/support/#definition-of-scaled-architecture). +### Validation and test results + +The [Quality Engineering - Enablement team](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/) does regular smoke and performance tests for the reference architectures to ensure they remain compliant. + +- Testing occurs against all reference architectures and cloud providers in an automated and ad-hoc fashion. This is done by two tools: + - The [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/gitlab-environment-toolkit) for building the environments. + - The [GitLab Performance Tool](https://gitlab.com/gitlab-org/quality/performance) for performance testing. +- Network latency on the test environments between components on all Cloud Providers were measured at <5ms. Note that this is shared as an observation and not as an implicit recommendation. +- We aim to have a "test smart" approach where architectures tested have a good range that can also apply to others. Testing focuses on 10k Omnibus on GCP as the testing has shown this is a good bellwether for the other architectures and cloud providers as well as Cloud Native Hybrids. +- Testing is done publicly and all results are shared. +- For more information about performance testing at GitLab, read [how our QA team leverages GitLab’s performance testing tool (and you can too)](https://about.gitlab.com/blog/2020/02/18/how-were-building-up-performance-testing-of-gitlab/). + +The following table details the testing done against the reference architectures along with the frequency and results. Additional testing is continuously evaluated, and the table is updated accordingly. + +<style> +table.test-coverage td { + border-left: 1px solid #dbdbdb; + border-right: 1px solid #dbdbdb; + border-bottom: 1px solid #dbdbdb; +} + +table.test-coverage th { + border-left: 1px solid #dbdbdb; + border-right: 1px solid #dbdbdb; + border-bottom: 1px solid #dbdbdb; +} +</style> + +<table class="test-coverage"> + <col> + <colgroup span="2"></colgroup> + <colgroup span="2"></colgroup> + <tr> + <th rowspan="2">Reference<br/>Architecture</th> + <th style="text-align: center" colspan="2" scope="colgroup">GCP (* also proxy for Bare-Metal)</th> + <th style="text-align: center" colspan="2" scope="colgroup">AWS</th> + <th style="text-align: center" colspan="2" scope="colgroup">Azure</th> + </tr> + <tr> + <th scope="col">Omnibus</th> + <th scope="col">Cloud Native Hybrid</th> + <th scope="col">Omnibus</th> + <th scope="col">Cloud Native Hybrid</th> + <th scope="col">Omnibus</th> + <th scope="col">Cloud Native Hybrid</th> + </tr> + <tr> + <th scope="row">1k</th> + <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/1k">Daily</a> (to be moved to Weekly)</td> + <td></td> + <td></td> + <td></td> + <td></td> + <td></td> + </tr> + <tr> + <th scope="row">2k</th> + <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/2k">Daily</a> (to be moved to Weekly)</td> + <td></td> + <td></td> + <td></td> + <td></td> + <td></td> + </tr> + <tr> + <th scope="row">3k</th> + <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/3k">Weekly</a></td> + <td></td> + <td></td> + <td></td> + <td></td> + <td></td> + </tr> + <tr> + <th scope="row">5k</th> + <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/5k">Weekly</a></td> + <td></td> + <td></td> + <td></td> + <td></td> + <td></td> + </tr> + <tr> + <th scope="row">10k</th> + <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k">Daily</a></td> + <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/10k-Cloud-Native-Hybrid">Ad-Hoc</a></td> + <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/10k">Ad-Hoc (inc Cloud Services)</a></td> + <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/10k-Cloud-Native-Hybrid">Ad-Hoc</a></td> + <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/10k">Ad-Hoc</a></td> + <td></td> + </tr> + <tr> + <th scope="row">25k</th> + <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/25k">Weekly</a></td> + <td></td> + <td></td> + <td></td> + <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/25k">Ad-Hoc</a></td> + <td></td> + </tr> + <tr> + <th scope="row">50k</th> + <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/50k">Weekly</a></td> + <td></td> + <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/50k">Ad-Hoc (inc Cloud Services)</a></td> + <td></td> + <td></td> + <td></td> + </tr> +</table> + +The Standard Reference Architectures are designed to be platform agnostic, with everything being run on VMs via [Omnibus GitLab](https://docs.gitlab.com/omnibus/). While testing occurs primarily on GCP, ad-hoc testing has shown that they perform similarly on equivalently specced hardware on other Cloud Providers or if run on premises (bare-metal). + +### Cost to run + +<table class="test-coverage"> + <col> + <colgroup span="2"></colgroup> + <colgroup span="2"></colgroup> + <tr> + <th rowspan="2">Reference<br/>Architecture</th> + <th style="text-align: center" colspan="2" scope="colgroup">GCP</th> + <th style="text-align: center" colspan="2" scope="colgroup">AWS</th> + <th style="text-align: center" colspan="2" scope="colgroup">Azure</th> + </tr> + <tr> + <th scope="col">Omnibus</th> + <th scope="col">Cloud Native Hybrid</th> + <th scope="col">Omnibus</th> + <th scope="col">Cloud Native Hybrid</th> + <th scope="col">Omnibus</th> + <th scope="col">Cloud Native Hybrid</th> + </tr> + <tr> + <th scope="row">1k</th> + <td><a href="https://cloud.google.com/products/calculator#id=a6d6a94a-c7dc-4c22-85c4-7c5747f272ed">Calculated cost</a></td> + <td></td> + <td></td> + <td></td> + <td></td> + <td></td> + </tr> + <tr> + <th scope="row">2k</th> + <td><a href="https://cloud.google.com/products/calculator#id=84d11491-d72a-493c-a16e-650931faa658">Calculated cost</a></td> + <td></td> + <td></td> + <td></td> + <td></td> + <td></td> + </tr> + <tr> + <th scope="row">3k</th> + <td><a href="https://cloud.google.com/products/calculator/#id=ac4838e6-9c40-4a36-ac43-6d1bc1843e08">Calculated cost</a></td> + <td></td> + <td></td> + <td></td> + <td></td> + <td></td> + </tr> + <tr> + <th scope="row">5k</th> + <td><a href="https://cloud.google.com/products/calculator/#id=8742e8ea-c08f-4e0a-b058-02f3a1c38a2f">Calculated cost</a></td> + <td></td> + <td></td> + <td></td> + <td></td> + <td></td> + </tr> + <tr> + <th scope="row">10k</th> + <td><a href="https://cloud.google.com/products/calculator#id=e77713f6-dc0b-4bb3-bcef-cea904ac8efd">Calculated cost</a></td> + <td></td> + <td></td> + <td></td> + <td></td> + <td></td> + </tr> + <tr> + <th scope="row">25k</th> + <td><a href="https://cloud.google.com/products/calculator#id=925386e1-c01c-4c0a-8d7d-ebde1824b7b0">Calculated cost</a></td> + <td></td> + <td></td> + <td></td> + <td></td> + <td></td> + </tr> + <tr> + <th scope="row">50k</th> + <td><a href="https://cloud.google.com/products/calculator/#id=8006396b-88ee-40cd-a1c8-77cdefa4d3c8">Calculated cost</a></td> + <td></td> + <td></td> + <td></td> + <td></td> + <td></td> + </tr> +</table> + ## Availability Components GitLab comes with the following components for your use, listed from least to @@ -191,33 +393,3 @@ The reference architectures for user counts [3,000](3k_users.md) and up support In the specific case you have the requirement to achieve HA but have a lower user count, select modifications to the [3,000 user](3k_users.md) architecture are supported. For more details, [refer to this section in the architecture's documentation](3k_users.md#supported-modifications-for-lower-user-counts-ha). - -## Testing process and results - -The [Quality Engineering - Enablement team](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/) does regular smoke and performance tests for the reference architectures to ensure they remain compliant. - -In this section, we detail some of the process as well as the results. - -Note the following about the testing process: - -- Testing occurs against all main reference architectures and cloud providers in an automated and ad-hoc fashion. - This is achieved through two tools built by the team: - - The [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit) for building the environments. - - The [GitLab Performance Tool](https://gitlab.com/gitlab-org/quality/performance) for performance testing. -- Network latency on the test environments between components on all Cloud Providers were measured at <5ms. Note that this is shared as an observation and not as an implicit recommendation. -- We aim to have a "test smart" approach where architectures tested have a good range that can also apply to others. Testing focuses on 10k Omnibus on GCP as the testing has shown this is a good bellwether for the other architectures and cloud providers as well as Cloud Native Hybrids. -- Testing is done publicly and all results are shared. - -Τhe following table details the testing done against the reference architectures along with the frequency and results. Note that this list above is non exhaustive. Additional testing is continuously evaluated and iterated on, and the table is updated accordingly. - -| Reference<br/>Architecture<br/>Size | Bare-Metal | GCP | AWS | Azure | -|-----------------------------|------------|-----|-----|-------| -| 1k | <i>Refer to GCP<sup>1</sup></i> | [Standard - Weekly](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/1k)<sup>1</sup> | - | - | -| 2k | <i>Refer to GCP<sup>1</sup></i> | [Standard - Weekly](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/2k)<sup>1</sup> | - | - | -| 3k | <i>Refer to GCP<sup>1</sup></i> | [Standard - Weekly](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/3k)<sup>1</sup> | - | - | -| 5k | <i>Refer to GCP<sup>1</sup></i> | [Standard - Weekly](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/5k)<sup>1</sup> | - | - | -| 10k | <i>Refer to GCP<sup>1</sup></i> | [Standard - Daily](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k)<sup>1</sup> <br/> [Standard (inc Cloud Services) - Ad-Hoc](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/10k) <br/> [Cloud Native Hybrid - Ad-Hoc](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/10k-Cloud-Native-Hybrid) | [Standard (inc Cloud Services) - Ad-Hoc](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/10k) <br/> [Cloud Native Hybrid - Ad-Hoc](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/10k-Cloud-Native-Hybrid) | [Standard - Ad-Hoc](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/10k) | -| 25k | <i>Refer to GCP<sup>1</sup></i> | [Standard - Weekly](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/25k)<sup>1</sup> | - | [Standard - Ad-Hoc](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/25k) | -| 50k | <i>Refer to GCP<sup>1</sup></i> | [Standard - Weekly](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/50k)<sup>1</sup> | [Standard (inc Cloud Services) - Ad-Hoc](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/50k) | - | - -1. The Standard Reference Architectures are designed to be platform agnostic, with everything being run on VMs via [Omnibus GitLab](https://docs.gitlab.com/omnibus/). While testing occurs primarily on GCP, ad-hoc testing has shown that they perform similarly on equivalently specced hardware on other Cloud Providers or if run on premises (bare-metal). diff --git a/doc/administration/restart_gitlab.md b/doc/administration/restart_gitlab.md index 9579d413bf8..88909b2b9d8 100644 --- a/doc/administration/restart_gitlab.md +++ b/doc/administration/restart_gitlab.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # How to restart GitLab **(FREE SELF)** Depending on how you installed GitLab, there are different methods to restart -its service(s). +its services. ## Omnibus installations diff --git a/doc/administration/server_hooks.md b/doc/administration/server_hooks.md index 2a431d17774..9d8bc03e5e9 100644 --- a/doc/administration/server_hooks.md +++ b/doc/administration/server_hooks.md @@ -10,171 +10,135 @@ disqus_identifier: 'https://docs.gitlab.com/ee/administration/custom_hooks.html' > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/196051) in GitLab 12.8 replacing Custom Hooks. -Git supports hooks that are executed on different actions. These hooks run on the server and can be -used to enforce specific commit policies or perform other tasks based on the state of the -repository. +Server hooks run custom logic on the GitLab server. Users can use them to run Git-related tasks such as: -Git supports the following hooks: +- Enforcing specific commit policies. +- Performing tasks based on the state of the repository. -- `pre-receive` -- `post-receive` -- `update` +Server hooks use `pre-receive`, `post-receive`, and `update` +[Git server-side hooks](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks#_server_side_hooks). -See [the Git documentation](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks#_server_side_hooks) -for more information about each hook type. +GitLab administrators configure server hooks on the file system of the GitLab server. If you don't have file system access, +alternatives to server hooks include: -Server-side Git hooks can be configured for: +- [Webhooks](../user/project/integrations/webhooks.md). +- [GitLab CI/CD](../ci/index.md). +- [Push rules](../push_rules/push_rules.md), for a user-configurable Git hook interface. -- [A single repository](#create-a-server-hook-for-a-repository). -- [All repositories](#create-a-global-server-hook-for-all-repositories). +[Geo](geo/index.md) doesn't replicate server hooks to secondary nodes. -Note the following about server hooks: +## Create a server hook for a single repository -- Server hooks must be configured on the file system of the GitLab server. Only GitLab server - administrators are able to complete these tasks. If you don't have file system access, see - possible alternatives such as: - - [Webhooks](../user/project/integrations/webhooks.md). - - [GitLab CI/CD](../ci/index.md). - - [Push Rules](../push_rules/push_rules.md), for a user-configurable Git hook - interface. -- Server hooks aren't replicated to [Geo](geo/index.md) secondary nodes. +To create a server hook for a single repository: -## Create a server hook for a repository +1. On the top bar, select **Menu > Admin**. +1. Go to **Overview > Projects** and select the project you want to add a server hook to. +1. On the page that appears, locate the value of **Gitaly relative path**. This path is where server hooks must be located. + - If you are using [hashed storage](repository_storage_types.md#hashed-storage), see + [Translate hashed storage paths](repository_storage_types.md#translate-hashed-storage-paths) for information on + interpreting the relative path. + - If you are not using [hashed storage](repository_storage_types.md#hashed-storage): + - For Omnibus GitLab installations, the path is usually `/var/opt/gitlab/git-data/repositories/<group>/<project>.git`. + - For an installation from source, the path is usually `/home/git/repositories/<group>/<project>.git`. +1. On the file system, create a new directory in the correct location called `custom_hooks`. +1. In the new `custom_hooks` directory, create a file with a name that matches the hook type. For example, for a + `pre-receive` server hook, the filename should be `pre-receive` with no extension. +1. Make the server hook file executable and ensure that it's owned by the Git user. +1. Write the code to make the server hook function as expected. Server hooks can be in any programming language. Ensure + the [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) at the top reflects the language type. For + example, if the script is in Ruby the shebang is probably `#!/usr/bin/env ruby`. -If you are not using [hashed storage](repository_storage_types.md#hashed-storage), the project's -repository directory might not exactly match the instructions below. In that case: - -- For an installation from source, the path is usually - `/home/git/repositories/<group>/<project>.git`. -- For Omnibus GitLab installs, the path is usually - `/var/opt/gitlab/git-data/repositories/<group>/<project>.git`. - -Follow the steps below to set up a server-side hook for a repository: - -1. Go to **Admin area > Projects** and select the project you want to add a server hook to. -1. Locate the **Gitaly relative path** on the page that appears. This is where the server hook - must be implemented. For information on interpreting the relative path, see - [Translate hashed storage paths](repository_storage_types.md#translate-hashed-storage-paths). -1. On the file system, create a new directory in this location called `custom_hooks`. -1. Inside the new `custom_hooks` directory, create a file with a name matching the hook type. For - example, for a pre-receive hook the filename should be `pre-receive` with no extension. -1. Make the hook file executable and ensure that it's owned by the Git user. -1. Write the code to make the server hook function as expected. Hooks can be in any language. Ensure - the ["shebang"](https://en.wikipedia.org/wiki/Shebang_(Unix)) at the top properly reflects the - language type. For example, if the script is in Ruby the shebang is probably - `#!/usr/bin/env ruby`. - -Assuming the hook code is properly implemented, the hook code is executed as appropriate. +If the server hook code is properly implemented, it should execute when the Git hook is next triggered. ## Create a global server hook for all repositories -To create a Git hook that applies to all of the repositories in your instance, set a global server -hook. The default global server hook directory is in the GitLab Shell directory. Any -hook added there applies to all repositories, including: +To create a Git hook that applies to all repositories, set a global server hook. The default global server hook directory +is in the GitLab Shell directory. Any server hook added there applies to all repositories, including: + +- [Project and group wiki](../user/project/wiki/index.md) repositories. Their storage directory names are in the format + `<id>.wiki.git`. +- [Design management](../user/project/issues/design_management.md) repositories under a project. Their storage directory + names are in the format `<id>.design.git`. -- [Project and group wiki](../user/project/wiki/index.md) repositories, - whose storage directory names are in the format `<id>.wiki.git`. -- [Design management](../user/project/issues/design_management.md) repositories under a - project, whose storage directory names are in the format `<id>.design.git`. +### Choose a server hook directory -The default directory: +Before creating a global server hook, you must choose a directory for it. The default global server hook directory: +- For Omnibus GitLab installations is usually `/opt/gitlab/embedded/service/gitlab-shell/hooks`. - For an installation from source is usually `/home/git/gitlab-shell/hooks`. -- For Omnibus GitLab installs is usually `/opt/gitlab/embedded/service/gitlab-shell/hooks`. -To use a different directory for global server hooks, set `custom_hooks_dir` in Gitaly -configuration: +To use a different directory for global server hooks, set `custom_hooks_dir` in Gitaly configuration: -- For Omnibus installations, this is set in `gitlab.rb`. +- For Omnibus installations, set in `gitlab.rb`. - For source installations, the configuration location depends on the GitLab version. For: - - GitLab 13.0 and earlier, this is set in `gitlab-shell/config.yml`. - - GitLab 13.1 and later, this is set in `gitaly/config.toml` under the `[hooks]` section. - -NOTE: -The `custom_hooks_dir` value in `gitlab-shell/config.yml` is still honored in GitLab 13.1 and later -if the value in `gitaly/config.toml` is blank or non-existent. + - GitLab 13.0 and earlier, set in `gitlab-shell/config.yml`. + - GitLab 13.1 and later, set in `gitaly/config.toml` under the `[hooks]` section. However, GitLab honors the + `custom_hooks_dir` value in `gitlab-shell/config.yml` if the value in `gitaly/config.toml` is blank or non-existent. -Follow the steps below to set up a global server hook for all repositories: +### Create the global server hook -1. On the GitLab server, navigate to the configured global server hook directory. -1. Create a new directory in this location. Depending on the type of hook, it can be either a - `pre-receive.d`, `post-receive.d`, or `update.d` directory. -1. Inside this new directory, add your hook. Hooks can be in any language. Ensure the - ["shebang"](https://en.wikipedia.org/wiki/Shebang_(Unix)) at the top properly reflects the - language type. For example, if the script is in Ruby the shebang is probably - `#!/usr/bin/env ruby`. -1. Make the hook file executable and ensure that it's owned by the Git user. +To create a global server hook for all repositories: -Now test the hook to check whether it is functioning properly. +1. On the GitLab server, go to the configured global server hook directory. +1. Create a new directory in this location called `pre-receive.d`, `post-receive.d`, or `update.d`, depending on the type + of server hook. Any other names are ignored. +1. Inside this new directory, add your server hook. Server hooks can be in any programming language. Ensure the + [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) at the top reflects the language type. For example, if the + script is in Ruby the shebang is probably `#!/usr/bin/env ruby`. +1. Make the hook file executable, ensure that it's owned by the Git user, and ensure it does not match the backup file + pattern (`*~`). -## Chained hooks +If the server hook code is properly implemented, it should execute when the Git hook is next triggered. -Server hooks set [per project](#create-a-server-hook-for-a-repository) or -[globally](#create-a-global-server-hook-for-all-repositories) can be executed in a chain. +## Chained server hooks -Server hooks are searched for and executed in the following order of priority: +GitLab can execute server hooks in a chain. GitLab searches for and executes server hooks in the following order: -- Built-in GitLab server hooks. These are not user-customizable. -- `<project>.git/custom_hooks/<hook_name>`: Per-project hooks. This was kept for backwards - compatibility. +- Built-in GitLab server hooks. These server hooks are not customizable by users. +- `<project>.git/custom_hooks/<hook_name>`: Per-project hooks. This location is kept for backwards compatibility. - `<project>.git/custom_hooks/<hook_name>.d/*`: Location for per-project hooks. -- `<custom_hooks_dir>/<hook_name>.d/*`: Location for all executable global hook files - except editor backup files. +- `<custom_hooks_dir>/<hook_name>.d/*`: Location for all executable global hook files except editor backup files. -Within a directory, server hooks: +Within a server hooks directory, hooks: - Are executed in alphabetical order. - Stop executing when a hook exits with a non-zero value. -`<hook_name>.d` must be either `pre-receive.d`, `post-receive.d`, or `update.d` to work properly. -Any other names are ignored. +## Environment variables available to server hooks -Files in `.d` directories must be executable and not match the backup file pattern (`*~`). +You can pass any environment variable to server hooks, but you should only rely on supported environment variables. -For `<project>.git` you need to [translate](repository_storage_types.md#translate-hashed-storage-paths) -your project name into the hashed storage format that GitLab uses. +The following GitLab environment variables are supported for all server hooks: -## Environment Variables +| Environment variable | Description | +|:---------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------| +| `GL_ID` | GitLab identifier of user that initiated the push. For example, `user-2234`. | +| `GL_PROJECT_PATH` | (GitLab 13.2 and later) GitLab project path. | +| `GL_PROTOCOL` | (GitLab 13.2 and later) Protocol used for this change. One of: `http` (Git `push` using HTTP), `ssh` (Git `push` using SSH), or `web` (all other actions). | +| `GL_REPOSITORY` | `project-<id>` where `id` is the ID of the project. | +| `GL_USERNAME` | GitLab username of the user that initiated the push. | -The following set of environment variables are available to server hooks. - -| Environment variable | Description | -|:---------------------|:----------------------------------------------------------------------------| -| `GL_ID` | GitLab identifier of user that initiated the push. For example, `user-2234` | -| `GL_PROJECT_PATH` | (GitLab 13.2 and later) GitLab project path | -| `GL_PROTOCOL` | (GitLab 13.2 and later) Protocol used for this change. One of: `http` (Git Push using HTTP), `ssh` (Git Push using SSH), or `web` (all other actions). | -| `GL_REPOSITORY` | `project-<id>` where `id` is the ID of the project | -| `GL_USERNAME` | GitLab username of the user that initiated the push | - -Pre-receive and post-receive server hooks can also access the following Git environment variables. +The following Git environment variables are supported for `pre-receive` and `post-receive` server hooks: | Environment variable | Description | |:-----------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `GIT_ALTERNATE_OBJECT_DIRECTORIES` | Alternate object directories in the quarantine environment. See [Git `receive-pack` documentation](https://git-scm.com/docs/git-receive-pack#_quarantine_environment). | | `GIT_OBJECT_DIRECTORY` | GitLab project path in the quarantine environment. See [Git `receive-pack` documentation](https://git-scm.com/docs/git-receive-pack#_quarantine_environment). | -| `GIT_PUSH_OPTION_COUNT` | Number of push options. See [Git `pre-receive` documentation](https://git-scm.com/docs/githooks#pre-receive). | -| `GIT_PUSH_OPTION_<i>` | Value of push options where `i` is from `0` to `GIT_PUSH_OPTION_COUNT - 1`. See [Git `pre-receive` documentation](https://git-scm.com/docs/githooks#pre-receive). | - -NOTE: -While other environment variables can be passed to server hooks, your application should not rely on -them as they can change. +| `GIT_PUSH_OPTION_COUNT` | Number of [push options](../user/project/push_options.md). See [Git `pre-receive` documentation](https://git-scm.com/docs/githooks#pre-receive). | +| `GIT_PUSH_OPTION_<i>` | Value of [push options](../user/project/push_options.md) where `i` is from `0` to `GIT_PUSH_OPTION_COUNT - 1`. See [Git `pre-receive` documentation](https://git-scm.com/docs/githooks#pre-receive). | ## Custom error messages -To have custom error messages appear in the GitLab UI when a commit is declined or an error occurs -during the Git hook, your script should: +You can have custom error messages appear in the GitLab UI when a commit is declined or an error occurs during the Git +hook. To display a custom error message, your script must: - Send the custom error messages to either the script's `stdout` or `stderr`. - Prefix each message with `GL-HOOK-ERR:` with no characters appearing before the prefix. -### Example custom error message - -This hook script written in Bash generates the following message in the GitLab UI: +For example: ```shell #!/bin/sh echo "GL-HOOK-ERR: My custom error message."; exit 1 ``` - - diff --git a/doc/administration/sidekiq.md b/doc/administration/sidekiq.md index 989a024d6ab..fea6ac9e554 100644 --- a/doc/administration/sidekiq.md +++ b/doc/administration/sidekiq.md @@ -4,77 +4,131 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Configuring Sidekiq **(FREE SELF)** +# Configure an external Sidekiq instance **(FREE SELF)** -This section discusses how to configure an external Sidekiq instance using the -bundled Sidekiq in the GitLab package. +You can configure an external Sidekiq instance by using the Sidekiq that's +bundled in the GitLab package. Sidekiq requires connection to the Redis, +PostgreSQL, and Gitaly instances. -Sidekiq requires connection to the Redis, PostgreSQL and Gitaly instance. -To configure the Sidekiq node: +## Required configuration + +To configure Sidekiq: 1. SSH into the Sidekiq server. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab package -you want using steps 1 and 2 from the GitLab downloads page. -**Do not complete any other steps on the download page.** -1. Open `/etc/gitlab/gitlab.rb` with your editor. -1. Generate the Sidekiq configuration: +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package + using steps 1 and 2. **Do not complete any other steps.** +1. Edit `/etc/gitlab/gitlab.rb` with the following information and make sure + to replace with your values: ```ruby - ## Optional: Enable extra Sidekiq processes - sidekiq_cluster['enable'] = true - sidekiq['queue_groups'] = [ - "elastic_commit_indexer", - "*" - ] - ``` + ## + ## To maintain uniformity of links across nodes, the + ##`external_url` on the Sidekiq server should point to the external URL that users + ## use to access GitLab. This can be either: + ## + ## - The `external_url` set on your application server. + ## - The URL of a external load balancer, which routes traffic to the GitLab application server. + ## -1. Setup Sidekiq's connection to Redis: + external_url 'https://gitlab.example.com' + + ## Prevent database migrations from running on upgrade automatically + gitlab_rails['auto_migrate'] = false + + ######################################## + ##### Services Disabled ### + ######################################## + # + # When running GitLab on just one server, you have a single `gitlab.rb` + # to enable all services you want to run. + # When running GitLab on N servers, you have N `gitlab.rb` files. + # Enable only the services you want to run on each + # specific server, while disabling all others. + # + nginx['enable'] = false + grafana['enable'] = false + prometheus['enable'] = false + gitlab_rails['auto_migrate'] = false + alertmanager['enable'] = false + gitaly['enable'] = false + gitlab_workhorse['enable'] = false + nginx['enable'] = false + postgres_exporter['enable'] = false + postgresql['enable'] = false + redis['enable'] = false + redis_exporter['enable'] = false + puma['enable'] = false + gitlab_exporter['enable'] = false + + ####################################### + ### Sidekiq configuration ### + ####################################### + sidekiq['enable'] = true + sidekiq['listen_address'] = "0.0.0.0" + + ## Set number of Sidekiq queue processes to the same number as available CPUs + sidekiq['queue_groups'] = ['*'] * 4 + + ## Set number of Sidekiq threads per queue process to the recommend number of 10 + sidekiq['max_concurrency'] = 10 + + ######################################## + #### Redis ### + ######################################## - ```ruby ## Must be the same in every sentinel node redis['master_name'] = 'gitlab-redis' ## The same password for Redis authentication you set up for the master node. - redis['master_password'] = 'YOUR_PASSOWORD' - - ## A list of sentinels with `host` and `port` - gitlab_rails['redis_sentinels'] = [ - {'host' => '10.10.1.34', 'port' => 26379}, - {'host' => '10.10.1.35', 'port' => 26379}, - {'host' => '10.10.1.36', 'port' => 26379}, - ] - ``` + redis['master_password'] = '<redis_master_password>' -1. Set up Sidekiq's connection to Gitaly: + ####################################### + ### Gitaly ### + ####################################### - ```ruby + ## Replace <gitaly_token> with the one you set up, see + ## https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#about-the-gitaly-token git_data_dirs({ 'default' => { 'gitaly_address' => 'tcp://gitaly:8075' }, }) - gitlab_rails['gitaly_token'] = 'YOUR_TOKEN' - ``` + gitlab_rails['gitaly_token'] = '<gitaly_token>' -1. Set up Sidekiq's connection to PostgreSQL: + ####################################### + ### Postgres ### + ####################################### - ```ruby - gitlab_rails['db_host'] = '10.10.1.30' - gitlab_rails['db_password'] = 'YOUR_PASSOWORD' + # Replace <database_host> and <database_password> + gitlab_rails['db_host'] = '<database_host>' + gitlab_rails['db_password'] = '<database_password>' gitlab_rails['db_port'] = '5432' gitlab_rails['db_adapter'] = 'postgresql' gitlab_rails['db_encoding'] = 'unicode' gitlab_rails['auto_migrate'] = false + + # Add the Sidekiq nodes to PostgreSQL's trusted addresses. + # In the following example, 10.10.1.30/32 is the private IP + # of the Sidekiq server. + postgresql['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.10.1.30/32) ``` - Remember to add the Sidekiq nodes to PostgreSQL's trusted addresses: +1. Reconfigure GitLab: - ```ruby - postgresql['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.10.1.30/32 10.10.1.31/32 10.10.1.32/32 10.10.1.33/32 10.10.1.38/32) + ```shell + sudo gitlab-ctl reconfigure ``` -1. If you run multiple Sidekiq nodes with a shared file storage, such as NFS, you must - specify the UIDs and GIDs to ensure they match between servers. Specifying the UIDs - and GIDs prevents permissions issues in the file system. This advice is similar to the - [advice for Geo setups](geo/replication/multiple_servers.md#step-4-configure-the-frontend-application-nodes-on-the-geo-secondary-site): +1. Restart the Sidekiq nodes after completing the process and finishing the database migrations. + +## Configure multiple Sidekiq nodes with shared storage + +If you run multiple Sidekiq nodes with a shared file storage, such as NFS, you must +specify the UIDs and GIDs to ensure they match between servers. Specifying the UIDs +and GIDs prevents permissions issues in the file system. This advice is similar to the +[advice for Geo setups](geo/replication/multiple_servers.md#step-4-configure-the-frontend-application-nodes-on-the-geo-secondary-site). + +To set up multiple Sidekiq nodes: + +1. Edit `/etc/gitlab/gitlab.rb`: ```ruby user['uid'] = 9000 @@ -85,180 +139,87 @@ you want using steps 1 and 2 from the GitLab downloads page. registry['gid'] = 9002 ``` -1. Disable other services: +1. Reconfigure GitLab: - ```ruby - nginx['enable'] = false - grafana['enable'] = false - prometheus['enable'] = false - gitlab_rails['auto_migrate'] = false - alertmanager['enable'] = false - gitaly['enable'] = false - gitlab_monitor['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = false - postgres_exporter['enable'] = false - postgresql['enable'] = false - redis['enable'] = false - redis_exporter['enable'] = false - puma['enable'] = false - gitlab_exporter['enable'] = false + ```shell + sudo gitlab-ctl reconfigure ``` -1. If you're using the Container Registry and it's running on a different node than Sidekiq, then - configure the registry URL: +## Configure the Container Registry when using an external Sidekiq + +If you're using the Container Registry and it's running on a different +node than Sidekiq, follow the steps below. + +1. Edit `/etc/gitlab/gitlab.rb`, and configure the registry URL: ```ruby registry_external_url 'https://registry.example.com' gitlab_rails['registry_api_url'] = "https://registry.example.com" ``` - - You must also copy the `registry.key` file to each Sidekiq node. -1. Define the `external_url`. To maintain uniformity of links across nodes, the - `external_url` on the Sidekiq server should point to the external URL that users - will use to access GitLab. This will either be the `external_url` set on your - application server or the URL of a external load balancer which will route traffic - to the GitLab application server: +1. Reconfigure GitLab: - ```ruby - external_url 'https://gitlab.example.com' + ```shell + sudo gitlab-ctl reconfigure ``` -1. (Optional) If you want to collect Sidekiq metrics, enable the Sidekiq metrics server. - To make metrics available from `localhost:8082/metrics`, set the following values: +1. In the instance where Container Registry is hosted, copy the `registry.key` + file to the Sidekiq node. + +## Configure the Sidekiq metrics server + +If you want to collect Sidekiq metrics, enable the Sidekiq metrics server. +To make metrics available from `localhost:8082/metrics`: + +To configure the metrics server: + +1. Edit `/etc/gitlab/gitlab.rb`: ```ruby sidekiq['metrics_enabled'] = true sidekiq['listen_address'] = "localhost" sidekiq['listen_port'] = "8082" - + # Optionally log all the metrics server logs to log/sidekiq_exporter.log sidekiq['exporter_log_enabled'] = true ``` -1. (Optional) If you use health check probes to observe Sidekiq, - set a separate port for health checks. - Configuring health checks is only necessary if there is something that actually probes them. - For more information about health checks, see the [Sidekiq health check page](sidekiq_health_check.md). - Enable health checks for Sidekiq: +1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +## Configure health checks + +If you use health check probes to observe Sidekiq, +you can set a separate port for health checks. +Configuring health checks is only necessary if there is something that actually probes them. +For more information about health checks, see the [Sidekiq health check page](sidekiq_health_check.md). + +To enable health checks for Sidekiq: + +1. Edit `/etc/gitlab/gitlab.rb`: ```ruby - sidekiq['health_checks_enabled'] = true - sidekiq['health_checks_listen_address'] = "localhost" - sidekiq['health_checks_listen_port'] = "8092" + sidekiq['health_checks_enabled'] = true + sidekiq['health_checks_listen_address'] = "localhost" + sidekiq['health_checks_listen_port'] = "8092" ``` NOTE: - If health check settings are not set, they will default to the metrics exporter settings. + If health check settings are not set, they default to the metrics exporter settings. This default is deprecated and is set to be removed in [GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/347509). -1. Run `gitlab-ctl reconfigure`. - -You will need to restart the Sidekiq nodes after an update has occurred and database -migrations performed. - -## Example configuration - -Here's what the ending `/etc/gitlab/gitlab.rb` would look like: - -```ruby -######################################## -##### Services Disabled ### -######################################## - -nginx['enable'] = false -grafana['enable'] = false -prometheus['enable'] = false -gitlab_rails['auto_migrate'] = false -alertmanager['enable'] = false -gitaly['enable'] = false -gitlab_workhorse['enable'] = false -nginx['enable'] = false -postgres_exporter['enable'] = false -postgresql['enable'] = false -redis['enable'] = false -redis_exporter['enable'] = false -puma['enable'] = false -gitlab_exporter['enable'] = false - -######################################## -#### Redis ### -######################################## - -## Must be the same in every sentinel node -redis['master_name'] = 'gitlab-redis' - -## The same password for Redis authentication you set up for the master node. -redis['master_password'] = 'YOUR_PASSOWORD' - -## A list of sentinels with `host` and `port` -gitlab_rails['redis_sentinels'] = [ - {'host' => '10.10.1.34', 'port' => 26379}, - {'host' => '10.10.1.35', 'port' => 26379}, - {'host' => '10.10.1.36', 'port' => 26379}, - ] - -####################################### -### Gitaly ### -####################################### - -git_data_dirs({ - 'default' => { 'gitaly_address' => 'tcp://gitaly:8075' }, -}) -gitlab_rails['gitaly_token'] = 'YOUR_TOKEN' - -####################################### -### Postgres ### -####################################### -gitlab_rails['db_host'] = '10.10.1.30' -gitlab_rails['db_password'] = 'YOUR_PASSOWORD' -gitlab_rails['db_port'] = '5432' -gitlab_rails['db_adapter'] = 'postgresql' -gitlab_rails['db_encoding'] = 'unicode' -gitlab_rails['auto_migrate'] = false - -####################################### -### Sidekiq configuration ### -####################################### -sidekiq['metrics_enabled'] = true -sidekiq['exporter_log_enabled'] = false -sidekiq['listen_port'] = "8082" - -sidekiq['health_checks_enabled'] = true -sidekiq['health_checks_listen_address'] = "localhost" -sidekiq['health_checks_listen_port'] = "8092" - -####################################### -### Monitoring configuration ### -####################################### -consul['enable'] = true -consul['monitoring_service_discovery'] = true - -consul['configuration'] = { - bind_addr: '10.10.1.48', - retry_join: %w(10.10.1.34 10.10.1.35 10.10.1.36) -} - -# Set the network addresses that the exporters will listen on -node_exporter['listen_address'] = '10.10.1.48:9100' - -# Rails Status for prometheus -gitlab_rails['monitoring_whitelist'] = ['10.10.1.42', '127.0.0.1'] - -# Container Registry URL for cleanup jobs -registry_external_url 'https://registry.example.com' -gitlab_rails['registry_api_url'] = "https://registry.example.com" - -# External URL (this should match the URL used to access your GitLab instance) -external_url 'https://gitlab.example.com' -``` - -## Further reading - -Related Sidekiq configuration: - -1. [Extra Sidekiq processes](operations/extra_sidekiq_processes.md) -1. [Extra Sidekiq routing](operations/extra_sidekiq_routing.md) -1. [Using the GitLab-Sidekiq chart](https://docs.gitlab.com/charts/charts/gitlab/sidekiq/) -1. [Sidekiq health checks](sidekiq_health_check.md) +1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +## Related topics + +- [Extra Sidekiq processes](operations/extra_sidekiq_processes.md) +- [Extra Sidekiq routing](operations/extra_sidekiq_routing.md) +- [Using the GitLab-Sidekiq chart](https://docs.gitlab.com/charts/charts/gitlab/sidekiq/) +- [Sidekiq health checks](sidekiq_health_check.md) diff --git a/doc/administration/troubleshooting/diagnostics_tools.md b/doc/administration/troubleshooting/diagnostics_tools.md index 53d4810b920..d510df5976c 100644 --- a/doc/administration/troubleshooting/diagnostics_tools.md +++ b/doc/administration/troubleshooting/diagnostics_tools.md @@ -23,3 +23,7 @@ running on. [strace-parser](https://gitlab.com/wchandler/strace-parser) is a small tool to analyze and summarize raw `strace` data. + +## kubesos + +The [`kubesos`](https://gitlab.com/gitlab-com/support/toolbox/kubesos/) utiltity retrieves GitLab cluster configuration and logs from GitLab Cloud Native chart deployments. diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 33a81c12811..265c5278fd6 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -78,6 +78,43 @@ Notify.test_email(e, "Test email for #{n}", 'Test email').deliver_now Notify.test_email(u.email, "Test email for #{u.name}", 'Test email').deliver_now ``` +## Limiting output + +Adding a semicolon(`;`) and a follow-up statement at the end of a statement prevents the default implicit return output. This is useful if you are already explicitly printing details and potentially have a lot of return output: + +```ruby +puts ActiveRecord::Base.descendants; :ok +Project.select(&:pages_deployed?).each {|p| puts p.pages_url }; true +``` + +## Get or store the result of last operation + +Underscore(`_`) represents the implicit return of the previous statement. You can use this to quickly assign a variable from the output of the previous command: + +```ruby +Project.last +# => #<Project id:2537 root/discard>> +project = _ +# => #<Project id:2537 root/discard>> +project.id +# => 2537 +``` + +## Open object in irb + +Sometimes it is easier to navigate through a method if you are within the context of the object. You can shim into the namespace of `Object` to let you open `irb` within the context of any object: + +```ruby +Object.define_method(:irb) { binding.irb } + +project = Project.last +# => #<Project id:2537 root/discard>> +project.irb +# Notice new context +irb(#<Project>)> web_url +# => "https://gitlab-example/root/discard" +``` + ## Query the database using an ActiveRecord Model ```ruby @@ -818,7 +855,7 @@ conflicting_permanent_redirects = RedirectRoute.matching_path_and_descendants(pa conflicting_permanent_redirects.destroy_all ``` -## Merge Requests +## Merge requests ### Close a merge request properly (if merged but still marked as open) @@ -1287,7 +1324,7 @@ has more information about Service Ping. ### Generate or get the cached Service Ping ```ruby -Gitlab::UsageData.to_json +Gitlab::Usage::ServicePingReport.for(output: :all_metrics_values, cached: true) ``` ### Generate a fresh new Service Ping @@ -1295,7 +1332,7 @@ Gitlab::UsageData.to_json This also refreshes the cached Service Ping displayed in the Admin Area ```ruby -Gitlab::UsageData.to_json(force_refresh: true) +Gitlab::Usage::ServicePingReport.for(output: :all_metrics_values) ``` ### Generate and print @@ -1332,7 +1369,7 @@ cluster = Clusters::Cluster.find_by(name: 'cluster_name') Delete cluster without associated resources: ```ruby -# Find users with the Administrator role +# Find users with the administrator access user = User.find_by(username: 'admin_user') # Find the cluster with the ID @@ -1377,3 +1414,18 @@ Gitlab::CurrentSettings.elasticsearch_url Gitlab::CurrentSettings.elasticsearch_indexing ``` + +#### Changing the Elasticsearch password + +```ruby +es_url = Gitlab::CurrentSettings.current_application_settings + +# Confirm the current ElasticSearch URL +es_url.elasticsearch_url + +# Set the ElasticSearch URL +es_url.elasticsearch_url = "http://<username>:<password>@your.es.host:<port>" + +# Save the change +es_url.save! +``` diff --git a/doc/administration/troubleshooting/group_saml_scim.md b/doc/administration/troubleshooting/group_saml_scim.md index 8ec6b35ec39..a30ade058f6 100644 --- a/doc/administration/troubleshooting/group_saml_scim.md +++ b/doc/administration/troubleshooting/group_saml_scim.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -40,7 +40,8 @@ User claims and attributes: SCIM mapping: - + + Group Sync: diff --git a/doc/administration/troubleshooting/img/AzureAD-scim_provisioning.png b/doc/administration/troubleshooting/img/AzureAD-scim_provisioning.png Binary files differnew file mode 100644 index 00000000000..b8edcfa31c2 --- /dev/null +++ b/doc/administration/troubleshooting/img/AzureAD-scim_provisioning.png diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md index e4d1696ea93..47fd424b1fd 100644 --- a/doc/administration/troubleshooting/postgresql.md +++ b/doc/administration/troubleshooting/postgresql.md @@ -98,6 +98,15 @@ This section is for links to information elsewhere in the GitLab documentation. - [Common Geo errors](../geo/replication/troubleshooting.md#fixing-common-errors). +- Mismatch in `pg_dump` and `psql` versions: + + ```plaintext + Dumping PostgreSQL database gitlabhq_production ... pg_dump: error: server version: 13.3; pg_dump version: 14.2 + pg_dump: error: aborting because of server version mismatch + ``` + + To fix this, see [Backup and restore a non-packaged PostgreSQL database](https://docs.gitlab.com/omnibus/settings/database.html#backup-and-restore-a-non-packaged-postgresql-database). + ## Support topics ### Database deadlocks diff --git a/doc/administration/user_settings.md b/doc/administration/user_settings.md index 2c1bb781882..2e879f8789d 100644 --- a/doc/administration/user_settings.md +++ b/doc/administration/user_settings.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/api/access_requests.md b/doc/api/access_requests.md index 9c217a98c3d..08f3b78787b 100644 --- a/doc/api/access_requests.md +++ b/doc/api/access_requests.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" --- diff --git a/doc/api/api_resources.md b/doc/api/api_resources.md index 783823f80fb..3d54402ea0b 100644 --- a/doc/api/api_resources.md +++ b/doc/api/api_resources.md @@ -14,7 +14,6 @@ Available resources for the [GitLab REST API](index.md) can be grouped in the fo See also: -- [V3 to V4](v3_to_v4.md). - Adding [deploy keys for multiple projects](deploy_keys.md#add-deploy-keys-to-multiple-projects). - [API Resources for various templates](#templates-api-resources). diff --git a/doc/api/appearance.md b/doc/api/appearance.md index 5e7ffbff25c..7b98b226cde 100644 --- a/doc/api/appearance.md +++ b/doc/api/appearance.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/api/applications.md b/doc/api/applications.md index bbf12438f28..aeb6a0452c5 100644 --- a/doc/api/applications.md +++ b/doc/api/applications.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/api/audit_events.md b/doc/api/audit_events.md index 888323f9362..4ddd851ebda 100644 --- a/doc/api/audit_events.md +++ b/doc/api/audit_events.md @@ -312,7 +312,7 @@ Example response: ### Retrieve a specific project audit event -Only available to users with at least the [Maintainer role](../user/permissions.md) for the project. +Only available to users with at least the Maintainer role for the project. ```plaintext GET /projects/:id/audit_events/:audit_event_id diff --git a/doc/api/avatar.md b/doc/api/avatar.md index 5db8c30cb9a..bc47d3a4d9e 100644 --- a/doc/api/avatar.md +++ b/doc/api/avatar.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/api/commits.md b/doc/api/commits.md index 6347af451a2..d7be9559527 100644 --- a/doc/api/commits.md +++ b/doc/api/commits.md @@ -8,6 +8,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w This API operates on [repository commits](https://git-scm.com/book/en/v2/Git-Basics-Recording-Changes-to-the-Repository). Read more about [GitLab-specific information](../user/project/repository/index.md#commit-changes-to-a-repository) for commits. +## Responses + +In commit responses, `created_at` and `committed_date` are identical. +However, `committed_date` and `authored_date` are generated from different sources, +and may not be identical. + ## List repository commits Get a list of repository commits in a project. @@ -740,9 +746,9 @@ Example response: } ``` -## List Merge Requests associated with a commit +## List merge requests associated with a commit -Get a list of Merge Requests related to the specified commit. +Get a list of merge requests related to the specified commit. ```plaintext GET /projects/:id/repository/commits/:sha/merge_requests diff --git a/doc/api/container_registry.md b/doc/api/container_registry.md index 68d339837b2..b61ec34b882 100644 --- a/doc/api/container_registry.md +++ b/doc/api/container_registry.md @@ -372,7 +372,8 @@ DELETE /projects/:id/registry/repositories/:repository_id/tags | `keep_n` | integer | no | The amount of latest tags of given name to keep. | | `older_than` | string | no | Tags to delete that are older than the given time, written in human readable form `1h`, `1d`, `1month`. | -This API call performs the following operations: +This API returns [HTTP response status code 202](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/202) +if successful, and performs the following operations: - It orders all tags by creation date. The creation date is the time of the manifest creation, not the time of tag push. @@ -396,10 +397,6 @@ If your Container Registry has a large number of tags to delete, only some of them will be deleted, and you might need to call this API multiple times. To schedule tags for automatic deletion, use a [cleanup policy](../user/packages/container_registry/reduce_container_registry_storage.md#cleanup-policy) instead. -NOTE: -In GitLab 12.4 and later, individual tags are deleted. -For more details, see the [discussion](https://gitlab.com/gitlab-org/gitlab/-/issues/15737). - Examples: 1. Remove tag names that are matching the regex (Git SHA), keep always at least 5, @@ -430,3 +427,29 @@ Examples: curl --request DELETE --data 'name_regex_delete=.*' --data 'older_than=1month' \ --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/registry/repositories/2/tags" ``` + +## Instance-wide endpoints + +Beside the group- and project-specific GitLab APIs explained above, +the Container Registry has its own endpoints. +To query those, follow the Registry's built-in mechanism to obtain and use an +[authentication token](https://docs.docker.com/registry/spec/auth/token/). + +NOTE: +These are different from project or personal access tokens in the GitLab application. + +### Listing all container repositories + +```plaintext +GET /v2/_catalog +``` + +To list all container repositories on your GitLab instance, admin credentials are required: + +```shell +$ curl --request GET --user "<admin-username>:<admin-password>" "https://gitlab.example.com/jwt/auth?service=container_registry&scope=registry:catalog:*" +{"token":" ... "} + +$ curl --header "Authorization: Bearer <token_from_above>" https://gitlab.example.com:5050/v2/_catalog +{"repositories":["user/project1", "group/subgroup/project2", ... ]} +``` diff --git a/doc/api/dependency_proxy.md b/doc/api/dependency_proxy.md index 5401c007c0d..028584e69ff 100644 --- a/doc/api/dependency_proxy.md +++ b/doc/api/dependency_proxy.md @@ -12,8 +12,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/273655) from GitLab Premium to GitLab Free in 13.6. Schedules for deletion the cached manifests and blobs for a group. This endpoint requires the -[Owner role](../user/permissions.md) -for the group. +Owner role for the group. ```plaintext DELETE /groups/:id/dependency_proxy/cache diff --git a/doc/api/deploy_keys.md b/doc/api/deploy_keys.md index b244384bd6a..ee6887e9d04 100644 --- a/doc/api/deploy_keys.md +++ b/doc/api/deploy_keys.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## List all deploy keys **(FREE SELF)** Get a list of all deploy keys across all projects of the GitLab instance. This -endpoint requires an administrator role and is not available on GitLab.com. +endpoint requires administrator access and is not available on GitLab.com. ```plaintext GET /deploy_keys diff --git a/doc/api/deploy_tokens.md b/doc/api/deploy_tokens.md index 0f2b9a13a3f..77c2fe5e162 100644 --- a/doc/api/deploy_tokens.md +++ b/doc/api/deploy_tokens.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/21811) in GitLab 12.9. -Get a list of all deploy tokens across the GitLab instance. This endpoint requires the Administrator role. +Get a list of all deploy tokens across the GitLab instance. This endpoint requires administrator access. ```plaintext GET /deploy_tokens @@ -49,7 +49,7 @@ Example response: ## Project deploy tokens -Project deploy token API endpoints require the [Maintainer role](../user/permissions.md) or higher +Project deploy token API endpoints require the Maintainer role or higher for the project. ### List project deploy tokens @@ -165,7 +165,7 @@ curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" \ ## Group deploy tokens -Users with at least the [Maintainer role](../user/permissions.md) for the group can list group deploy +Users with at least the Maintainer role for the group can list group deploy tokens. Only group Owners can create and delete group deploy tokens. ### List group deploy tokens diff --git a/doc/api/deployments.md b/doc/api/deployments.md index 70b3e76c3dd..29f9bd3e2ee 100644 --- a/doc/api/deployments.md +++ b/doc/api/deployments.md @@ -23,7 +23,7 @@ GET /projects/:id/deployments | `updated_after` | datetime | no | Return deployments updated after the specified date. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`). | | `updated_before` | datetime | no | Return deployments updated before the specified date. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`). | | `environment` | string | no | The [name of the environment](../ci/environments/index.md) to filter deployments by. | -| `status` | string | no | The status to filter deployments by. One of `created`, `running`, `success`, `failed`, `canceled`. +| `status` | string | no | The status to filter deployments by. One of `created`, `running`, `success`, `failed`, `canceled`, `blocked`. ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/deployments" @@ -201,6 +201,7 @@ Example response: "sha": "a91957a858320c0e17f3a0eca7cfacbff50ea29a", "created_at": "2016-08-11T11:32:35.444Z", "updated_at": "2016-08-11T11:34:01.123Z", + "status": "success", "user": { "name": "Administrator", "username": "root", @@ -264,6 +265,29 @@ Example response: } ``` +Deployments created by users on GitLab Premium or higher include the `approvals` and `pending_approval_count` properties: + +```json +{ + "status": "created", + "pending_approval_count": 0, + "approvals": [ + { + "user": { + "id": 49, + "username": "project_6_bot", + "name": "****", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/e83ac685f68ea07553ad3054c738c709?s=80&d=identicon", + "web_url": "http://localhost:3000/project_6_bot" + }, + "status": "approved" + } + ], + ... +} +``` + ## Create a deployment ```plaintext @@ -311,6 +335,29 @@ Example response: } ``` +Deployments created by users on GitLab Premium or higher include the `approvals` and `pending_approval_count` properties: + +```json +{ + "status": "created", + "pending_approval_count": 0, + "approvals": [ + { + "user": { + "id": 49, + "username": "project_6_bot", + "name": "****", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/e83ac685f68ea07553ad3054c738c709?s=80&d=identicon", + "web_url": "http://localhost:3000/project_6_bot" + }, + "status": "approved" + } + ], + ... +} +``` + ## Update a deployment ```plaintext @@ -354,6 +401,29 @@ Example response: } ``` +Deployments created by users on GitLab Premium or higher include the `approvals` and `pending_approval_count` properties: + +```json +{ + "status": "created", + "pending_approval_count": 0, + "approvals": [ + { + "user": { + "id": 49, + "username": "project_6_bot", + "name": "****", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/e83ac685f68ea07553ad3054c738c709?s=80&d=identicon", + "web_url": "http://localhost:3000/project_6_bot" + }, + "status": "approved" + } + ], + ... +} +``` + ## List of merge requests associated with a deployment > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35739) in GitLab 12.7. @@ -378,7 +448,10 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/a ## Approve or reject a blocked deployment **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/343864) in GitLab 14.7 [with a flag](../administration/feature_flags.md) named `deployment_approvals`. Disabled by default. This feature is not ready for production use. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/343864) in GitLab 14.7 [with a flag](../administration/feature_flags.md) named `deployment_approvals`. Disabled by default. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/347342) in GitLab 14.8. + +See [Deployment Approvals](../ci/environments/deployment_approvals.md) for more information about this feature. ```plaintext POST /projects/:id/deployments/:deployment_id/approval diff --git a/doc/api/dora/metrics.md b/doc/api/dora/metrics.md index 3e7b7800034..3f078945b0f 100644 --- a/doc/api/dora/metrics.md +++ b/doc/api/dora/metrics.md @@ -10,7 +10,7 @@ type: reference, api > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/279039) in GitLab 13.10. > - The legacy key/value pair `{ "<date>" => "<value>" }` was removed from the payload in GitLab 14.0. -All methods require at least the Reporter [role](../../user/permissions.md). +All methods require at least the Reporter role. ## Get project-level DORA metrics diff --git a/doc/api/epics.md b/doc/api/epics.md index f3137559220..deb74cf21e9 100644 --- a/doc/api/epics.md +++ b/doc/api/epics.md @@ -40,12 +40,13 @@ are paginated. Read more on [pagination](index.md#pagination). WARNING: -> `reference` attribute in response is deprecated in favour of `references`. -> Introduced in [GitLab 12.6](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20354) +In [GitLab 12.6](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20354) and later, +the `reference` attribute in responses is deprecated in favor of `references`. NOTE: -> `references.relative` is relative to the group that the epic is being requested. When epic is fetched from its origin group -> `relative` format would be the same as `short` format and when requested cross groups it is expected to be the same as `full` format. +`references.relative` is relative to the group that the epic is being requested from. When an epic +is fetched from its origin group, the `relative` format is the same as the `short` format. +When an epic is requested across groups, the `relative` format is expected to be the same as the `full` format. ## List epics for a group diff --git a/doc/api/error_tracking.md b/doc/api/error_tracking.md index c62d33f82f4..dbf832b301f 100644 --- a/doc/api/error_tracking.md +++ b/doc/api/error_tracking.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -11,7 +11,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Error Tracking project settings The project settings API allows you to retrieve the [Error Tracking](../operations/error_tracking.md) -settings for a project. Only for users with [Maintainer role](../user/permissions.md) for the project. +settings for a project. Only for users with Maintainer role for the project. ### Get Error Tracking settings @@ -42,7 +42,7 @@ Example response: ### Enable or disable the Error Tracking project settings The API allows you to enable or disable the Error Tracking settings for a project. Only for users with the -[Maintainer role](../user/permissions.md) for the project. +Maintainer role for the project. ```plaintext PATCH /projects/:id/error_tracking/settings @@ -75,7 +75,7 @@ Example response: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68384) in GitLab 14.3. For [integrated error tracking](https://gitlab.com/gitlab-org/gitlab/-/issues/329596) feature. Only for users with the -[Maintainer role](../user/permissions.md) for the project. +Maintainer role for the project. ### List project client keys diff --git a/doc/api/geo_nodes.md b/doc/api/geo_nodes.md index fe1674649b6..a152c443902 100644 --- a/doc/api/geo_nodes.md +++ b/doc/api/geo_nodes.md @@ -467,6 +467,19 @@ Example response: "uploads_verified_count": null, "uploads_verification_failed_count": null, "uploads_verified_in_percentage": "0.00%", + "job_artifacts_count": 5, + "job_artifacts_checksum_total_count": 5, + "job_artifacts_checksummed_count": 5, + "job_artifacts_checksum_failed_count": 0, + "job_artifacts_synced_count": 5, + "job_artifacts_failed_count": 0, + "job_artifacts_registry_count": 5, + "job_artifacts_verification_total_count": 5, + "job_artifacts_verified_count": 5, + "job_artifacts_verification_failed_count": 0, + "job_artifacts_synced_in_percentage": "100.00%", + "job_artifacts_verified_in_percentage": "100.00%", + "job_artifacts_synced_missing_on_primary_count": 0, }, { "geo_node_id": 2, @@ -623,6 +636,19 @@ Example response: "uploads_verified_count": null, "uploads_verification_failed_count": null, "uploads_verified_in_percentage": "0.00%", + "job_artifacts_count": 5, + "job_artifacts_checksum_total_count": 5, + "job_artifacts_checksummed_count": 5, + "job_artifacts_checksum_failed_count": 0, + "job_artifacts_synced_count": 5, + "job_artifacts_failed_count": 0, + "job_artifacts_registry_count": 5, + "job_artifacts_verification_total_count": 5, + "job_artifacts_verified_count": 5, + "job_artifacts_verification_failed_count": 0, + "job_artifacts_synced_in_percentage": "100.00%", + "job_artifacts_verified_in_percentage": "100.00%", + "job_artifacts_synced_missing_on_primary_count": 0, } ] ``` @@ -776,6 +802,19 @@ Example response: "uploads_verified_count": null, "uploads_verification_failed_count": null, "uploads_verified_in_percentage": "0.00%", + "job_artifacts_count": 5, + "job_artifacts_checksum_total_count": 5, + "job_artifacts_checksummed_count": 5, + "job_artifacts_checksum_failed_count": 0, + "job_artifacts_synced_count": 5, + "job_artifacts_failed_count": 0, + "job_artifacts_registry_count": 5, + "job_artifacts_verification_total_count": 5, + "job_artifacts_verified_count": 5, + "job_artifacts_verification_failed_count": 0, + "job_artifacts_synced_in_percentage": "100.00%", + "job_artifacts_verified_in_percentage": "100.00%", + "job_artifacts_synced_missing_on_primary_count": 0, } ``` diff --git a/doc/api/graphql/index.md b/doc/api/graphql/index.md index 14512286ade..d0e65b8c4eb 100644 --- a/doc/api/graphql/index.md +++ b/doc/api/graphql/index.md @@ -152,7 +152,7 @@ Root-level queries are defined in ### Multiplex queries GitLab supports batching queries into a single request using -[apollo-link-batch-http](https://www.apollographql.com/docs/link/links/batch-http/). More +[`@apollo/client/link/batch-http`](https://www.apollographql.com/docs/react/api/link/apollo-link-batch-http/). More information about multiplexed queries is also available for [GraphQL Ruby](https://graphql-ruby.org/queries/multiplex.html), the library GitLab uses on the backend. diff --git a/doc/api/graphql/reference/index.md b/doc/api/graphql/reference/index.md index 4ca40d1fa11..18c99b7d151 100644 --- a/doc/api/graphql/reference/index.md +++ b/doc/api/graphql/reference/index.md @@ -376,7 +376,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="queryrunnersactive"></a>`active` | [`Boolean`](#boolean) | Filter runners by active (true) or paused (false) status. | +| <a id="queryrunnersactive"></a>`active` **{warning-solid}** | [`Boolean`](#boolean) | **Deprecated** in 14.8. This was renamed. Use: `paused`. | +| <a id="queryrunnerspaused"></a>`paused` | [`Boolean`](#boolean) | Filter runners by `paused` (true) or `active` (false) status. | | <a id="queryrunnerssearch"></a>`search` | [`String`](#string) | Filter by full token or partial text in description field. | | <a id="queryrunnerssort"></a>`sort` | [`CiRunnerSort`](#cirunnersort) | Sort order of results. | | <a id="queryrunnersstatus"></a>`status` | [`CiRunnerStatus`](#cirunnerstatus) | Filter runners by status. | @@ -1017,6 +1018,30 @@ Input type: `CommitCreateInput` | <a id="mutationcommitcreatecontent"></a>`content` | [`[String!]`](#string) | Contents of the commit. | | <a id="mutationcommitcreateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +### `Mutation.configureContainerScanning` + +Configure Container Scanning for a project by enabling Container Scanning in a new or modified +`.gitlab-ci.yml` file in a new branch. The new branch and a URL to +create a merge request are part of the response. + +Input type: `ConfigureContainerScanningInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationconfigurecontainerscanningclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationconfigurecontainerscanningprojectpath"></a>`projectPath` | [`ID!`](#id) | Full path of the project. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationconfigurecontainerscanningbranch"></a>`branch` | [`String`](#string) | Branch that has the new/modified `.gitlab-ci.yml` file. | +| <a id="mutationconfigurecontainerscanningclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationconfigurecontainerscanningerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationconfigurecontainerscanningsuccesspath"></a>`successPath` | [`String`](#string) | Redirect path to use when the response is successful. | + ### `Mutation.configureDependencyScanning` Configure Dependency Scanning for a project by enabling Dependency Scanning in a new or modified @@ -1118,7 +1143,7 @@ Input type: `ConfigureSecretDetectionInput` ### `Mutation.corpusCreate` -Available only when feature flag `corpus_management` is enabled. This flag is disabled by default, because the feature is experimental and is subject to change without notice. +Available only when feature flag `corpus_management` is enabled. This flag is enabled by default. Input type: `CorpusCreateInput` @@ -1395,6 +1420,8 @@ Input type: `CreateIssueInput` | <a id="mutationcreateissuelocked"></a>`locked` | [`Boolean`](#boolean) | Indicates discussion is locked on the issue. | | <a id="mutationcreateissuemergerequesttoresolvediscussionsof"></a>`mergeRequestToResolveDiscussionsOf` | [`MergeRequestID`](#mergerequestid) | IID of a merge request for which to resolve discussions. | | <a id="mutationcreateissuemilestoneid"></a>`milestoneId` | [`MilestoneID`](#milestoneid) | ID of the milestone to assign to the issue. On update milestone will be removed if set to null. | +| <a id="mutationcreateissuemoveafterid"></a>`moveAfterId` | [`IssueID`](#issueid) | Global ID of issue that should be placed after the current issue. | +| <a id="mutationcreateissuemovebeforeid"></a>`moveBeforeId` | [`IssueID`](#issueid) | Global ID of issue that should be placed before the current issue. | | <a id="mutationcreateissueprojectpath"></a>`projectPath` | [`ID!`](#id) | Project full path the issue is associated with. | | <a id="mutationcreateissuetitle"></a>`title` | [`String!`](#string) | Title of the issue. | | <a id="mutationcreateissuetype"></a>`type` | [`IssueType`](#issuetype) | Type of the issue. | @@ -1489,11 +1516,9 @@ Input type: `CreateSnippetInput` | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mutationcreatesnippetblobactions"></a>`blobActions` | [`[SnippetBlobActionInputType!]`](#snippetblobactioninputtype) | Actions to perform over the snippet repository and blobs. | -| <a id="mutationcreatesnippetcaptcharesponse"></a>`captchaResponse` **{warning-solid}** | [`String`](#string) | **Deprecated:** Use spam protection with HTTP headers instead. Deprecated in 13.11. | | <a id="mutationcreatesnippetclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationcreatesnippetdescription"></a>`description` | [`String`](#string) | Description of the snippet. | | <a id="mutationcreatesnippetprojectpath"></a>`projectPath` | [`ID`](#id) | Full path of the project the snippet is associated with. | -| <a id="mutationcreatesnippetspamlogid"></a>`spamLogId` **{warning-solid}** | [`Int`](#int) | **Deprecated:** Use spam protection with HTTP headers instead. Deprecated in 13.11. | | <a id="mutationcreatesnippettitle"></a>`title` | [`String!`](#string) | Title of the snippet. | | <a id="mutationcreatesnippetuploadedfiles"></a>`uploadedFiles` | [`[String!]`](#string) | Paths to files uploaded in the snippet description. | | <a id="mutationcreatesnippetvisibilitylevel"></a>`visibilityLevel` | [`VisibilityLevelsEnum!`](#visibilitylevelsenum) | Visibility level of the snippet. | @@ -1502,13 +1527,9 @@ Input type: `CreateSnippetInput` | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="mutationcreatesnippetcaptchasitekey"></a>`captchaSiteKey` **{warning-solid}** | [`String`](#string) | **Deprecated:** Use spam protection with HTTP headers instead. Deprecated in 13.11. | | <a id="mutationcreatesnippetclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationcreatesnippeterrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | -| <a id="mutationcreatesnippetneedscaptcharesponse"></a>`needsCaptchaResponse` **{warning-solid}** | [`Boolean`](#boolean) | **Deprecated:** Use spam protection with HTTP headers instead. Deprecated in 13.11. | | <a id="mutationcreatesnippetsnippet"></a>`snippet` | [`Snippet`](#snippet) | Snippet after mutation. | -| <a id="mutationcreatesnippetspam"></a>`spam` **{warning-solid}** | [`Boolean`](#boolean) | **Deprecated:** Use spam protection with HTTP headers instead. Deprecated in 13.11. | -| <a id="mutationcreatesnippetspamlogid"></a>`spamLogId` **{warning-solid}** | [`Int`](#int) | **Deprecated:** Use spam protection with HTTP headers instead. Deprecated in 13.11. | ### `Mutation.createTestCase` @@ -4157,12 +4178,13 @@ Input type: `RunnerUpdateInput` | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mutationrunnerupdateaccesslevel"></a>`accessLevel` | [`CiRunnerAccessLevel`](#cirunneraccesslevel) | Access level of the runner. | -| <a id="mutationrunnerupdateactive"></a>`active` | [`Boolean`](#boolean) | Indicates the runner is allowed to receive jobs. | +| <a id="mutationrunnerupdateactive"></a>`active` **{warning-solid}** | [`Boolean`](#boolean) | **Deprecated:** This was renamed. Please use `paused`. Deprecated in 14.8. | | <a id="mutationrunnerupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationrunnerupdatedescription"></a>`description` | [`String`](#string) | Description of the runner. | | <a id="mutationrunnerupdateid"></a>`id` | [`CiRunnerID!`](#cirunnerid) | ID of the runner to update. | | <a id="mutationrunnerupdatelocked"></a>`locked` | [`Boolean`](#boolean) | Indicates the runner is locked. | | <a id="mutationrunnerupdatemaximumtimeout"></a>`maximumTimeout` | [`Int`](#int) | Maximum timeout (in seconds) for jobs processed by the runner. | +| <a id="mutationrunnerupdatepaused"></a>`paused` | [`Boolean`](#boolean) | Indicates the runner is not allowed to receive jobs. | | <a id="mutationrunnerupdateprivateprojectsminutescostfactor"></a>`privateProjectsMinutesCostFactor` | [`Float`](#float) | Private projects' "minutes cost factor" associated with the runner (GitLab.com only). | | <a id="mutationrunnerupdatepublicprojectsminutescostfactor"></a>`publicProjectsMinutesCostFactor` | [`Float`](#float) | Public projects' "minutes cost factor" associated with the runner (GitLab.com only). | | <a id="mutationrunnerupdaterununtagged"></a>`runUntagged` | [`Boolean`](#boolean) | Indicates the runner is able to run untagged jobs. | @@ -4282,6 +4304,28 @@ Input type: `SecurityPolicyProjectUnassignInput` | <a id="mutationsecuritypolicyprojectunassignclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationsecuritypolicyprojectunassignerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +### `Mutation.securityTrainingUpdate` + +Input type: `SecurityTrainingUpdateInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationsecuritytrainingupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationsecuritytrainingupdateisenabled"></a>`isEnabled` | [`Boolean!`](#boolean) | Sets the training provider as enabled for the project. | +| <a id="mutationsecuritytrainingupdateisprimary"></a>`isPrimary` | [`Boolean`](#boolean) | Sets the training provider as primary for the project. | +| <a id="mutationsecuritytrainingupdateprojectpath"></a>`projectPath` | [`ID!`](#id) | Full path of the project. | +| <a id="mutationsecuritytrainingupdateproviderid"></a>`providerId` | [`SecurityTrainingProviderID!`](#securitytrainingproviderid) | ID of the provider. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationsecuritytrainingupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationsecuritytrainingupdateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationsecuritytrainingupdatetraining"></a>`training` | [`ProjectSecurityTraining`](#projectsecuritytraining) | Represents the training entity subject to mutation. | + ### `Mutation.terraformStateDelete` Input type: `TerraformStateDeleteInput` @@ -4336,6 +4380,27 @@ Input type: `TerraformStateUnlockInput` | <a id="mutationterraformstateunlockclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationterraformstateunlockerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +### `Mutation.timelineEventCreate` + +Input type: `TimelineEventCreateInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationtimelineeventcreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationtimelineeventcreateincidentid"></a>`incidentId` | [`IssueID!`](#issueid) | Incident ID of the timeline event. | +| <a id="mutationtimelineeventcreatenote"></a>`note` | [`String!`](#string) | Text note of the timeline event. | +| <a id="mutationtimelineeventcreateoccurredat"></a>`occurredAt` | [`Time!`](#time) | Timestamp of when the event occurred. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationtimelineeventcreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationtimelineeventcreateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationtimelineeventcreatetimelineevent"></a>`timelineEvent` | [`TimelineEventType`](#timelineeventtype) | Timeline event. | + ### `Mutation.timelineEventDestroy` Input type: `TimelineEventDestroyInput` @@ -4355,6 +4420,27 @@ Input type: `TimelineEventDestroyInput` | <a id="mutationtimelineeventdestroyerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | <a id="mutationtimelineeventdestroytimelineevent"></a>`timelineEvent` | [`TimelineEventType`](#timelineeventtype) | Timeline event. | +### `Mutation.timelineEventUpdate` + +Input type: `TimelineEventUpdateInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationtimelineeventupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationtimelineeventupdateid"></a>`id` | [`IncidentManagementTimelineEventID!`](#incidentmanagementtimelineeventid) | ID of the timeline event to update. | +| <a id="mutationtimelineeventupdatenote"></a>`note` | [`String`](#string) | Text note of the timeline event. | +| <a id="mutationtimelineeventupdateoccurredat"></a>`occurredAt` | [`Time`](#time) | Timestamp when the event occurred. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationtimelineeventupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationtimelineeventupdateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationtimelineeventupdatetimelineevent"></a>`timelineEvent` | [`TimelineEventType`](#timelineeventtype) | Timeline event. | + ### `Mutation.todoCreate` Input type: `TodoCreateInput` @@ -4846,11 +4932,9 @@ Input type: `UpdateSnippetInput` | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mutationupdatesnippetblobactions"></a>`blobActions` | [`[SnippetBlobActionInputType!]`](#snippetblobactioninputtype) | Actions to perform over the snippet repository and blobs. | -| <a id="mutationupdatesnippetcaptcharesponse"></a>`captchaResponse` **{warning-solid}** | [`String`](#string) | **Deprecated:** Use spam protection with HTTP headers instead. Deprecated in 13.11. | | <a id="mutationupdatesnippetclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationupdatesnippetdescription"></a>`description` | [`String`](#string) | Description of the snippet. | | <a id="mutationupdatesnippetid"></a>`id` | [`SnippetID!`](#snippetid) | Global ID of the snippet to update. | -| <a id="mutationupdatesnippetspamlogid"></a>`spamLogId` **{warning-solid}** | [`Int`](#int) | **Deprecated:** Use spam protection with HTTP headers instead. Deprecated in 13.11. | | <a id="mutationupdatesnippettitle"></a>`title` | [`String`](#string) | Title of the snippet. | | <a id="mutationupdatesnippetvisibilitylevel"></a>`visibilityLevel` | [`VisibilityLevelsEnum`](#visibilitylevelsenum) | Visibility level of the snippet. | @@ -4858,13 +4942,9 @@ Input type: `UpdateSnippetInput` | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="mutationupdatesnippetcaptchasitekey"></a>`captchaSiteKey` **{warning-solid}** | [`String`](#string) | **Deprecated:** Use spam protection with HTTP headers instead. Deprecated in 13.11. | | <a id="mutationupdatesnippetclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationupdatesnippeterrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | -| <a id="mutationupdatesnippetneedscaptcharesponse"></a>`needsCaptchaResponse` **{warning-solid}** | [`Boolean`](#boolean) | **Deprecated:** Use spam protection with HTTP headers instead. Deprecated in 13.11. | | <a id="mutationupdatesnippetsnippet"></a>`snippet` | [`Snippet`](#snippet) | Snippet after mutation. | -| <a id="mutationupdatesnippetspam"></a>`spam` **{warning-solid}** | [`Boolean`](#boolean) | **Deprecated:** Use spam protection with HTTP headers instead. Deprecated in 13.11. | -| <a id="mutationupdatesnippetspamlogid"></a>`spamLogId` **{warning-solid}** | [`Int`](#int) | **Deprecated:** Use spam protection with HTTP headers instead. Deprecated in 13.11. | ### `Mutation.userCalloutCreate` @@ -4885,6 +4965,25 @@ Input type: `UserCalloutCreateInput` | <a id="mutationusercalloutcreateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | <a id="mutationusercalloutcreateusercallout"></a>`userCallout` | [`UserCallout!`](#usercallout) | User callout dismissed. | +### `Mutation.userPreferencesUpdate` + +Input type: `UserPreferencesUpdateInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationuserpreferencesupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationuserpreferencesupdateissuessort"></a>`issuesSort` | [`IssueSort`](#issuesort) | Sort order for issue lists. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationuserpreferencesupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationuserpreferencesupdateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationuserpreferencesupdateuserpreferences"></a>`userPreferences` | [`UserPreferences`](#userpreferences) | User preferences after mutation. | + ### `Mutation.vulnerabilityConfirm` Input type: `VulnerabilityConfirmInput` @@ -5057,7 +5156,7 @@ Input type: `VulnerabilityRevertToDetectedInput` ### `Mutation.workItemCreate` -Available only when feature flag `work_items` is enabled. This flag is disabled by default, because the feature is experimental and is subject to change without notice. +Creates a work item. Available only when feature flag `work_items` is enabled. The feature is experimental and is subject to change without notice. Input type: `WorkItemCreateInput` @@ -5079,6 +5178,50 @@ Input type: `WorkItemCreateInput` | <a id="mutationworkitemcreateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | <a id="mutationworkitemcreateworkitem"></a>`workItem` | [`WorkItem`](#workitem) | Created work item. | +### `Mutation.workItemDelete` + +Deletes a work item. Available only when feature flag `work_items` is enabled. The feature is experimental and is subject to change without notice. + +Input type: `WorkItemDeleteInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationworkitemdeleteclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationworkitemdeleteid"></a>`id` | [`WorkItemID!`](#workitemid) | Global ID of the work item. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationworkitemdeleteclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationworkitemdeleteerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationworkitemdeleteproject"></a>`project` | [`Project`](#project) | Project the deleted work item belonged to. | + +### `Mutation.workItemUpdate` + +Updates a work item by Global ID. Available only when feature flag `work_items` is enabled. The feature is experimental and is subject to change without notice. + +Input type: `WorkItemUpdateInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationworkitemupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationworkitemupdateid"></a>`id` | [`WorkItemID!`](#workitemid) | Global ID of the work item. | +| <a id="mutationworkitemupdatestateevent"></a>`stateEvent` | [`WorkItemStateEvent`](#workitemstateevent) | Close or reopen a work item. | +| <a id="mutationworkitemupdatetitle"></a>`title` | [`String`](#string) | Title of the work item. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationworkitemupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationworkitemupdateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationworkitemupdateworkitem"></a>`workItem` | [`WorkItem`](#workitem) | Updated work item. | + ## Connections Some types in our schema are `Connection` types - they represent a paginated @@ -6648,6 +6791,29 @@ The edge type for [`JiraProject`](#jiraproject). | <a id="jiraprojectedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | | <a id="jiraprojectedgenode"></a>`node` | [`JiraProject`](#jiraproject) | The item at the end of the edge. | +#### `JobArtifactRegistryConnection` + +The connection type for [`JobArtifactRegistry`](#jobartifactregistry). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="jobartifactregistryconnectionedges"></a>`edges` | [`[JobArtifactRegistryEdge]`](#jobartifactregistryedge) | A list of edges. | +| <a id="jobartifactregistryconnectionnodes"></a>`nodes` | [`[JobArtifactRegistry]`](#jobartifactregistry) | A list of nodes. | +| <a id="jobartifactregistryconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +#### `JobArtifactRegistryEdge` + +The edge type for [`JobArtifactRegistry`](#jobartifactregistry). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="jobartifactregistryedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | +| <a id="jobartifactregistryedgenode"></a>`node` | [`JobArtifactRegistry`](#jobartifactregistry) | The item at the end of the edge. | + #### `JobNeedUnionConnection` The connection type for [`JobNeedUnion`](#jobneedunion). @@ -8490,6 +8656,18 @@ Describes a rule for who can approve merge requests. | <a id="approvalruletype"></a>`type` | [`ApprovalRuleType`](#approvalruletype) | Type of the rule. | | <a id="approvalruleusers"></a>`users` | [`UserCoreConnection`](#usercoreconnection) | List of users added as approvers for the rule. (see [Connections](#connections)) | +### `AssetType` + +Represents a vulnerability asset type. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="assettypename"></a>`name` | [`String!`](#string) | Name of the asset. | +| <a id="assettypetype"></a>`type` | [`String!`](#string) | Type of the asset. | +| <a id="assettypeurl"></a>`url` | [`String!`](#string) | URL of the asset. | + ### `AwardEmoji` An emoji awarded by a user. @@ -8680,6 +8858,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | ---- | ---- | ----------- | | <a id="boardepicancestorsauthorusername"></a>`authorUsername` | [`String`](#string) | Filter epics by author. | | <a id="boardepicancestorsconfidential"></a>`confidential` | [`Boolean`](#boolean) | Filter epics by given confidentiality. | +| <a id="boardepicancestorscreatedafter"></a>`createdAfter` | [`Time`](#time) | Epics created after this date. | +| <a id="boardepicancestorscreatedbefore"></a>`createdBefore` | [`Time`](#time) | Epics created before this date. | | <a id="boardepicancestorsenddate"></a>`endDate` **{warning-solid}** | [`Time`](#time) | **Deprecated** in 13.5. Use timeframe.end. | | <a id="boardepicancestorsiid"></a>`iid` | [`ID`](#id) | IID of the epic, e.g., "1". | | <a id="boardepicancestorsiidstartswith"></a>`iidStartsWith` | [`String`](#string) | Filter epics by IID for autocomplete. | @@ -8696,6 +8876,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="boardepicancestorsstartdate"></a>`startDate` **{warning-solid}** | [`Time`](#time) | **Deprecated** in 13.5. Use timeframe.start. | | <a id="boardepicancestorsstate"></a>`state` | [`EpicState`](#epicstate) | Filter epics by state. | | <a id="boardepicancestorstimeframe"></a>`timeframe` | [`Timeframe`](#timeframe) | List items overlapping the given timeframe. | +| <a id="boardepicancestorsupdatedafter"></a>`updatedAfter` | [`Time`](#time) | Epics updated after this date. | +| <a id="boardepicancestorsupdatedbefore"></a>`updatedBefore` | [`Time`](#time) | Epics updated before this date. | ##### `BoardEpic.children` @@ -8713,6 +8895,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | ---- | ---- | ----------- | | <a id="boardepicchildrenauthorusername"></a>`authorUsername` | [`String`](#string) | Filter epics by author. | | <a id="boardepicchildrenconfidential"></a>`confidential` | [`Boolean`](#boolean) | Filter epics by given confidentiality. | +| <a id="boardepicchildrencreatedafter"></a>`createdAfter` | [`Time`](#time) | Epics created after this date. | +| <a id="boardepicchildrencreatedbefore"></a>`createdBefore` | [`Time`](#time) | Epics created before this date. | | <a id="boardepicchildrenenddate"></a>`endDate` **{warning-solid}** | [`Time`](#time) | **Deprecated** in 13.5. Use timeframe.end. | | <a id="boardepicchildreniid"></a>`iid` | [`ID`](#id) | IID of the epic, e.g., "1". | | <a id="boardepicchildreniidstartswith"></a>`iidStartsWith` | [`String`](#string) | Filter epics by IID for autocomplete. | @@ -8729,6 +8913,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="boardepicchildrenstartdate"></a>`startDate` **{warning-solid}** | [`Time`](#time) | **Deprecated** in 13.5. Use timeframe.start. | | <a id="boardepicchildrenstate"></a>`state` | [`EpicState`](#epicstate) | Filter epics by state. | | <a id="boardepicchildrentimeframe"></a>`timeframe` | [`Timeframe`](#timeframe) | List items overlapping the given timeframe. | +| <a id="boardepicchildrenupdatedafter"></a>`updatedAfter` | [`Time`](#time) | Epics updated after this date. | +| <a id="boardepicchildrenupdatedbefore"></a>`updatedBefore` | [`Time`](#time) | Epics updated before this date. | ##### `BoardEpic.currentUserTodos` @@ -9015,31 +9201,51 @@ Represents the total number of issues and their weights for a particular day. | Name | Type | Description | | ---- | ---- | ----------- | | <a id="cirunneraccesslevel"></a>`accessLevel` | [`CiRunnerAccessLevel!`](#cirunneraccesslevel) | Access level of the runner. | -| <a id="cirunneractive"></a>`active` | [`Boolean!`](#boolean) | Indicates the runner is allowed to receive jobs. | +| <a id="cirunneractive"></a>`active` **{warning-solid}** | [`Boolean!`](#boolean) | **Deprecated** in 14.8. Use paused. | | <a id="cirunneradminurl"></a>`adminUrl` | [`String`](#string) | Admin URL of the runner. Only available for administrators. | | <a id="cirunnercontactedat"></a>`contactedAt` | [`Time`](#time) | Timestamp of last contact from this runner. | | <a id="cirunnercreatedat"></a>`createdAt` | [`Time`](#time) | Timestamp of creation of this runner. | | <a id="cirunnerdescription"></a>`description` | [`String`](#string) | Description of the runner. | | <a id="cirunnereditadminurl"></a>`editAdminUrl` | [`String`](#string) | Admin form URL of the runner. Only available for administrators. | | <a id="cirunnerexecutorname"></a>`executorName` | [`String`](#string) | Executor last advertised by the runner. Available only when feature flag `graphql_ci_runner_executor` is enabled. This flag is disabled by default, because the feature is experimental and is subject to change without notice. | +| <a id="cirunnergroups"></a>`groups` | [`GroupConnection`](#groupconnection) | Groups the runner is associated with. For group runners only. (see [Connections](#connections)) | | <a id="cirunnerid"></a>`id` | [`CiRunnerID!`](#cirunnerid) | ID of the runner. | | <a id="cirunneripaddress"></a>`ipAddress` | [`String`](#string) | IP address of the runner. | | <a id="cirunnerjobcount"></a>`jobCount` | [`Int`](#int) | Number of jobs processed by the runner (limited to 1000, plus one to indicate that more items exist). | | <a id="cirunnerlocked"></a>`locked` | [`Boolean`](#boolean) | Indicates the runner is locked. | | <a id="cirunnermaximumtimeout"></a>`maximumTimeout` | [`Int`](#int) | Maximum timeout (in seconds) for jobs processed by the runner. | +| <a id="cirunnerpaused"></a>`paused` | [`Boolean!`](#boolean) | Indicates the runner is paused and not available to run jobs. | | <a id="cirunnerprivateprojectsminutescostfactor"></a>`privateProjectsMinutesCostFactor` | [`Float`](#float) | Private projects' "minutes cost factor" associated with the runner (GitLab.com only). | | <a id="cirunnerprojectcount"></a>`projectCount` | [`Int`](#int) | Number of projects that the runner is associated with. | +| <a id="cirunnerprojects"></a>`projects` | [`ProjectConnection`](#projectconnection) | Projects the runner is associated with. For project runners only. (see [Connections](#connections)) | | <a id="cirunnerpublicprojectsminutescostfactor"></a>`publicProjectsMinutesCostFactor` | [`Float`](#float) | Public projects' "minutes cost factor" associated with the runner (GitLab.com only). | | <a id="cirunnerrevision"></a>`revision` | [`String`](#string) | Revision of the runner. | | <a id="cirunnerrununtagged"></a>`runUntagged` | [`Boolean!`](#boolean) | Indicates the runner is able to run untagged jobs. | | <a id="cirunnerrunnertype"></a>`runnerType` | [`CiRunnerType!`](#cirunnertype) | Type of the runner. | | <a id="cirunnershortsha"></a>`shortSha` | [`String`](#string) | First eight characters of the runner's token used to authenticate new job requests. Used as the runner's unique ID. | | <a id="cirunnertaglist"></a>`tagList` | [`[String!]`](#string) | Tags associated with the runner. | +| <a id="cirunnertokenexpiresat"></a>`tokenExpiresAt` | [`Time`](#time) | Runner token expiration time. | | <a id="cirunneruserpermissions"></a>`userPermissions` | [`RunnerPermissions!`](#runnerpermissions) | Permissions for the current user on the resource. | | <a id="cirunnerversion"></a>`version` | [`String`](#string) | Version of the runner. | #### Fields with arguments +##### `CiRunner.jobs` + +Jobs assigned to the runner. + +Returns [`CiJobConnection`](#cijobconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="cirunnerjobsstatuses"></a>`statuses` | [`[CiJobStatus!]`](#cijobstatus) | Filter jobs by status. | + ##### `CiRunner.status` Status of the runner. @@ -9174,7 +9380,7 @@ Represents a code quality degradation on the pipeline. | <a id="codequalitydegradationfingerprint"></a>`fingerprint` | [`String!`](#string) | Unique fingerprint to identify the code quality degradation. For example, an MD5 hash. | | <a id="codequalitydegradationline"></a>`line` | [`Int!`](#int) | Line on which the code quality degradation occurred. | | <a id="codequalitydegradationpath"></a>`path` | [`String!`](#string) | Relative path to the file containing the code quality degradation. | -| <a id="codequalitydegradationseverity"></a>`severity` | [`CodeQualityDegradationSeverity!`](#codequalitydegradationseverity) | Status of the degradation (BLOCKER, CRITICAL, MAJOR, MINOR, INFO). | +| <a id="codequalitydegradationseverity"></a>`severity` | [`CodeQualityDegradationSeverity!`](#codequalitydegradationseverity) | Status of the degradation (BLOCKER, CRITICAL, MAJOR, MINOR, INFO, UNKNOWN). | ### `Commit` @@ -9518,6 +9724,7 @@ Represents a DAST profile schedule. | <a id="dastprofileschedulecadence"></a>`cadence` | [`DastProfileCadence`](#dastprofilecadence) | Cadence of the DAST profile schedule. | | <a id="dastprofilescheduleid"></a>`id` | [`DastProfileScheduleID!`](#dastprofilescheduleid) | ID of the DAST profile schedule. | | <a id="dastprofileschedulenextrunat"></a>`nextRunAt` | [`Time`](#time) | Next run time of the DAST profile schedule in the given timezone. | +| <a id="dastprofilescheduleownervalid"></a>`ownerValid` | [`Boolean`](#boolean) | Status of the current owner of the DAST profile schedule. | | <a id="dastprofileschedulestartsat"></a>`startsAt` | [`Time`](#time) | Start time of the DAST profile schedule in the given timezone. | | <a id="dastprofilescheduletimezone"></a>`timezone` | [`String`](#string) | Time zone of the start time of the DAST profile schedule. | @@ -10177,6 +10384,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | ---- | ---- | ----------- | | <a id="epicancestorsauthorusername"></a>`authorUsername` | [`String`](#string) | Filter epics by author. | | <a id="epicancestorsconfidential"></a>`confidential` | [`Boolean`](#boolean) | Filter epics by given confidentiality. | +| <a id="epicancestorscreatedafter"></a>`createdAfter` | [`Time`](#time) | Epics created after this date. | +| <a id="epicancestorscreatedbefore"></a>`createdBefore` | [`Time`](#time) | Epics created before this date. | | <a id="epicancestorsenddate"></a>`endDate` **{warning-solid}** | [`Time`](#time) | **Deprecated** in 13.5. Use timeframe.end. | | <a id="epicancestorsiid"></a>`iid` | [`ID`](#id) | IID of the epic, e.g., "1". | | <a id="epicancestorsiidstartswith"></a>`iidStartsWith` | [`String`](#string) | Filter epics by IID for autocomplete. | @@ -10193,6 +10402,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="epicancestorsstartdate"></a>`startDate` **{warning-solid}** | [`Time`](#time) | **Deprecated** in 13.5. Use timeframe.start. | | <a id="epicancestorsstate"></a>`state` | [`EpicState`](#epicstate) | Filter epics by state. | | <a id="epicancestorstimeframe"></a>`timeframe` | [`Timeframe`](#timeframe) | List items overlapping the given timeframe. | +| <a id="epicancestorsupdatedafter"></a>`updatedAfter` | [`Time`](#time) | Epics updated after this date. | +| <a id="epicancestorsupdatedbefore"></a>`updatedBefore` | [`Time`](#time) | Epics updated before this date. | ##### `Epic.children` @@ -10210,6 +10421,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | ---- | ---- | ----------- | | <a id="epicchildrenauthorusername"></a>`authorUsername` | [`String`](#string) | Filter epics by author. | | <a id="epicchildrenconfidential"></a>`confidential` | [`Boolean`](#boolean) | Filter epics by given confidentiality. | +| <a id="epicchildrencreatedafter"></a>`createdAfter` | [`Time`](#time) | Epics created after this date. | +| <a id="epicchildrencreatedbefore"></a>`createdBefore` | [`Time`](#time) | Epics created before this date. | | <a id="epicchildrenenddate"></a>`endDate` **{warning-solid}** | [`Time`](#time) | **Deprecated** in 13.5. Use timeframe.end. | | <a id="epicchildreniid"></a>`iid` | [`ID`](#id) | IID of the epic, e.g., "1". | | <a id="epicchildreniidstartswith"></a>`iidStartsWith` | [`String`](#string) | Filter epics by IID for autocomplete. | @@ -10226,6 +10439,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="epicchildrenstartdate"></a>`startDate` **{warning-solid}** | [`Time`](#time) | **Deprecated** in 13.5. Use timeframe.start. | | <a id="epicchildrenstate"></a>`state` | [`EpicState`](#epicstate) | Filter epics by state. | | <a id="epicchildrentimeframe"></a>`timeframe` | [`Timeframe`](#timeframe) | List items overlapping the given timeframe. | +| <a id="epicchildrenupdatedafter"></a>`updatedAfter` | [`Time`](#time) | Epics updated after this date. | +| <a id="epicchildrenupdatedbefore"></a>`updatedBefore` | [`Time`](#time) | Epics updated before this date. | ##### `Epic.currentUserTodos` @@ -10531,6 +10746,7 @@ Represents an external resource to send audit events to. | <a id="externalauditeventdestinationdestinationurl"></a>`destinationUrl` | [`String!`](#string) | External destination to send audit events to. | | <a id="externalauditeventdestinationgroup"></a>`group` | [`Group!`](#group) | Group the destination belongs to. | | <a id="externalauditeventdestinationid"></a>`id` | [`ID!`](#id) | ID of the destination. | +| <a id="externalauditeventdestinationverificationtoken"></a>`verificationToken` | [`String!`](#string) | Verification token to validate source of event. | ### `ExternalIssue` @@ -10588,6 +10804,22 @@ four standard [pagination arguments](#connection-pagination-arguments): | ---- | ---- | ----------- | | <a id="geonodegroupwikirepositoryregistriesids"></a>`ids` | [`[ID!]`](#id) | Filters registries by their ID. | +##### `GeoNode.jobArtifactRegistries` + +Find Job Artifact registries on this Geo node Available only when feature flag `geo_job_artifact_replication` is enabled. This flag is disabled by default, because the feature is experimental and is subject to change without notice. + +Returns [`JobArtifactRegistryConnection`](#jobartifactregistryconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="geonodejobartifactregistriesids"></a>`ids` | [`[ID!]`](#id) | Filters registries by their ID. | + ##### `GeoNode.lfsObjectRegistries` Find LFS object registries on this Geo node. @@ -10770,6 +11002,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="groupparent"></a>`parent` | [`Group`](#group) | Parent group. | | <a id="grouppath"></a>`path` | [`String!`](#string) | Path of the namespace. | | <a id="groupprojectcreationlevel"></a>`projectCreationLevel` | [`String`](#string) | Permission level required to create projects in the group. | +| <a id="grouprecentissueboards"></a>`recentIssueBoards` | [`BoardConnection`](#boardconnection) | List of recently visited boards of the group. Maximum size is 4. (see [Connections](#connections)) | | <a id="grouprepositorysizeexcessprojectcount"></a>`repositorySizeExcessProjectCount` | [`Int!`](#int) | Number of projects in the root namespace where the repository size exceeds the limit. | | <a id="grouprequestaccessenabled"></a>`requestAccessEnabled` | [`Boolean`](#boolean) | Indicates if users can request access to namespace. | | <a id="grouprequiretwofactorauthentication"></a>`requireTwoFactorAuthentication` | [`Boolean`](#boolean) | Indicates if all users in this group are required to set up two-factor authentication. | @@ -10898,6 +11131,8 @@ Returns [`Epic`](#epic). | ---- | ---- | ----------- | | <a id="groupepicauthorusername"></a>`authorUsername` | [`String`](#string) | Filter epics by author. | | <a id="groupepicconfidential"></a>`confidential` | [`Boolean`](#boolean) | Filter epics by given confidentiality. | +| <a id="groupepiccreatedafter"></a>`createdAfter` | [`Time`](#time) | Epics created after this date. | +| <a id="groupepiccreatedbefore"></a>`createdBefore` | [`Time`](#time) | Epics created before this date. | | <a id="groupepicenddate"></a>`endDate` **{warning-solid}** | [`Time`](#time) | **Deprecated** in 13.5. Use timeframe.end. | | <a id="groupepiciid"></a>`iid` | [`ID`](#id) | IID of the epic, e.g., "1". | | <a id="groupepiciidstartswith"></a>`iidStartsWith` | [`String`](#string) | Filter epics by IID for autocomplete. | @@ -10914,6 +11149,8 @@ Returns [`Epic`](#epic). | <a id="groupepicstartdate"></a>`startDate` **{warning-solid}** | [`Time`](#time) | **Deprecated** in 13.5. Use timeframe.start. | | <a id="groupepicstate"></a>`state` | [`EpicState`](#epicstate) | Filter epics by state. | | <a id="groupepictimeframe"></a>`timeframe` | [`Timeframe`](#timeframe) | List items overlapping the given timeframe. | +| <a id="groupepicupdatedafter"></a>`updatedAfter` | [`Time`](#time) | Epics updated after this date. | +| <a id="groupepicupdatedbefore"></a>`updatedBefore` | [`Time`](#time) | Epics updated before this date. | ##### `Group.epicBoard` @@ -10943,6 +11180,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | ---- | ---- | ----------- | | <a id="groupepicsauthorusername"></a>`authorUsername` | [`String`](#string) | Filter epics by author. | | <a id="groupepicsconfidential"></a>`confidential` | [`Boolean`](#boolean) | Filter epics by given confidentiality. | +| <a id="groupepicscreatedafter"></a>`createdAfter` | [`Time`](#time) | Epics created after this date. | +| <a id="groupepicscreatedbefore"></a>`createdBefore` | [`Time`](#time) | Epics created before this date. | | <a id="groupepicsenddate"></a>`endDate` **{warning-solid}** | [`Time`](#time) | **Deprecated** in 13.5. Use timeframe.end. | | <a id="groupepicsiid"></a>`iid` | [`ID`](#id) | IID of the epic, e.g., "1". | | <a id="groupepicsiidstartswith"></a>`iidStartsWith` | [`String`](#string) | Filter epics by IID for autocomplete. | @@ -10959,6 +11198,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="groupepicsstartdate"></a>`startDate` **{warning-solid}** | [`Time`](#time) | **Deprecated** in 13.5. Use timeframe.start. | | <a id="groupepicsstate"></a>`state` | [`EpicState`](#epicstate) | Filter epics by state. | | <a id="groupepicstimeframe"></a>`timeframe` | [`Timeframe`](#timeframe) | List items overlapping the given timeframe. | +| <a id="groupepicsupdatedafter"></a>`updatedAfter` | [`Time`](#time) | Epics updated after this date. | +| <a id="groupepicsupdatedbefore"></a>`updatedBefore` | [`Time`](#time) | Epics updated before this date. | ##### `Group.groupMembers` @@ -11058,12 +11299,15 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="groupiterationsenddate"></a>`endDate` **{warning-solid}** | [`Time`](#time) | **Deprecated** in 13.5. Use timeframe.end. | | <a id="groupiterationsid"></a>`id` | [`ID`](#id) | Global ID of the Iteration to look up. | | <a id="groupiterationsiid"></a>`iid` | [`ID`](#id) | Internal ID of the Iteration to look up. | +| <a id="groupiterationsin"></a>`in` | [`[IterationSearchableField!]`](#iterationsearchablefield) | Fields in which the fuzzy-search should be performed with the query given in the argument `search`. Defaults to `[title]`. | | <a id="groupiterationsincludeancestors"></a>`includeAncestors` | [`Boolean`](#boolean) | Whether to include ancestor iterations. Defaults to true. | | <a id="groupiterationsiterationcadenceids"></a>`iterationCadenceIds` | [`[IterationsCadenceID!]`](#iterationscadenceid) | Global iteration cadence IDs by which to look up the iterations. | +| <a id="groupiterationssearch"></a>`search` | [`String`](#string) | Query used for fuzzy-searching in the fields selected in the argument `in`. Returns all iterations if empty. | +| <a id="groupiterationssort"></a>`sort` | [`IterationSort`](#iterationsort) | List iterations by sort order. If unspecified, an arbitrary order (subject to change) is used. | | <a id="groupiterationsstartdate"></a>`startDate` **{warning-solid}** | [`Time`](#time) | **Deprecated** in 13.5. Use timeframe.start. | | <a id="groupiterationsstate"></a>`state` | [`IterationState`](#iterationstate) | Filter iterations by state. | | <a id="groupiterationstimeframe"></a>`timeframe` | [`Timeframe`](#timeframe) | List items overlapping the given timeframe. | -| <a id="groupiterationstitle"></a>`title` | [`String`](#string) | Fuzzy search by title. | +| <a id="groupiterationstitle"></a>`title` **{warning-solid}** | [`String`](#string) | **Deprecated** in 15.4. The argument will be removed in 15.4. Please use `search` and `in` fields instead. | ##### `Group.label` @@ -11126,6 +11370,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="groupmergerequestssourcebranches"></a>`sourceBranches` | [`[String!]`](#string) | Array of source branch names. All resolved merge requests will have one of these branches as their source. | | <a id="groupmergerequestsstate"></a>`state` | [`MergeRequestState`](#mergerequeststate) | Merge request state. If provided, all resolved merge requests will have this state. | | <a id="groupmergerequeststargetbranches"></a>`targetBranches` | [`[String!]`](#string) | Array of target branch names. All resolved merge requests will have one of these branches as their target. | +| <a id="groupmergerequestsupdatedafter"></a>`updatedAfter` | [`Time`](#time) | Merge requests updated after this timestamp. | +| <a id="groupmergerequestsupdatedbefore"></a>`updatedBefore` | [`Time`](#time) | Merge requests updated before this timestamp. | ##### `Group.milestones` @@ -11208,8 +11454,9 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="grouprunnersactive"></a>`active` | [`Boolean`](#boolean) | Filter runners by active (true) or paused (false) status. | +| <a id="grouprunnersactive"></a>`active` **{warning-solid}** | [`Boolean`](#boolean) | **Deprecated** in 14.8. This was renamed. Use: `paused`. | | <a id="grouprunnersmembership"></a>`membership` | [`RunnerMembershipFilter`](#runnermembershipfilter) | Control which runners to include in the results. | +| <a id="grouprunnerspaused"></a>`paused` | [`Boolean`](#boolean) | Filter runners by `paused` (true) or `active` (false) status. | | <a id="grouprunnerssearch"></a>`search` | [`String`](#string) | Filter by full token or partial text in description field. | | <a id="grouprunnerssort"></a>`sort` | [`CiRunnerSort`](#cirunnersort) | Sort order of results. | | <a id="grouprunnersstatus"></a>`status` | [`CiRunnerStatus`](#cirunnerstatus) | Filter runners by status. | @@ -11331,6 +11578,20 @@ Represents a Group Membership. | <a id="groupmemberuser"></a>`user` | [`UserCore`](#usercore) | User that is associated with the member object. | | <a id="groupmemberuserpermissions"></a>`userPermissions` | [`GroupPermissions!`](#grouppermissions) | Permissions for the current user on the resource. | +#### Fields with arguments + +##### `GroupMember.mergeRequestInteraction` + +Find a merge request. + +Returns [`UserMergeRequestInteraction`](#usermergerequestinteraction). + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="groupmembermergerequestinteractionid"></a>`id` | [`MergeRequestID!`](#mergerequestid) | Global ID of the merge request. | + ### `GroupPermissions` #### Fields @@ -11749,6 +12010,23 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="jirauserjiradisplayname"></a>`jiraDisplayName` | [`String!`](#string) | Display name of the Jira user. | | <a id="jirauserjiraemail"></a>`jiraEmail` | [`String`](#string) | Email of the Jira user, returned only for users with public emails. | +### `JobArtifactRegistry` + +Represents the Geo replication and verification state of a job_artifact. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="jobartifactregistryartifactid"></a>`artifactId` | [`ID!`](#id) | ID of the Job Artifact. | +| <a id="jobartifactregistrycreatedat"></a>`createdAt` | [`Time`](#time) | Timestamp when the JobArtifactRegistry was created. | +| <a id="jobartifactregistryid"></a>`id` | [`ID!`](#id) | ID of the JobArtifactRegistry. | +| <a id="jobartifactregistrylastsyncfailure"></a>`lastSyncFailure` | [`String`](#string) | Error message during sync of the JobArtifactRegistry. | +| <a id="jobartifactregistrylastsyncedat"></a>`lastSyncedAt` | [`Time`](#time) | Timestamp of the most recent successful sync of the JobArtifactRegistry. | +| <a id="jobartifactregistryretryat"></a>`retryAt` | [`Time`](#time) | Timestamp after which the JobArtifactRegistry should be resynced. | +| <a id="jobartifactregistryretrycount"></a>`retryCount` | [`Int`](#int) | Number of consecutive failed sync attempts of the JobArtifactRegistry. | +| <a id="jobartifactregistrystate"></a>`state` | [`RegistryState`](#registrystate) | Sync state of the JobArtifactRegistry. | + ### `JobPermissions` #### Fields @@ -12022,7 +12300,7 @@ A user assigned to a merge request. | <a id="mergerequestassigneeid"></a>`id` | [`ID!`](#id) | ID of the user. | | <a id="mergerequestassigneelocation"></a>`location` | [`String`](#string) | Location of the user. | | <a id="mergerequestassigneemergerequestinteraction"></a>`mergeRequestInteraction` | [`UserMergeRequestInteraction`](#usermergerequestinteraction) | Details of this user's interactions with the merge request. | -| <a id="mergerequestassigneename"></a>`name` | [`String!`](#string) | Human-readable name of the user. Will return `****` if the user is a project bot and the requester does not have permission to read resource access tokens. | +| <a id="mergerequestassigneename"></a>`name` | [`String!`](#string) | Human-readable name of the user. Returns `****` if the user is a project bot and the requester does not have permission to view the project. | | <a id="mergerequestassigneenamespace"></a>`namespace` | [`Namespace`](#namespace) | Personal namespace of the user. | | <a id="mergerequestassigneeprojectmemberships"></a>`projectMemberships` | [`ProjectMemberConnection`](#projectmemberconnection) | Project memberships of the user. (see [Connections](#connections)) | | <a id="mergerequestassigneepublicemail"></a>`publicEmail` | [`String`](#string) | User's public email. | @@ -12066,6 +12344,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="mergerequestassigneeassignedmergerequestssourcebranches"></a>`sourceBranches` | [`[String!]`](#string) | Array of source branch names. All resolved merge requests will have one of these branches as their source. | | <a id="mergerequestassigneeassignedmergerequestsstate"></a>`state` | [`MergeRequestState`](#mergerequeststate) | Merge request state. If provided, all resolved merge requests will have this state. | | <a id="mergerequestassigneeassignedmergerequeststargetbranches"></a>`targetBranches` | [`[String!]`](#string) | Array of target branch names. All resolved merge requests will have one of these branches as their target. | +| <a id="mergerequestassigneeassignedmergerequestsupdatedafter"></a>`updatedAfter` | [`Time`](#time) | Merge requests updated after this timestamp. | +| <a id="mergerequestassigneeassignedmergerequestsupdatedbefore"></a>`updatedBefore` | [`Time`](#time) | Merge requests updated before this timestamp. | ##### `MergeRequestAssignee.authoredMergeRequests` @@ -12098,6 +12378,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="mergerequestassigneeauthoredmergerequestssourcebranches"></a>`sourceBranches` | [`[String!]`](#string) | Array of source branch names. All resolved merge requests will have one of these branches as their source. | | <a id="mergerequestassigneeauthoredmergerequestsstate"></a>`state` | [`MergeRequestState`](#mergerequeststate) | Merge request state. If provided, all resolved merge requests will have this state. | | <a id="mergerequestassigneeauthoredmergerequeststargetbranches"></a>`targetBranches` | [`[String!]`](#string) | Array of target branch names. All resolved merge requests will have one of these branches as their target. | +| <a id="mergerequestassigneeauthoredmergerequestsupdatedafter"></a>`updatedAfter` | [`Time`](#time) | Merge requests updated after this timestamp. | +| <a id="mergerequestassigneeauthoredmergerequestsupdatedbefore"></a>`updatedBefore` | [`Time`](#time) | Merge requests updated before this timestamp. | ##### `MergeRequestAssignee.groups` @@ -12147,6 +12429,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="mergerequestassigneereviewrequestedmergerequestssourcebranches"></a>`sourceBranches` | [`[String!]`](#string) | Array of source branch names. All resolved merge requests will have one of these branches as their source. | | <a id="mergerequestassigneereviewrequestedmergerequestsstate"></a>`state` | [`MergeRequestState`](#mergerequeststate) | Merge request state. If provided, all resolved merge requests will have this state. | | <a id="mergerequestassigneereviewrequestedmergerequeststargetbranches"></a>`targetBranches` | [`[String!]`](#string) | Array of target branch names. All resolved merge requests will have one of these branches as their target. | +| <a id="mergerequestassigneereviewrequestedmergerequestsupdatedafter"></a>`updatedAfter` | [`Time`](#time) | Merge requests updated after this timestamp. | +| <a id="mergerequestassigneereviewrequestedmergerequestsupdatedbefore"></a>`updatedBefore` | [`Time`](#time) | Merge requests updated before this timestamp. | ##### `MergeRequestAssignee.snippets` @@ -12277,7 +12561,7 @@ A user assigned to a merge request as a reviewer. | <a id="mergerequestreviewerid"></a>`id` | [`ID!`](#id) | ID of the user. | | <a id="mergerequestreviewerlocation"></a>`location` | [`String`](#string) | Location of the user. | | <a id="mergerequestreviewermergerequestinteraction"></a>`mergeRequestInteraction` | [`UserMergeRequestInteraction`](#usermergerequestinteraction) | Details of this user's interactions with the merge request. | -| <a id="mergerequestreviewername"></a>`name` | [`String!`](#string) | Human-readable name of the user. Will return `****` if the user is a project bot and the requester does not have permission to read resource access tokens. | +| <a id="mergerequestreviewername"></a>`name` | [`String!`](#string) | Human-readable name of the user. Returns `****` if the user is a project bot and the requester does not have permission to view the project. | | <a id="mergerequestreviewernamespace"></a>`namespace` | [`Namespace`](#namespace) | Personal namespace of the user. | | <a id="mergerequestreviewerprojectmemberships"></a>`projectMemberships` | [`ProjectMemberConnection`](#projectmemberconnection) | Project memberships of the user. (see [Connections](#connections)) | | <a id="mergerequestreviewerpublicemail"></a>`publicEmail` | [`String`](#string) | User's public email. | @@ -12321,6 +12605,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="mergerequestreviewerassignedmergerequestssourcebranches"></a>`sourceBranches` | [`[String!]`](#string) | Array of source branch names. All resolved merge requests will have one of these branches as their source. | | <a id="mergerequestreviewerassignedmergerequestsstate"></a>`state` | [`MergeRequestState`](#mergerequeststate) | Merge request state. If provided, all resolved merge requests will have this state. | | <a id="mergerequestreviewerassignedmergerequeststargetbranches"></a>`targetBranches` | [`[String!]`](#string) | Array of target branch names. All resolved merge requests will have one of these branches as their target. | +| <a id="mergerequestreviewerassignedmergerequestsupdatedafter"></a>`updatedAfter` | [`Time`](#time) | Merge requests updated after this timestamp. | +| <a id="mergerequestreviewerassignedmergerequestsupdatedbefore"></a>`updatedBefore` | [`Time`](#time) | Merge requests updated before this timestamp. | ##### `MergeRequestReviewer.authoredMergeRequests` @@ -12353,6 +12639,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="mergerequestreviewerauthoredmergerequestssourcebranches"></a>`sourceBranches` | [`[String!]`](#string) | Array of source branch names. All resolved merge requests will have one of these branches as their source. | | <a id="mergerequestreviewerauthoredmergerequestsstate"></a>`state` | [`MergeRequestState`](#mergerequeststate) | Merge request state. If provided, all resolved merge requests will have this state. | | <a id="mergerequestreviewerauthoredmergerequeststargetbranches"></a>`targetBranches` | [`[String!]`](#string) | Array of target branch names. All resolved merge requests will have one of these branches as their target. | +| <a id="mergerequestreviewerauthoredmergerequestsupdatedafter"></a>`updatedAfter` | [`Time`](#time) | Merge requests updated after this timestamp. | +| <a id="mergerequestreviewerauthoredmergerequestsupdatedbefore"></a>`updatedBefore` | [`Time`](#time) | Merge requests updated before this timestamp. | ##### `MergeRequestReviewer.groups` @@ -12402,6 +12690,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="mergerequestreviewerreviewrequestedmergerequestssourcebranches"></a>`sourceBranches` | [`[String!]`](#string) | Array of source branch names. All resolved merge requests will have one of these branches as their source. | | <a id="mergerequestreviewerreviewrequestedmergerequestsstate"></a>`state` | [`MergeRequestState`](#mergerequeststate) | Merge request state. If provided, all resolved merge requests will have this state. | | <a id="mergerequestreviewerreviewrequestedmergerequeststargetbranches"></a>`targetBranches` | [`[String!]`](#string) | Array of target branch names. All resolved merge requests will have one of these branches as their target. | +| <a id="mergerequestreviewerreviewrequestedmergerequestsupdatedafter"></a>`updatedAfter` | [`Time`](#time) | Merge requests updated after this timestamp. | +| <a id="mergerequestreviewerreviewrequestedmergerequestsupdatedbefore"></a>`updatedBefore` | [`Time`](#time) | Merge requests updated before this timestamp. | ##### `MergeRequestReviewer.snippets` @@ -13157,6 +13447,19 @@ Represents the Geo sync and verification state of a pipeline artifact. | <a id="pipelineartifactregistryretrycount"></a>`retryCount` | [`Int`](#int) | Number of consecutive failed sync attempts of the PipelineArtifactRegistry. | | <a id="pipelineartifactregistrystate"></a>`state` | [`RegistryState`](#registrystate) | Sync state of the PipelineArtifactRegistry. | +### `PipelineCounts` + +Represents pipeline counts for the project. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="pipelinecountsall"></a>`all` | [`Int`](#int) | Total number of pipelines for the project. | +| <a id="pipelinecountsfinished"></a>`finished` | [`Int`](#int) | Number of pipelines with scope FINISHED for the project. | +| <a id="pipelinecountspending"></a>`pending` | [`Int`](#int) | Number of pipelines with scope PENDING for the project. | +| <a id="pipelinecountsrunning"></a>`running` | [`Int`](#int) | Number of pipelines with scope RUNNING for the project. | + ### `PipelineMessage` #### Fields @@ -13184,10 +13487,13 @@ Represents vulnerability finding of a security report on the pipeline. | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="pipelinesecurityreportfindingassets"></a>`assets` | [`[AssetType!]`](#assettype) | List of assets associated with the vulnerability. | | <a id="pipelinesecurityreportfindingconfidence"></a>`confidence` | [`String`](#string) | Type of the security report that found the vulnerability. | | <a id="pipelinesecurityreportfindingdescription"></a>`description` | [`String`](#string) | Description of the vulnerability finding. | +| <a id="pipelinesecurityreportfindingevidence"></a>`evidence` | [`VulnerabilityEvidence`](#vulnerabilityevidence) | Evidence for the vulnerability. | | <a id="pipelinesecurityreportfindingfalsepositive"></a>`falsePositive` | [`Boolean`](#boolean) | Indicates whether the vulnerability is a false positive. | -| <a id="pipelinesecurityreportfindingidentifiers"></a>`identifiers` | [`[VulnerabilityIdentifier!]!`](#vulnerabilityidentifier) | Identifiers of the vulnerabilit finding. | +| <a id="pipelinesecurityreportfindingidentifiers"></a>`identifiers` | [`[VulnerabilityIdentifier!]!`](#vulnerabilityidentifier) | Identifiers of the vulnerability finding. | +| <a id="pipelinesecurityreportfindinglinks"></a>`links` | [`[VulnerabilityLink!]`](#vulnerabilitylink) | List of links associated with the vulnerability. | | <a id="pipelinesecurityreportfindinglocation"></a>`location` | [`VulnerabilityLocation`](#vulnerabilitylocation) | Location metadata for the vulnerability. Its fields depend on the type of security scan that found the vulnerability. | | <a id="pipelinesecurityreportfindingname"></a>`name` | [`String`](#string) | Name of the vulnerability finding. | | <a id="pipelinesecurityreportfindingproject"></a>`project` | [`Project`](#project) | Project on which the vulnerability finding was found. | @@ -13197,6 +13503,7 @@ Represents vulnerability finding of a security report on the pipeline. | <a id="pipelinesecurityreportfindingseverity"></a>`severity` | [`VulnerabilitySeverity`](#vulnerabilityseverity) | Severity of the vulnerability finding. | | <a id="pipelinesecurityreportfindingsolution"></a>`solution` | [`String`](#string) | URL to the vulnerability's details page. | | <a id="pipelinesecurityreportfindingstate"></a>`state` | [`VulnerabilityState`](#vulnerabilitystate) | Finding status. | +| <a id="pipelinesecurityreportfindingtitle"></a>`title` | [`String`](#string) | Title of the vulnerability finding. | | <a id="pipelinesecurityreportfindinguuid"></a>`uuid` | [`String`](#string) | Name of the vulnerability finding. | ### `Project` @@ -13213,6 +13520,7 @@ Represents vulnerability finding of a security report on the pipeline. | <a id="projectautoclosereferencedissues"></a>`autocloseReferencedIssues` | [`Boolean`](#boolean) | Indicates if issues referenced by merge requests and commits within the default branch are closed automatically. | | <a id="projectavatarurl"></a>`avatarUrl` | [`String`](#string) | URL to avatar image file of the project. | | <a id="projectcicdsettings"></a>`ciCdSettings` | [`ProjectCiCdSetting`](#projectcicdsetting) | CI/CD settings for the project. | +| <a id="projectciconfigpathordefault"></a>`ciConfigPathOrDefault` | [`String!`](#string) | Path of the CI configuration file. | | <a id="projectcijobtokenscope"></a>`ciJobTokenScope` | [`CiJobTokenScopeType`](#cijobtokenscopetype) | The CI Job Tokens scope of access. | | <a id="projectclusteragents"></a>`clusterAgents` | [`ClusterAgentConnection`](#clusteragentconnection) | Cluster agents associated with the project. (see [Connections](#connections)) | | <a id="projectcodecoveragesummary"></a>`codeCoverageSummary` | [`CodeCoverageSummary`](#codecoveragesummary) | Code coverage summary associated with the project. | @@ -13220,7 +13528,7 @@ Represents vulnerability finding of a security report on the pipeline. | <a id="projectcontainerexpirationpolicy"></a>`containerExpirationPolicy` | [`ContainerExpirationPolicy`](#containerexpirationpolicy) | Container expiration policy of the project. | | <a id="projectcontainerregistryenabled"></a>`containerRegistryEnabled` | [`Boolean`](#boolean) | Indicates if Container Registry is enabled for the current user. | | <a id="projectcontainerrepositoriescount"></a>`containerRepositoriesCount` | [`Int!`](#int) | Number of container repositories in the project. | -| <a id="projectcorpuses"></a>`corpuses` | [`CoverageFuzzingCorpusConnection`](#coveragefuzzingcorpusconnection) | Find corpuses of the project. Available only when feature flag `corpus_management` is enabled. This flag is disabled by default, because the feature is experimental and is subject to change without notice. (see [Connections](#connections)) | +| <a id="projectcorpuses"></a>`corpuses` | [`CoverageFuzzingCorpusConnection`](#coveragefuzzingcorpusconnection) | Find corpuses of the project. Available only when feature flag `corpus_management` is enabled. This flag is enabled by default. (see [Connections](#connections)) | | <a id="projectcreatedat"></a>`createdAt` | [`Time`](#time) | Timestamp of the project creation. | | <a id="projectdastscannerprofiles"></a>`dastScannerProfiles` | [`DastScannerProfileConnection`](#dastscannerprofileconnection) | DAST scanner profiles associated with the project. (see [Connections](#connections)) | | <a id="projectdastsiteprofiles"></a>`dastSiteProfiles` | [`DastSiteProfileConnection`](#dastsiteprofileconnection) | DAST Site Profiles associated with the project. (see [Connections](#connections)) | @@ -13234,7 +13542,6 @@ Represents vulnerability finding of a security report on the pipeline. | <a id="projecthttpurltorepo"></a>`httpUrlToRepo` | [`String`](#string) | URL to connect to the project via HTTPS. | | <a id="projectid"></a>`id` | [`ID!`](#id) | ID of the project. | | <a id="projectimportstatus"></a>`importStatus` | [`String`](#string) | Status of import background job of the project. | -| <a id="projectincidentmanagementescalationpolicies"></a>`incidentManagementEscalationPolicies` | [`EscalationPolicyTypeConnection`](#escalationpolicytypeconnection) | Incident Management escalation policies of the project. (see [Connections](#connections)) | | <a id="projectissuesenabled"></a>`issuesEnabled` | [`Boolean`](#boolean) | Indicates if Issues are enabled for the current user. | | <a id="projectjiraimportstatus"></a>`jiraImportStatus` | [`String`](#string) | Status of Jira import background job of the project. | | <a id="projectjiraimports"></a>`jiraImports` | [`JiraImportConnection`](#jiraimportconnection) | Jira imports into the project. (see [Connections](#connections)) | @@ -13256,6 +13563,7 @@ Represents vulnerability finding of a security report on the pipeline. | <a id="projectprintingmergerequestlinkenabled"></a>`printingMergeRequestLinkEnabled` | [`Boolean`](#boolean) | Indicates if a link to create or view a merge request should display after a push to Git repositories of the project from the command line. | | <a id="projectpublicjobs"></a>`publicJobs` | [`Boolean`](#boolean) | Indicates if there is public access to pipelines and job details of the project, including output logs and artifacts. | | <a id="projectpushrules"></a>`pushRules` | [`PushRules`](#pushrules) | Project's push rules settings. | +| <a id="projectrecentissueboards"></a>`recentIssueBoards` | [`BoardConnection`](#boardconnection) | List of recently visited boards of the project. Maximum size is 4. (see [Connections](#connections)) | | <a id="projectremovesourcebranchaftermerge"></a>`removeSourceBranchAfterMerge` | [`Boolean`](#boolean) | Indicates if `Delete source branch` option should be enabled by default for all new merge requests of the project. | | <a id="projectrepository"></a>`repository` | [`Repository`](#repository) | Git repository of the project. | | <a id="projectrepositorysizeexcess"></a>`repositorySizeExcess` | [`Float`](#float) | Size of repository that exceeds the limit in bytes. | @@ -13542,6 +13850,22 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="projectenvironmentssearch"></a>`search` | [`String`](#string) | Search query for environment name. | | <a id="projectenvironmentsstates"></a>`states` | [`[String!]`](#string) | States of environments that should be included in result. | +##### `Project.incidentManagementEscalationPolicies` + +Incident Management escalation policies of the project. + +Returns [`EscalationPolicyTypeConnection`](#escalationpolicytypeconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="projectincidentmanagementescalationpoliciesname"></a>`name` | [`String`](#string) | Fuzzy search by escalation policy name. | + ##### `Project.incidentManagementEscalationPolicy` Incident Management escalation policy of the project. @@ -13553,6 +13877,7 @@ Returns [`EscalationPolicyType`](#escalationpolicytype). | Name | Type | Description | | ---- | ---- | ----------- | | <a id="projectincidentmanagementescalationpolicyid"></a>`id` | [`IncidentManagementEscalationPolicyID!`](#incidentmanagementescalationpolicyid) | ID of the escalation policy. | +| <a id="projectincidentmanagementescalationpolicyname"></a>`name` | [`String`](#string) | Fuzzy search by escalation policy name. | ##### `Project.incidentManagementOncallSchedules` @@ -13754,12 +14079,15 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="projectiterationsenddate"></a>`endDate` **{warning-solid}** | [`Time`](#time) | **Deprecated** in 13.5. Use timeframe.end. | | <a id="projectiterationsid"></a>`id` | [`ID`](#id) | Global ID of the Iteration to look up. | | <a id="projectiterationsiid"></a>`iid` | [`ID`](#id) | Internal ID of the Iteration to look up. | +| <a id="projectiterationsin"></a>`in` | [`[IterationSearchableField!]`](#iterationsearchablefield) | Fields in which the fuzzy-search should be performed with the query given in the argument `search`. Defaults to `[title]`. | | <a id="projectiterationsincludeancestors"></a>`includeAncestors` | [`Boolean`](#boolean) | Whether to include ancestor iterations. Defaults to true. | | <a id="projectiterationsiterationcadenceids"></a>`iterationCadenceIds` | [`[IterationsCadenceID!]`](#iterationscadenceid) | Global iteration cadence IDs by which to look up the iterations. | +| <a id="projectiterationssearch"></a>`search` | [`String`](#string) | Query used for fuzzy-searching in the fields selected in the argument `in`. Returns all iterations if empty. | +| <a id="projectiterationssort"></a>`sort` | [`IterationSort`](#iterationsort) | List iterations by sort order. If unspecified, an arbitrary order (subject to change) is used. | | <a id="projectiterationsstartdate"></a>`startDate` **{warning-solid}** | [`Time`](#time) | **Deprecated** in 13.5. Use timeframe.start. | | <a id="projectiterationsstate"></a>`state` | [`IterationState`](#iterationstate) | Filter iterations by state. | | <a id="projectiterationstimeframe"></a>`timeframe` | [`Timeframe`](#timeframe) | List items overlapping the given timeframe. | -| <a id="projectiterationstitle"></a>`title` | [`String`](#string) | Fuzzy search by title. | +| <a id="projectiterationstitle"></a>`title` **{warning-solid}** | [`String`](#string) | **Deprecated** in 15.4. The argument will be removed in 15.4. Please use `search` and `in` fields instead. | ##### `Project.jobs` @@ -13848,6 +14176,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="projectmergerequestssourcebranches"></a>`sourceBranches` | [`[String!]`](#string) | Array of source branch names. All resolved merge requests will have one of these branches as their source. | | <a id="projectmergerequestsstate"></a>`state` | [`MergeRequestState`](#mergerequeststate) | Merge request state. If provided, all resolved merge requests will have this state. | | <a id="projectmergerequeststargetbranches"></a>`targetBranches` | [`[String!]`](#string) | Array of target branch names. All resolved merge requests will have one of these branches as their target. | +| <a id="projectmergerequestsupdatedafter"></a>`updatedAfter` | [`Time`](#time) | Merge requests updated after this timestamp. | +| <a id="projectmergerequestsupdatedbefore"></a>`updatedBefore` | [`Time`](#time) | Merge requests updated before this timestamp. | ##### `Project.milestones` @@ -13923,6 +14253,20 @@ Returns [`Pipeline`](#pipeline). | <a id="projectpipelineiid"></a>`iid` | [`ID`](#id) | IID of the Pipeline. For example, "1". | | <a id="projectpipelinesha"></a>`sha` | [`String`](#string) | SHA of the Pipeline. For example, "dyd0f15ay83993f5ab66k927w28673882x99100b". | +##### `Project.pipelineCounts` + +Build pipeline counts of the project. + +Returns [`PipelineCounts`](#pipelinecounts). + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="projectpipelinecountsref"></a>`ref` | [`String`](#string) | Filter pipelines by the ref they are run for. | +| <a id="projectpipelinecountssha"></a>`sha` | [`String`](#string) | Filter pipelines by the SHA of the commit they are run for. | +| <a id="projectpipelinecountssource"></a>`source` | [`String`](#string) | Filter pipelines by their source. | + ##### `Project.pipelines` Build pipelines of the project. @@ -14229,6 +14573,20 @@ Represents a Project Membership. | <a id="projectmemberuser"></a>`user` | [`UserCore`](#usercore) | User that is associated with the member object. | | <a id="projectmemberuserpermissions"></a>`userPermissions` | [`ProjectPermissions!`](#projectpermissions) | Permissions for the current user on the resource. | +#### Fields with arguments + +##### `ProjectMember.mergeRequestInteraction` + +Find a merge request. + +Returns [`UserMergeRequestInteraction`](#usermergerequestinteraction). + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="projectmembermergerequestinteractionid"></a>`id` | [`MergeRequestID!`](#mergerequestid) | Global ID of the merge request. | + ### `ProjectPermissions` #### Fields @@ -14539,6 +14897,8 @@ Returns [`Tree`](#tree). | <a id="repositoryblobcanmodifyblob"></a>`canModifyBlob` | [`Boolean`](#boolean) | Whether the current user can modify the blob. | | <a id="repositoryblobcodeowners"></a>`codeOwners` | [`[UserCore!]`](#usercore) | List of code owners for the blob. | | <a id="repositoryblobeditblobpath"></a>`editBlobPath` | [`String`](#string) | Web path to edit the blob in the old-style editor. | +| <a id="repositoryblobenvironmentexternalurlforroutemap"></a>`environmentExternalUrlForRouteMap` | [`String`](#string) | Web path to blob on an environment. | +| <a id="repositoryblobenvironmentformattedexternalurl"></a>`environmentFormattedExternalUrl` | [`String`](#string) | Environment on which the blob is available. | | <a id="repositoryblobexternalstorage"></a>`externalStorage` | [`String`](#string) | External storage being used, if enabled (for instance, 'LFS'). | | <a id="repositoryblobexternalstorageurl"></a>`externalStorageUrl` | [`String`](#string) | Web path to download the raw blob via external storage, if enabled. | | <a id="repositoryblobfiletype"></a>`fileType` | [`String`](#string) | Expected format of the blob based on the extension. | @@ -14548,6 +14908,7 @@ Returns [`Tree`](#tree). | <a id="repositoryblobid"></a>`id` | [`ID!`](#id) | ID of the blob. | | <a id="repositoryblobideeditpath"></a>`ideEditPath` | [`String`](#string) | Web path to edit this blob in the Web IDE. | | <a id="repositoryblobideforkandeditpath"></a>`ideForkAndEditPath` | [`String`](#string) | Web path to edit this blob in the Web IDE using a forked project. | +| <a id="repositorybloblanguage"></a>`language` | [`String`](#string) | Blob language. | | <a id="repositorybloblfsoid"></a>`lfsOid` | [`String`](#string) | LFS OID of the blob. | | <a id="repositoryblobmode"></a>`mode` | [`String`](#string) | Blob mode. | | <a id="repositoryblobname"></a>`name` | [`String`](#string) | Blob name. | @@ -14640,6 +15001,7 @@ Counts of requirements by their state. | Name | Type | Description | | ---- | ---- | ----------- | | <a id="rootstoragestatisticsbuildartifactssize"></a>`buildArtifactsSize` | [`Float!`](#float) | CI artifacts size in bytes. | +| <a id="rootstoragestatisticsdependencyproxysize"></a>`dependencyProxySize` | [`Float!`](#float) | Dependency Proxy sizes in bytes. | | <a id="rootstoragestatisticslfsobjectssize"></a>`lfsObjectsSize` | [`Float!`](#float) | LFS objects size in bytes. | | <a id="rootstoragestatisticspackagessize"></a>`packagesSize` | [`Float!`](#float) | Packages size in bytes. | | <a id="rootstoragestatisticspipelineartifactssize"></a>`pipelineArtifactsSize` | [`Float!`](#float) | CI pipeline artifacts size in bytes. | @@ -15497,7 +15859,7 @@ Core represention of a GitLab user. | <a id="usercoregroupmemberships"></a>`groupMemberships` | [`GroupMemberConnection`](#groupmemberconnection) | Group memberships of the user. (see [Connections](#connections)) | | <a id="usercoreid"></a>`id` | [`ID!`](#id) | ID of the user. | | <a id="usercorelocation"></a>`location` | [`String`](#string) | Location of the user. | -| <a id="usercorename"></a>`name` | [`String!`](#string) | Human-readable name of the user. Will return `****` if the user is a project bot and the requester does not have permission to read resource access tokens. | +| <a id="usercorename"></a>`name` | [`String!`](#string) | Human-readable name of the user. Returns `****` if the user is a project bot and the requester does not have permission to view the project. | | <a id="usercorenamespace"></a>`namespace` | [`Namespace`](#namespace) | Personal namespace of the user. | | <a id="usercoreprojectmemberships"></a>`projectMemberships` | [`ProjectMemberConnection`](#projectmemberconnection) | Project memberships of the user. (see [Connections](#connections)) | | <a id="usercorepublicemail"></a>`publicEmail` | [`String`](#string) | User's public email. | @@ -15541,6 +15903,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="usercoreassignedmergerequestssourcebranches"></a>`sourceBranches` | [`[String!]`](#string) | Array of source branch names. All resolved merge requests will have one of these branches as their source. | | <a id="usercoreassignedmergerequestsstate"></a>`state` | [`MergeRequestState`](#mergerequeststate) | Merge request state. If provided, all resolved merge requests will have this state. | | <a id="usercoreassignedmergerequeststargetbranches"></a>`targetBranches` | [`[String!]`](#string) | Array of target branch names. All resolved merge requests will have one of these branches as their target. | +| <a id="usercoreassignedmergerequestsupdatedafter"></a>`updatedAfter` | [`Time`](#time) | Merge requests updated after this timestamp. | +| <a id="usercoreassignedmergerequestsupdatedbefore"></a>`updatedBefore` | [`Time`](#time) | Merge requests updated before this timestamp. | ##### `UserCore.authoredMergeRequests` @@ -15573,6 +15937,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="usercoreauthoredmergerequestssourcebranches"></a>`sourceBranches` | [`[String!]`](#string) | Array of source branch names. All resolved merge requests will have one of these branches as their source. | | <a id="usercoreauthoredmergerequestsstate"></a>`state` | [`MergeRequestState`](#mergerequeststate) | Merge request state. If provided, all resolved merge requests will have this state. | | <a id="usercoreauthoredmergerequeststargetbranches"></a>`targetBranches` | [`[String!]`](#string) | Array of target branch names. All resolved merge requests will have one of these branches as their target. | +| <a id="usercoreauthoredmergerequestsupdatedafter"></a>`updatedAfter` | [`Time`](#time) | Merge requests updated after this timestamp. | +| <a id="usercoreauthoredmergerequestsupdatedbefore"></a>`updatedBefore` | [`Time`](#time) | Merge requests updated before this timestamp. | ##### `UserCore.groups` @@ -15622,6 +15988,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="usercorereviewrequestedmergerequestssourcebranches"></a>`sourceBranches` | [`[String!]`](#string) | Array of source branch names. All resolved merge requests will have one of these branches as their source. | | <a id="usercorereviewrequestedmergerequestsstate"></a>`state` | [`MergeRequestState`](#mergerequeststate) | Merge request state. If provided, all resolved merge requests will have this state. | | <a id="usercorereviewrequestedmergerequeststargetbranches"></a>`targetBranches` | [`[String!]`](#string) | Array of target branch names. All resolved merge requests will have one of these branches as their target. | +| <a id="usercorereviewrequestedmergerequestsupdatedafter"></a>`updatedAfter` | [`Time`](#time) | Merge requests updated after this timestamp. | +| <a id="usercorereviewrequestedmergerequestsupdatedbefore"></a>`updatedBefore` | [`Time`](#time) | Merge requests updated before this timestamp. | ##### `UserCore.snippets` @@ -15726,6 +16094,14 @@ fields relate to interactions between the two entities. | ---- | ---- | ----------- | | <a id="userpermissionscreatesnippet"></a>`createSnippet` | [`Boolean!`](#boolean) | Indicates the user can perform `create_snippet` on this resource. | +### `UserPreferences` + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="userpreferencesissuessort"></a>`issuesSort` | [`IssueSort`](#issuesort) | Sort order for issue lists. | + ### `UserStatus` #### Fields @@ -15988,6 +16364,44 @@ Represents the vulnerability details URL field. | <a id="vulnerabilitydetailurlname"></a>`name` | [`String`](#string) | Name of the field. | | <a id="vulnerabilitydetailurltext"></a>`text` | [`String`](#string) | Text of the URL. | +### `VulnerabilityEvidence` + +Represents a Vulnerability Evidence. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="vulnerabilityevidencerequest"></a>`request` | [`VulnerabilityRequest`](#vulnerabilityrequest) | HTTP request of the Vulnerability Evidence. | +| <a id="vulnerabilityevidenceresponse"></a>`response` | [`VulnerabilityResponse`](#vulnerabilityresponse) | HTTP response of the Vulnerability Evidence. | +| <a id="vulnerabilityevidencesource"></a>`source` | [`VulnerabilityEvidenceSource`](#vulnerabilityevidencesource) | Source of the Vulnerability Evidence. | +| <a id="vulnerabilityevidencesummary"></a>`summary` | [`String`](#string) | Summary of the Vulnerability Evidence. | +| <a id="vulnerabilityevidencesupportingmessages"></a>`supportingMessages` | [`[VulnerabilityEvidenceSupportingMessage!]`](#vulnerabilityevidencesupportingmessage) | Supporting messages of the Vulnerability Evidence. | + +### `VulnerabilityEvidenceSource` + +Represents a vulnerability evidence. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="vulnerabilityevidencesourceidentifier"></a>`identifier` | [`String!`](#string) | ID of the Vulnerability Evidence Source. | +| <a id="vulnerabilityevidencesourcename"></a>`name` | [`String!`](#string) | Name of the Vulnerability Evidence Source. | +| <a id="vulnerabilityevidencesourceurl"></a>`url` | [`String`](#string) | URL of the Vulnerability Evidence Source. | + +### `VulnerabilityEvidenceSupportingMessage` + +Represents a vulnerability evidence supporting message. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="vulnerabilityevidencesupportingmessagename"></a>`name` | [`String!`](#string) | Name of the vulnerability supporting message. | +| <a id="vulnerabilityevidencesupportingmessagerequest"></a>`request` | [`VulnerabilityRequest`](#vulnerabilityrequest) | HTTP request of the vulnerability evidence supporting message. | +| <a id="vulnerabilityevidencesupportingmessageresponse"></a>`response` | [`VulnerabilityResponse`](#vulnerabilityresponse) | HTTP response of the vulnerability evidence supporting message. | + ### `VulnerabilityExternalIssueLink` Represents an external issue link of a vulnerability. @@ -16070,8 +16484,11 @@ Represents the location of a vulnerability found by a Coverage Fuzzing scan. | Name | Type | Description | | ---- | ---- | ----------- | | <a id="vulnerabilitylocationcoveragefuzzingblobpath"></a>`blobPath` | [`String`](#string) | Blob path to the vulnerable file. | +| <a id="vulnerabilitylocationcoveragefuzzingcrashaddress"></a>`crashAddress` | [`String`](#string) | Relative address in memory were the crash occurred. | +| <a id="vulnerabilitylocationcoveragefuzzingcrashtype"></a>`crashType` | [`String`](#string) | Type of the crash. | | <a id="vulnerabilitylocationcoveragefuzzingendline"></a>`endLine` | [`String`](#string) | Number of the last relevant line in the vulnerable file. | | <a id="vulnerabilitylocationcoveragefuzzingfile"></a>`file` | [`String`](#string) | Path to the vulnerable file. | +| <a id="vulnerabilitylocationcoveragefuzzingstacktracesnippet"></a>`stacktraceSnippet` | [`String`](#string) | Stack trace recorded during fuzzing resulting the crash. | | <a id="vulnerabilitylocationcoveragefuzzingstartline"></a>`startLine` | [`String`](#string) | Number of the first relevant line in the vulnerable file. | | <a id="vulnerabilitylocationcoveragefuzzingvulnerableclass"></a>`vulnerableClass` | [`String`](#string) | Class containing the vulnerability. | | <a id="vulnerabilitylocationcoveragefuzzingvulnerablemethod"></a>`vulnerableMethod` | [`String`](#string) | Method containing the vulnerability. | @@ -16159,6 +16576,43 @@ Check permissions for the current user on a vulnerability. | <a id="vulnerabilitypermissionsreadvulnerabilityfeedback"></a>`readVulnerabilityFeedback` | [`Boolean!`](#boolean) | Indicates the user can perform `read_vulnerability_feedback` on this resource. | | <a id="vulnerabilitypermissionsupdatevulnerabilityfeedback"></a>`updateVulnerabilityFeedback` | [`Boolean!`](#boolean) | Indicates the user can perform `update_vulnerability_feedback` on this resource. | +### `VulnerabilityRequest` + +Represents a Vulnerability Request. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="vulnerabilityrequestbody"></a>`body` | [`String`](#string) | Body of the Vulnerability Request. | +| <a id="vulnerabilityrequestheaders"></a>`headers` | [`[VulnerabilityRequestResponseHeader!]!`](#vulnerabilityrequestresponseheader) | HTTP headers of the Vulnerability Request. | +| <a id="vulnerabilityrequestmethod"></a>`method` | [`String`](#string) | Method of the Vulnerability Request. | +| <a id="vulnerabilityrequesturl"></a>`url` | [`String`](#string) | URL of the Vulnerability Request. | + +### `VulnerabilityRequestResponseHeader` + +Represents a Vulnerability Request/Response Header. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="vulnerabilityrequestresponseheadername"></a>`name` | [`String`](#string) | Name of the Vulnerability Request/Response Header. | +| <a id="vulnerabilityrequestresponseheadervalue"></a>`value` | [`String`](#string) | Value of the Vulnerability Request/Response Header. | + +### `VulnerabilityResponse` + +Represents a Vulnerability Response. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="vulnerabilityresponsebody"></a>`body` | [`String`](#string) | Body of the Vulnerability Response. | +| <a id="vulnerabilityresponseheaders"></a>`headers` | [`[VulnerabilityRequestResponseHeader!]!`](#vulnerabilityrequestresponseheader) | HTTP headers of the Vulnerability Response. | +| <a id="vulnerabilityresponsereasonphrase"></a>`reasonPhrase` | [`String`](#string) | Reason Phrase of the Vulnerability Response. | +| <a id="vulnerabilityresponsestatuscode"></a>`statusCode` | [`Int`](#int) | Status Code of the Vulnerability Response. | + ### `VulnerabilityScanner` Represents a vulnerability scanner. @@ -16246,6 +16700,7 @@ Represents vulnerability letter grades with associated projects. | <a id="workitemdescriptionhtml"></a>`descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description`. | | <a id="workitemid"></a>`id` | [`WorkItemID!`](#workitemid) | Global ID of the work item. | | <a id="workitemiid"></a>`iid` | [`ID!`](#id) | Internal ID of the work item. | +| <a id="workitemstate"></a>`state` | [`WorkItemState!`](#workitemstate) | State of the work item. | | <a id="workitemtitle"></a>`title` | [`String!`](#string) | Title of the work item. | | <a id="workitemtitlehtml"></a>`titleHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `title`. | | <a id="workitemworkitemtype"></a>`workItemType` | [`WorkItemType!`](#workitemtype) | Type assigned to the work item. | @@ -16481,17 +16936,19 @@ Values for sorting runners. | <a id="cirunnersortcontacted_desc"></a>`CONTACTED_DESC` | Ordered by contacted_at in descending order. | | <a id="cirunnersortcreated_asc"></a>`CREATED_ASC` | Ordered by created_at in ascending order. | | <a id="cirunnersortcreated_desc"></a>`CREATED_DESC` | Ordered by created_at in descending order. | +| <a id="cirunnersorttoken_expires_at_asc"></a>`TOKEN_EXPIRES_AT_ASC` | Ordered by token_expires_at in ascending order. | +| <a id="cirunnersorttoken_expires_at_desc"></a>`TOKEN_EXPIRES_AT_DESC` | Ordered by token_expires_at in descending order. | ### `CiRunnerStatus` | Value | Description | | ----- | ----------- | -| <a id="cirunnerstatusactive"></a>`ACTIVE` **{warning-solid}** | **Deprecated** in 14.6. Use CiRunnerType.active instead. | +| <a id="cirunnerstatusactive"></a>`ACTIVE` **{warning-solid}** | **Deprecated** in 14.6. This was renamed. Use: [`CiRunner.paused`](#cirunnerpaused). | | <a id="cirunnerstatusnever_contacted"></a>`NEVER_CONTACTED` | Runner that has never contacted this instance. Set legacyMode to null to utilize this value. Will replace NOT_CONNECTED starting in 15.0. | | <a id="cirunnerstatusnot_connected"></a>`NOT_CONNECTED` **{warning-solid}** | **Deprecated** in 14.6. Use NEVER_CONTACTED instead. NEVER_CONTACTED will have a slightly different scope starting in 15.0, with STALE being returned instead after 3 months of no contact. | | <a id="cirunnerstatusoffline"></a>`OFFLINE` **{warning-solid}** | **Deprecated** in 14.6. This field will have a slightly different scope starting in 15.0, with STALE being returned after a certain period offline. | | <a id="cirunnerstatusonline"></a>`ONLINE` | Runner that contacted this instance within the last 2 hours. | -| <a id="cirunnerstatuspaused"></a>`PAUSED` **{warning-solid}** | **Deprecated** in 14.6. Use CiRunnerType.active instead. | +| <a id="cirunnerstatuspaused"></a>`PAUSED` **{warning-solid}** | **Deprecated** in 14.6. This was renamed. Use: [`CiRunner.paused`](#cirunnerpaused). | | <a id="cirunnerstatusstale"></a>`STALE` | Runner that has not contacted this instance within the last 3 months. Only available if legacyMode is null. Will be a possible return value starting in 15.0. | ### `CiRunnerType` @@ -16511,6 +16968,7 @@ Values for sorting runners. | <a id="codequalitydegradationseverityinfo"></a>`INFO` | Code Quality degradation has a status of info. | | <a id="codequalitydegradationseveritymajor"></a>`MAJOR` | Code Quality degradation has a status of major. | | <a id="codequalitydegradationseverityminor"></a>`MINOR` | Code Quality degradation has a status of minor. | +| <a id="codequalitydegradationseverityunknown"></a>`UNKNOWN` | Code Quality degradation has a status of unknown. | ### `CommitActionMode` @@ -16776,12 +17234,16 @@ Roadmap sort values. | Value | Description | | ----- | ----------- | +| <a id="epicsortcreated_at_asc"></a>`CREATED_AT_ASC` | Sort by created_at by ascending order. | +| <a id="epicsortcreated_at_desc"></a>`CREATED_AT_DESC` | Sort by created_at by descending order. | | <a id="epicsortend_date_asc"></a>`END_DATE_ASC` | Sort by end date in ascending order. | | <a id="epicsortend_date_desc"></a>`END_DATE_DESC` | Sort by end date in descending order. | | <a id="epicsortstart_date_asc"></a>`START_DATE_ASC` | Sort by start date in ascending order. | | <a id="epicsortstart_date_desc"></a>`START_DATE_DESC` | Sort by start date in descending order. | | <a id="epicsorttitle_asc"></a>`TITLE_ASC` | Sort by title in ascending order. | | <a id="epicsorttitle_desc"></a>`TITLE_DESC` | Sort by title in descending order. | +| <a id="epicsortupdated_at_asc"></a>`UPDATED_AT_ASC` | Sort by updated_at by ascending order. | +| <a id="epicsortupdated_at_desc"></a>`UPDATED_AT_DESC` | Sort by updated_at by descending order. | | <a id="epicsortend_date_asc"></a>`end_date_asc` **{warning-solid}** | **Deprecated** in 13.11. Use END_DATE_ASC. | | <a id="epicsortend_date_desc"></a>`end_date_desc` **{warning-solid}** | **Deprecated** in 13.11. Use END_DATE_DESC. | | <a id="epicsortstart_date_asc"></a>`start_date_asc` **{warning-solid}** | **Deprecated** in 13.11. Use START_DATE_ASC. | @@ -16992,6 +17454,23 @@ Issue type. | <a id="issuetyperequirement"></a>`REQUIREMENT` | Requirement issue type. | | <a id="issuetypetest_case"></a>`TEST_CASE` | Test Case issue type. | +### `IterationSearchableField` + +Fields to perform the search in. + +| Value | Description | +| ----- | ----------- | +| <a id="iterationsearchablefieldcadence_title"></a>`CADENCE_TITLE` | Search in cadence_title field. | +| <a id="iterationsearchablefieldtitle"></a>`TITLE` | Search in title field. | + +### `IterationSort` + +Iteration sort values. + +| Value | Description | +| ----- | ----------- | +| <a id="iterationsortcadence_and_due_date_asc"></a>`CADENCE_AND_DUE_DATE_ASC` | Sort by cadence id and due date in ascending order. | + ### `IterationState` State of a GitLab iteration. @@ -17111,6 +17590,8 @@ Values for sorting merge requests. | <a id="mergerequestsortmilestone_due_desc"></a>`MILESTONE_DUE_DESC` | Milestone due date by descending order. | | <a id="mergerequestsortpriority_asc"></a>`PRIORITY_ASC` | Priority by ascending order. | | <a id="mergerequestsortpriority_desc"></a>`PRIORITY_DESC` | Priority by descending order. | +| <a id="mergerequestsorttitle_asc"></a>`TITLE_ASC` | Title by ascending order. | +| <a id="mergerequestsorttitle_desc"></a>`TITLE_DESC` | Title by descending order. | | <a id="mergerequestsortupdated_asc"></a>`UPDATED_ASC` | Updated at ascending order. | | <a id="mergerequestsortupdated_desc"></a>`UPDATED_DESC` | Updated at descending order. | | <a id="mergerequestsortcreated_asc"></a>`created_asc` **{warning-solid}** | **Deprecated** in 13.5. This was renamed. Use: `CREATED_ASC`. | @@ -17663,6 +18144,7 @@ Name of the feature that the callout is for. | <a id="usercalloutfeaturenameenumsecurity_configuration_devops_alert"></a>`SECURITY_CONFIGURATION_DEVOPS_ALERT` | Callout feature name for security_configuration_devops_alert. | | <a id="usercalloutfeaturenameenumsecurity_configuration_upgrade_banner"></a>`SECURITY_CONFIGURATION_UPGRADE_BANNER` | Callout feature name for security_configuration_upgrade_banner. | | <a id="usercalloutfeaturenameenumsecurity_newsletter_callout"></a>`SECURITY_NEWSLETTER_CALLOUT` | Callout feature name for security_newsletter_callout. | +| <a id="usercalloutfeaturenameenumsecurity_training_feature_promotion"></a>`SECURITY_TRAINING_FEATURE_PROMOTION` | Callout feature name for security_training_feature_promotion. | | <a id="usercalloutfeaturenameenumsuggest_pipeline"></a>`SUGGEST_PIPELINE` | Callout feature name for suggest_pipeline. | | <a id="usercalloutfeaturenameenumsuggest_popover_dismissed"></a>`SUGGEST_POPOVER_DISMISSED` | Callout feature name for suggest_popover_dismissed. | | <a id="usercalloutfeaturenameenumtabs_position_highlight"></a>`TABS_POSITION_HIGHLIGHT` | Callout feature name for tabs_position_highlight. | @@ -17832,6 +18314,24 @@ Weight ID wildcard values. | <a id="weightwildcardidany"></a>`ANY` | Weight is assigned. | | <a id="weightwildcardidnone"></a>`NONE` | No weight is assigned. | +### `WorkItemState` + +State of a GitLab work item. + +| Value | Description | +| ----- | ----------- | +| <a id="workitemstateclosed"></a>`CLOSED` | In closed state. | +| <a id="workitemstateopen"></a>`OPEN` | In open state. | + +### `WorkItemStateEvent` + +Values for work item state events. + +| Value | Description | +| ----- | ----------- | +| <a id="workitemstateeventclose"></a>`CLOSE` | Closes the work item. | +| <a id="workitemstateeventreopen"></a>`REOPEN` | Reopens the work item. | + ## Scalar types Scalar values are atomic values, and do not have fields of their own. @@ -18308,6 +18808,12 @@ A `ReleasesLinkID` is a global ID. It is encoded as a string. An example `ReleasesLinkID` is: `"gid://gitlab/Releases::Link/1"`. +### `SecurityTrainingProviderID` + +A `SecurityTrainingProviderID` is a global ID. It is encoded as a string. + +An example `SecurityTrainingProviderID` is: `"gid://gitlab/Security::TrainingProvider/1"`. + ### `SnippetID` A `SnippetID` is a global ID. It is encoded as a string. @@ -18426,6 +18932,7 @@ One of: - [`Epic`](#epic) - [`Issue`](#issue) - [`MergeRequest`](#mergerequest) +- [`WorkItem`](#workitem) #### `JobNeedUnion` @@ -18614,6 +19121,20 @@ Implementations: | <a id="memberinterfaceupdatedat"></a>`updatedAt` | [`Time`](#time) | Date and time the membership was last updated. | | <a id="memberinterfaceuser"></a>`user` | [`UserCore`](#usercore) | User that is associated with the member object. | +##### Fields with arguments + +###### `MemberInterface.mergeRequestInteraction` + +Find a merge request. + +Returns [`UserMergeRequestInteraction`](#usermergerequestinteraction). + +####### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="memberinterfacemergerequestinteractionid"></a>`id` | [`MergeRequestID!`](#mergerequestid) | Global ID of the merge request. | + #### `NoteableInterface` Implementations: @@ -18733,7 +19254,7 @@ Implementations: | <a id="usergroupmemberships"></a>`groupMemberships` | [`GroupMemberConnection`](#groupmemberconnection) | Group memberships of the user. (see [Connections](#connections)) | | <a id="userid"></a>`id` | [`ID!`](#id) | ID of the user. | | <a id="userlocation"></a>`location` | [`String`](#string) | Location of the user. | -| <a id="username"></a>`name` | [`String!`](#string) | Human-readable name of the user. Will return `****` if the user is a project bot and the requester does not have permission to read resource access tokens. | +| <a id="username"></a>`name` | [`String!`](#string) | Human-readable name of the user. Returns `****` if the user is a project bot and the requester does not have permission to view the project. | | <a id="usernamespace"></a>`namespace` | [`Namespace`](#namespace) | Personal namespace of the user. | | <a id="userprojectmemberships"></a>`projectMemberships` | [`ProjectMemberConnection`](#projectmemberconnection) | Project memberships of the user. (see [Connections](#connections)) | | <a id="userpublicemail"></a>`publicEmail` | [`String`](#string) | User's public email. | @@ -18777,6 +19298,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="userassignedmergerequestssourcebranches"></a>`sourceBranches` | [`[String!]`](#string) | Array of source branch names. All resolved merge requests will have one of these branches as their source. | | <a id="userassignedmergerequestsstate"></a>`state` | [`MergeRequestState`](#mergerequeststate) | Merge request state. If provided, all resolved merge requests will have this state. | | <a id="userassignedmergerequeststargetbranches"></a>`targetBranches` | [`[String!]`](#string) | Array of target branch names. All resolved merge requests will have one of these branches as their target. | +| <a id="userassignedmergerequestsupdatedafter"></a>`updatedAfter` | [`Time`](#time) | Merge requests updated after this timestamp. | +| <a id="userassignedmergerequestsupdatedbefore"></a>`updatedBefore` | [`Time`](#time) | Merge requests updated before this timestamp. | ###### `User.authoredMergeRequests` @@ -18809,6 +19332,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="userauthoredmergerequestssourcebranches"></a>`sourceBranches` | [`[String!]`](#string) | Array of source branch names. All resolved merge requests will have one of these branches as their source. | | <a id="userauthoredmergerequestsstate"></a>`state` | [`MergeRequestState`](#mergerequeststate) | Merge request state. If provided, all resolved merge requests will have this state. | | <a id="userauthoredmergerequeststargetbranches"></a>`targetBranches` | [`[String!]`](#string) | Array of target branch names. All resolved merge requests will have one of these branches as their target. | +| <a id="userauthoredmergerequestsupdatedafter"></a>`updatedAfter` | [`Time`](#time) | Merge requests updated after this timestamp. | +| <a id="userauthoredmergerequestsupdatedbefore"></a>`updatedBefore` | [`Time`](#time) | Merge requests updated before this timestamp. | ###### `User.groups` @@ -18858,6 +19383,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="userreviewrequestedmergerequestssourcebranches"></a>`sourceBranches` | [`[String!]`](#string) | Array of source branch names. All resolved merge requests will have one of these branches as their source. | | <a id="userreviewrequestedmergerequestsstate"></a>`state` | [`MergeRequestState`](#mergerequeststate) | Merge request state. If provided, all resolved merge requests will have this state. | | <a id="userreviewrequestedmergerequeststargetbranches"></a>`targetBranches` | [`[String!]`](#string) | Array of target branch names. All resolved merge requests will have one of these branches as their target. | +| <a id="userreviewrequestedmergerequestsupdatedafter"></a>`updatedAfter` | [`Time`](#time) | Merge requests updated after this timestamp. | +| <a id="userreviewrequestedmergerequestsupdatedbefore"></a>`updatedBefore` | [`Time`](#time) | Merge requests updated before this timestamp. | ###### `User.snippets` diff --git a/doc/api/group_access_tokens.md b/doc/api/group_access_tokens.md index 37471b9d89d..45366885c5c 100644 --- a/doc/api/group_access_tokens.md +++ b/doc/api/group_access_tokens.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -58,7 +58,7 @@ POST groups/:id/access_tokens |-----------|---------|----------|---------------------| | `id` | integer or string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) | | `name` | String | yes | The name of the group access token | -| `scopes` | `Array[String]` | yes | [List of scopes](../user/project/settings/project_access_tokens.md#scopes-for-a-project-access-token) | +| `scopes` | `Array[String]` | yes | [List of scopes](../user/group/settings/group_access_tokens.md#scopes-for-a-group-access-token) | | `access_level` | Integer | no | A valid access level. Default value is 40 (Maintainer). Other allowed values are 10 (Guest), 20 (Reporter), and 30 (Developer). | | `expires_at` | Date | no | The token expires at midnight UTC on that date | diff --git a/doc/api/group_badges.md b/doc/api/group_badges.md index ecb73aa8a5e..360790daf8c 100644 --- a/doc/api/group_badges.md +++ b/doc/api/group_badges.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/api/group_clusters.md b/doc/api/group_clusters.md index eaecc74a96e..87829708d5e 100644 --- a/doc/api/group_clusters.md +++ b/doc/api/group_clusters.md @@ -17,7 +17,7 @@ Similarly to [project-level](../user/project/clusters/index.md) and group-level Kubernetes clusters allow you to connect a Kubernetes cluster to your group, enabling you to use the same cluster across multiple projects. -Users need at least the [Maintainer role](../user/permissions.md) for the group to use these endpoints. +Users need at least the Maintainer role for the group to use these endpoints. ## List group clusters diff --git a/doc/api/groups.md b/doc/api/groups.md index d7b4f0c8b54..2e0f9526e16 100644 --- a/doc/api/groups.md +++ b/doc/api/groups.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -808,7 +808,7 @@ Parameters: ### Options for `default_branch_protection` -The `default_branch_protection` attribute determines whether users with the Developer or [Maintainer role](../user/permissions.md) can push to the applicable [default branch](../user/project/repository/branches/default.md), as described in the following table: +The `default_branch_protection` attribute determines whether users with the Developer or Maintainer role can push to the applicable [default branch](../user/project/repository/branches/default.md), as described in the following table: | Value | Description | |-------|-------------------------------------------------------------------------------------------------------------| @@ -832,7 +832,8 @@ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ ## Transfer project to group -Transfer a project to the Group namespace. Available only to instance administrators, although an [alternative API endpoint](projects.md#transfer-a-project-to-a-new-namespace) is available which does not require instance administrator role. Transferring projects may fail when tagged packages exist in the project's repository. +Transfer a project to the Group namespace. Available only to instance administrators, although an [alternative API endpoint](projects.md#transfer-a-project-to-a-new-namespace) +is available which does not require administrator access on the instance. Transferring projects may fail when tagged packages exist in the project's repository. ```plaintext POST /groups/:id/projects/:project_id @@ -1089,6 +1090,79 @@ GET /groups?search=foobar ] ``` +## List provisioned users **(PREMIUM)** + +> Introduced in GitLab 14.8. + +Get a list of users provisioned by a given group. Does not include users provisioned by subgroups. + +Requires at least the Maintainer role on the group. + +```plaintext +GET /groups/:id/provisioned_users +``` + +Parameters: + +| Attribute | Type | Required | Description | +|:-----------------|:---------------|:---------|:-------------------------------------------------------------------------| +| `id` | integer/string | yes | ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) | +| `username` | string | no | Return single user with a specific username | +| `search` | string | no | Search users by name, email, username | +| `active` | boolean | no | Return only active users | +| `blocked` | boolean | no | Return only blocked users | +| `created_after` | datetime | no | Return users created after the specified time | +| `created_before` | datetime | no | Return users created before the specified time | + +Example response: + +```json +[ + { + id: 66, + username: "user22", + name: "John Doe22", + state: "active", + avatar_url: "https://www.gravatar.com/avatar/xxx?s=80&d=identicon", + web_url: "http://my.gitlab.com/user22", + created_at: "2021-09-10T12:48:22.381Z", + bio: "", + location: null, + public_email: "", + skype: "", + linkedin: "", + twitter: "", + website_url: "", + organization: null, + job_title: "", + pronouns: null, + bot: false, + work_information: null, + followers: 0, + following: 0, + local_time: null, + last_sign_in_at: null, + confirmed_at: "2021-09-10T12:48:22.330Z", + last_activity_on: null, + email: "user22@example.org", + theme_id: 1, + color_scheme_id: 1, + projects_limit: 100000, + current_sign_in_at: null, + identities: [ ], + can_create_group: true, + can_create_project: true, + two_factor_enabled: false, + external: false, + private_profile: false, + commit_email: "user22@example.org", + shared_runners_minutes_limit: null, + extra_shared_runners_minutes_limit: null + }, + ... +] +``` + ## Hooks **(PREMIUM)** Also called Group Hooks and Webhooks. diff --git a/doc/api/index.md b/doc/api/index.md index 69db971f58c..589bc0416a1 100644 --- a/doc/api/index.md +++ b/doc/api/index.md @@ -12,6 +12,9 @@ Use the GitLab APIs to automate GitLab. A REST API is available in GitLab. Usage instructions are below. + +For examples, see [How to use the API](#how-to-use-the-api). + For a list of the available resources and their endpoints, see [REST API resources](api_resources.md). @@ -31,7 +34,9 @@ GitLab provides an [SCIM API](scim.md) that both implements ## GraphQL API -A [GraphQL API](graphql/index.md) is available in GitLab. +A GraphQL API is available in GitLab. +For a list of the available resources and their endpoints, see +[GraphQL API resources](graphql/reference/index.md). With GraphQL, you can make an API request for only what you need, and it's versioned by default. @@ -60,7 +65,7 @@ month. Major API version changes, and removal of entire API versions, are done i with major GitLab releases. All deprecations and changes between versions are in the documentation. -For the changes between v3 and v4, see the [v3 to v4 documentation](v3_to_v4.md). +For the changes between v3 and v4, see the [v3 to v4 documentation](https://gitlab.com/gitlab-org/gitlab-foss/-/blob/11-0-stable/doc/api/v3_to_v4.md). ### Current status @@ -74,7 +79,7 @@ For example, the root of the v4 API is at `/api/v4`. ### Valid API request -If you have a GitLab instance at `gitlab.example.com`: +The following is a basic example of a request to the fictional `gitlab.example.com` endpoint: ```shell curl "https://gitlab.example.com/api/v4/projects" @@ -83,6 +88,10 @@ curl "https://gitlab.example.com/api/v4/projects" The API uses JSON to serialize data. You don't need to specify `.json` at the end of the API URL. +NOTE: +In the example above, replace `gitlab.example.com` with `gitlab.com` to query GitLab.com (GitLab SaaS). +Access can be denied due to authentication. For more information, see [Authentication](#authentication). + ### API request to expose HTTP response headers If you want to expose HTTP response headers, use the `--include` option: diff --git a/doc/api/instance_clusters.md b/doc/api/instance_clusters.md index f29ac5cd7f2..ab631757eab 100644 --- a/doc/api/instance_clusters.md +++ b/doc/api/instance_clusters.md @@ -16,8 +16,7 @@ With [instance-level Kubernetes clusters](../user/instance/clusters/index.md), you can connect a Kubernetes cluster to the GitLab instance and use the same cluster across all of the projects within your instance. -NOTE: -Users need the Administrator role to use these endpoints. +Users need administrator access to use these endpoints. ## List instance clusters diff --git a/doc/api/integrations.md b/doc/api/integrations.md index 4719fe69c7b..90bb26ffd3d 100644 --- a/doc/api/integrations.md +++ b/doc/api/integrations.md @@ -13,7 +13,7 @@ In GitLab 14.4, the `services` endpoint was [renamed](https://gitlab.com/gitlab- Calls to the Integrations API can be made to both `/projects/:id/services` and `/projects/:id/integrations`. The examples in this document refer to the endpoint at `/projects/:id/integrations`. -This API requires an access token with the [Maintainer or Owner role](../user/permissions.md). +This API requires an access token with the Maintainer or Owner role. ## List all active integrations @@ -319,11 +319,12 @@ Parameters: | ---------------------- | ------- | -------- | ----------- | | `api_key` | string | true | API key used for authentication with Datadog | | `api_url` | string | false | (Advanced) The full URL for your Datadog site | -<!-- | `archive_trace_events` | boolean | false | When enabled, job logs are collected by Datadog and displayed along with pipeline execution traces ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/346339) in GitLab 14.7) | --> -<!-- TODO: uncomment the archive_trace_events field once :datadog_integration_logs_collection is rolled out. Rollout issue: https://gitlab.com/gitlab-org/gitlab/-/issues/346339 --> | `datadog_env` | string | false | For self-managed deployments, set the env% tag for all the data sent to Datadog. | | `datadog_service` | string | false | Tag all data from this GitLab instance in Datadog. Useful when managing several self-managed deployments | | `datadog_site` | string | false | The Datadog site to send data to. To send data to the EU site, use `datadoghq.eu` | +| `datadog_tags` | string | false | Custom tags in Datadog. Specify one tag per line in the format: `key:value\nkey2:value2` ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79665) in GitLab 14.8.) | +<!-- | `archive_trace_events` | boolean | false | When enabled, job logs are collected by Datadog and displayed along with pipeline execution traces ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/346339) in GitLab 14.7) | --> +<!-- TODO: uncomment the archive_trace_events field once :datadog_integration_logs_collection is rolled out. Rollout issue: https://gitlab.com/gitlab-org/gitlab/-/issues/346339 --> ### Disable Datadog integration diff --git a/doc/api/issues.md b/doc/api/issues.md index 5d22952a876..5801f072062 100644 --- a/doc/api/issues.md +++ b/doc/api/issues.md @@ -29,7 +29,9 @@ When requested across groups or projects, it's expected to be the same as the `f ## List issues +> The `due_date` property was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/233420) in GitLab 13.3. > The `weight` property moved to GitLab Premium in 13.9. +> The `due_date` filters `any`, `today`, and `tomorrow` were [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78460) in GitLab 14.8. Get all issues the authenticated user has access to. By default it returns only issues created by the current user. To get all issues, @@ -61,7 +63,7 @@ GET /issues?state=opened | `confidential` | boolean | no | Filter confidential or public issues. | | `created_after` | datetime | no | Return issues created on or after the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | | `created_before` | datetime | no | Return issues created on or before the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | -| `due_date` | string | no | Return issues that have no due date, are overdue, or whose due date is this week, this month, or between two weeks ago and next month. Accepts: `0` (no due date), `overdue`, `week`, `month`, `next_month_and_previous_two_weeks`. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/233420) in GitLab 13.3)_ | +| `due_date` | string | no | Return issues that have no due date, are overdue, or whose due date is this week, this month, or between two weeks ago and next month. Accepts: `0` (no due date), `any`, `today`, `tomorrow`, `overdue`, `week`, `month`, `next_month_and_previous_two_weeks`. | | `epic_id` **(PREMIUM)** | integer | no | Return issues associated with the given epic ID. `None` returns issues that are not associated with an epic. `Any` returns issues that are associated with an epic. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/46887) in GitLab 13.6)_ | `iids[]` | integer array | no | Return only the issues having the given `iid` | | `in` | string | no | Modify the scope of the `search` attribute. `title`, `description`, or a string joining them with comma. Default is `title,description` | @@ -231,7 +233,9 @@ Please use `iid` of the `epic` attribute instead. ## List group issues +> The `due_date` property was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/233420) in GitLab 13.3. > The `weight` property moved to GitLab Premium in 13.9. +> The `due_date` filters `any`, `today`, and `tomorrow` were [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78460) in GitLab 14.8. Get a list of a group's issues. @@ -264,7 +268,7 @@ GET /groups/:id/issues?state=opened | `confidential` | boolean | no | Filter confidential or public issues. | | `created_after` | datetime | no | Return issues created on or after the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | | `created_before` | datetime | no | Return issues created on or before the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | -| `due_date` | string | no | Return issues that have no due date, are overdue, or whose due date is this week, this month, or between two weeks ago and next month. Accepts: `0` (no due date), `overdue`, `week`, `month`, `next_month_and_previous_two_weeks`. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/233420) in GitLab 13.3)_ | +| `due_date` | string | no | Return issues that have no due date, are overdue, or whose due date is this week, this month, or between two weeks ago and next month. Accepts: `0` (no due date), `any`, `today`, `tomorrow`, `overdue`, `week`, `month`, `next_month_and_previous_two_weeks`. | | `epic_id` **(PREMIUM)** | integer | no | Return issues associated with the given epic ID. `None` returns issues that are not associated with an epic. `Any` returns issues that are associated with an epic. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/46887) in GitLab 13.6)_ | `id` | integer/string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) owned by the authenticated user | | `iids[]` | integer array | no | Return only the issues having the given `iid` | @@ -431,7 +435,9 @@ Please use `iid` of the `epic` attribute instead. ## List project issues +> The `due_date` property was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/233420) in GitLab 13.3. > The `weight` property moved to GitLab Premium in 13.9. +> The `due_date` filters `any`, `today`, and `tomorrow` were [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78460) in GitLab 14.8. Get a list of a project's issues. @@ -464,7 +470,7 @@ GET /projects/:id/issues?state=opened | `confidential` | boolean | no | Filter confidential or public issues. | | `created_after` | datetime | no | Return issues created on or after the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | | `created_before` | datetime | no | Return issues created on or before the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | -| `due_date` | string | no | Return issues that have no due date, are overdue, or whose due date is this week, this month, or between two weeks ago and next month. Accepts: `0` (no due date), `overdue`, `week`, `month`, `next_month_and_previous_two_weeks`. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/233420) in GitLab 13.3)_ | +| `due_date` | string | no | Return issues that have no due date, are overdue, or whose due date is this week, this month, or between two weeks ago and next month. Accepts: `0` (no due date), `any`, `today`, `tomorrow`, `overdue`, `week`, `month`, `next_month_and_previous_two_weeks`. | | `epic_id` **(PREMIUM)** | integer | no | Return issues associated with the given epic ID. `None` returns issues that are not associated with an epic. `Any` returns issues that are associated with an epic. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/46887) in GitLab 13.6)_ | `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | | `iids[]` | integer array | no | Return only the issues having the given `iid` | @@ -2422,10 +2428,11 @@ POST /projects/:id/issues/:issue_iid/metric_images | `issue_iid` | integer | yes | The internal ID of a project's issue | | `file` | file | yes | The image file to be uploaded | | `url` | string | no | The URL to view more metric information | +| `url_text` | string | no | A description of the image or URL | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" --form 'file=@/path/to/file.png' \ ---form 'url=http://example.com' "https://gitlab.example.com/api/v4/projects/5/issues/93/metric_images" +--form 'url=http://example.com' --form 'url_text=Example website' "https://gitlab.example.com/api/v4/projects/5/issues/93/metric_images" ``` Example response: @@ -2436,7 +2443,8 @@ Example response: "created_at": "2020-11-13T00:06:18.084Z", "filename": "file.png", "file_path": "/uploads/-/system/issuable_metric_image/file/23/file.png", - "url": "http://example.com" + "url": "http://example.com", + "url_text": "Example website" } ``` @@ -2478,6 +2486,39 @@ Example response: ] ``` +## Update metric image + +Available only for Incident issues. + +```plaintext +PUT /projects/:id/issues/:issue_iid/metric_images/:image_id +``` + +| Attribute | Type | Required | Description | +|-------------|---------|----------|--------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `issue_iid` | integer | yes | The internal ID of a project's issue | +| `image_id` | integer | yes | The ID of the image | +| `url` | string | no | The URL to view more metric information | +| `url_text` | string | no | A description of the image or URL | + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" --request PUT --form 'url=http://example.com' --form 'url_text=Example website' "https://gitlab.example.com/api/v4/projects/5/issues/93/metric_images/1" +``` + +Example response: + +```json +{ + "id": 23, + "created_at": "2020-11-13T00:06:18.084Z", + "filename": "file.png", + "file_path": "/uploads/-/system/issuable_metric_image/file/23/file.png", + "url": "http://example.com", + "url_text": "Example website" +} +``` + ## Delete metric image Available only for Incident issues. diff --git a/doc/api/job_artifacts.md b/doc/api/job_artifacts.md index a874379506f..d272f259ddf 100644 --- a/doc/api/job_artifacts.md +++ b/doc/api/job_artifacts.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Testing +group: Pipeline Insights info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -293,7 +293,9 @@ FLAG: On self-managed GitLab, by default this feature is available. To hide the feature, ask an administrator to [disable the `bulk_expire_project_artifacts` flag](../administration/feature_flags.md). On GitLab.com, this feature is available. -[Expire artifacts of a project that can be deleted](https://gitlab.com/gitlab-org/gitlab/-/issues/223793) but that don't have an expiry time. +Delete artifacts of a project that can be deleted. + +By default, [artifacts from the most recent successful pipeline of each ref are kept](../ci/pipelines/job_artifacts.md#keep-artifacts-from-most-recent-successful-jobs). ```plaintext DELETE /projects/:id/artifacts diff --git a/doc/api/jobs.md b/doc/api/jobs.md index 8dcd898b8c3..89018548f5f 100644 --- a/doc/api/jobs.md +++ b/doc/api/jobs.md @@ -10,14 +10,16 @@ info: To determine the technical writer assigned to the Stage/Group associated w Get a list of jobs in a project. Jobs are sorted in descending order of their IDs. +By default, this request returns 20 results at a time because the API results [are paginated](index.md#pagination) + ```plaintext GET /projects/:id/jobs ``` -| Attribute | Type | Required | Description | -|-----------|--------------------------------|----------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `id` | integer/string | yes | ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | -| `scope` | string **or** array of strings | no | Scope of jobs to show. Either one of or an array of the following: `created`, `pending`, `running`, `failed`, `success`, `canceled`, `skipped`, or `manual`. All jobs are returned if `scope` is not provided. | +| Attribute | Type | Required | Description | +|-----------|--------------------------------|------------------------|-------------| +| `id` | integer/string | **{check-circle}** Yes | ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | +| `scope` | string **or** array of strings | **{dotted-circle}** No | Scope of jobs to show. Either one of or an array of the following: `created`, `pending`, `running`, `failed`, `success`, `canceled`, `skipped`, or `manual`. All jobs are returned if `scope` is not provided. | ```shell curl --globoff --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/jobs?scope[]=pending&scope[]=running" @@ -155,16 +157,18 @@ Example of response Get a list of jobs for a pipeline. +By default, this request returns 20 results at a time because the API results [are paginated](index.md#pagination) + ```plaintext GET /projects/:id/pipelines/:pipeline_id/jobs ``` -| Attribute | Type | Required | Description | -|-------------------|--------------------------------|----------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `id` | integer/string | yes | ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | -| `pipeline_id` | integer | yes | ID of a pipeline. Can also be obtained in CI jobs via the [predefined CI variable](../ci/variables/predefined_variables.md) `CI_PIPELINE_ID`. | -| `scope` | string **or** array of strings | no | Scope of jobs to show. Either one of or an array of the following: `created`, `pending`, `running`, `failed`, `success`, `canceled`, `skipped`, or `manual`. All jobs are returned if `scope` is not provided. | -| `include_retried` | boolean | no | Include retried jobs in the response. Defaults to `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/272627) in GitLab 13.9. | +| Attribute | Type | Required | Description | +|-------------------|--------------------------------|------------------------|-------------| +| `id` | integer/string | **{check-circle}** Yes | ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | +| `pipeline_id` | integer | **{check-circle}** Yes | ID of a pipeline. Can also be obtained in CI jobs via the [predefined CI variable](../ci/variables/predefined_variables.md) `CI_PIPELINE_ID`. | +| `scope` | string **or** array of strings | **{dotted-circle}** No | Scope of jobs to show. Either one of or an array of the following: `created`, `pending`, `running`, `failed`, `success`, `canceled`, `skipped`, or `manual`. All jobs are returned if `scope` is not provided. | +| `include_retried` | boolean | **{dotted-circle}** No | Include retried jobs in the response. Defaults to `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/272627) in GitLab 13.9. | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/pipelines/6/jobs?scope[]=pending&scope[]=running" @@ -316,11 +320,11 @@ Get a list of bridge jobs for a pipeline. GET /projects/:id/pipelines/:pipeline_id/bridges ``` -| Attribute | Type | Required | Description | -|---------------|--------------------------------|----------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `id` | integer/string | yes | ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | -| `pipeline_id` | integer | yes | ID of a pipeline. | -| `scope` | string **or** array of strings | no | Scope of jobs to show. Either one of or an array of the following: `created`, `pending`, `running`, `failed`, `success`, `canceled`, `skipped`, or `manual`. All jobs are returned if `scope` is not provided. | +| Attribute | Type | Required | Description | +|---------------|--------------------------------|------------------------|-------------| +| `id` | integer/string | **{check-circle}** Yes | ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | +| `pipeline_id` | integer | **{check-circle}** Yes | ID of a pipeline. | +| `scope` | string **or** array of strings | **{dotted-circle}** No | Scope of jobs to show. Either one of or an array of the following: `created`, `pending`, `running`, `failed`, `success`, `canceled`, `skipped`, or `manual`. All jobs are returned if `scope` is not provided. | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/pipelines/6/bridges?scope[]=pending&scope[]=running" @@ -483,9 +487,9 @@ GET /job/allowed_agents Supported attributes: -| Attribute | Type | Required | Description | -|:------------ |:---------|:---------|:----------------------| -| `CI_JOB_TOKEN` | string | yes | Token value associated with the GitLab-provided `CI_JOB_TOKEN` variable. | +| Attribute | Type | Required | Description | +|----------------|----------|------------------------|-------------| +| `CI_JOB_TOKEN` | string | **{check-circle}** Yes | Token value associated with the GitLab-provided `CI_JOB_TOKEN` variable. | Example request: @@ -558,10 +562,10 @@ Get a single job of a project GET /projects/:id/jobs/:job_id ``` -| Attribute | Type | Required | Description | -|-----------|----------------|----------|------------------------------------------------------------------------------------------------------------------| -| `id` | integer/string | yes | ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | -| `job_id` | integer | yes | ID of a job. | +| Attribute | Type | Required | Description | +|-----------|----------------|------------------------|-------------| +| `id` | integer/string | **{check-circle}** Yes | ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | +| `job_id` | integer | **{check-circle}** Yes | ID of a job. | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/jobs/8" @@ -635,10 +639,10 @@ Get a log (trace) of a specific job of a project: GET /projects/:id/jobs/:job_id/trace ``` -| Attribute | Type | Required | Description | -|-----------|----------------|----------|------------------------------------------------------------------------------------------------------------------| -| `id` | integer/string | yes | ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | -| `job_id` | integer | yes | ID of a job. | +| Attribute | Type | Required | Description | +|-----------|----------------|------------------------|-------------| +| `id` | integer/string | **{check-circle}** Yes | ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | +| `job_id` | integer | **{check-circle}** Yes | ID of a job. | ```shell curl --location --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/jobs/8/trace" @@ -659,10 +663,10 @@ Cancel a single job of a project POST /projects/:id/jobs/:job_id/cancel ``` -| Attribute | Type | Required | Description | -|-----------|----------------|----------|------------------------------------------------------------------------------------------------------------------| -| `id` | integer/string | yes | ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | -| `job_id` | integer | yes | ID of a job. | +| Attribute | Type | Required | Description | +|-----------|----------------|------------------------|-------------| +| `id` | integer/string | **{check-circle}** Yes | ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | +| `job_id` | integer | **{check-circle}** Yes | ID of a job. | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/jobs/1/cancel" @@ -709,10 +713,10 @@ Retry a single job of a project POST /projects/:id/jobs/:job_id/retry ``` -| Attribute | Type | Required | Description | -|-----------|----------------|----------|------------------------------------------------------------------------------------------------------------------| -| `id` | integer/string | yes | ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | -| `job_id` | integer | yes | ID of a job. | +| Attribute | Type | Required | Description | +|-----------|----------------|------------------------|-------------| +| `id` | integer/string | **{check-circle}** Yes | ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | +| `job_id` | integer | **{check-circle}** Yes | ID of a job. | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/jobs/1/retry" @@ -761,10 +765,10 @@ POST /projects/:id/jobs/:job_id/erase Parameters -| Attribute | Type | Required | Description | -|-----------|----------------|----------|------------------------------------------------------------------------------------------------------------------| -| `id` | integer/string | yes | ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | -| `job_id` | integer | yes | ID of a job. | +| Attribute | Type | Required | Description | +|-----------|----------------|------------------------|-------------| +| `id` | integer/string | **{check-circle}** Yes | ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | +| `job_id` | integer | **{check-circle}** Yes | ID of a job. | Example of request @@ -818,10 +822,10 @@ Triggers a manual action to start a job. POST /projects/:id/jobs/:job_id/play ``` -| Attribute | Type | Required | Description | -|-----------|----------------|----------|------------------------------------------------------------------------------------------------------------------| -| `id` | integer/string | yes | ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | -| `job_id` | integer | yes | ID of a job. | +| Attribute | Type | Required | Description | +|-----------|----------------|------------------------|-------------| +| `id` | integer/string | **{check-circle}** Yes | ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | +| `job_id` | integer | **{check-circle}** Yes | ID of a job. | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/jobs/1/play" diff --git a/doc/api/lint.md b/doc/api/lint.md index e5b5e0e2be8..a271b75c035 100644 --- a/doc/api/lint.md +++ b/doc/api/lint.md @@ -89,7 +89,7 @@ Example responses: The CI lint returns an expanded version of the configuration. The expansion does not work for CI configuration added with [`include: local`](../ci/yaml/index.md#includelocal), -or with [`extends:`](../ci/yaml/index.md#extends). +and the [`extends:`](../ci/yaml/index.md#extends) keyword is [not fully supported](https://gitlab.com/gitlab-org/gitlab/-/issues/258843). Example contents of a `.gitlab-ci.yml` passed to the CI Lint API with `include_merged_yaml` and `include_jobs` set as true: @@ -169,8 +169,9 @@ POST /projects/:id/ci/lint | Attribute | Type | Required | Description | | ---------- | ------- | -------- | -------- | | `content` | string | yes | The CI/CD configuration content. | -| `dry_run` | boolean | no | Run [pipeline creation simulation](../ci/lint.md#pipeline-simulation), or only do static check. This is false by default. | +| `dry_run` | boolean | no | Run [pipeline creation simulation](../ci/lint.md#simulate-a-pipeline), or only do static check. This is false by default. | | `include_jobs` | boolean | no | If the list of jobs that would exist in a static check or pipeline simulation should be included in the response. This is false by default. | +| `ref` | string | no | When `dry_run` is `true`, sets the branch or tag to use. Defaults to the project's default branch when not set. | Example request: @@ -220,6 +221,7 @@ GET /projects/:id/ci/lint | ---------- | ------- | -------- | -------- | | `dry_run` | boolean | no | Run pipeline creation simulation, or only do static check. | | `include_jobs` | boolean | no | If the list of jobs that would exist in a static check or pipeline simulation should be included in the response. This is false by default. | +| `ref` | string | no | When `dry_run` is `true`, sets the branch or tag to use. Defaults to the project's default branch when not set. | Example request: diff --git a/doc/api/members.md b/doc/api/members.md index 95565d40897..10072baf2ff 100644 --- a/doc/api/members.md +++ b/doc/api/members.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/api/merge_request_approvals.md b/doc/api/merge_request_approvals.md index b6021d494fd..6a0b66ac5dc 100644 --- a/doc/api/merge_request_approvals.md +++ b/doc/api/merge_request_approvals.md @@ -278,6 +278,12 @@ GET /projects/:id/approval_rules/:approval_rule_id > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11877) in GitLab 12.3. > - Moved to GitLab Premium in 13.9. +WARNING: +The Vulnerability-Check feature, including the Vulnerability-Check attributes listed here, is in its +end-of-life process. It is [deprecated](../update/deprecations.md#vulnerability-check) +for use in GitLab 14.8, and is planned for removal in GitLab 15.0. Users should migrate to the new +[Security Approval Policies](../user/application_security/policies/#scan-result-policy-editor). + You can create project approval rules using the following endpoint: ```plaintext @@ -295,11 +301,11 @@ POST /projects/:id/approval_rules | `user_ids` | Array | no | The ids of users as approvers | | `group_ids` | Array | no | The ids of groups as approvers | | `protected_branch_ids` | Array | no | The IDs of protected branches to scope the rule by. To identify the ID, [use the API](protected_branches.md#list-protected-branches). | -| `report_type` | string | no | The report type required when the rule type is `report_approver`. The supported report types are: `vulnerability`, `license_scanning`, `code_coverage`. | -| `scanners` | Array | no | The security scanners the `Vulnerability-Check` approval rule considers. The supported scanners are: `sast`, `secret_detection`, `dependency_scanning`, `container_scanning`, `dast`, `coverage_fuzzing`, `api_fuzzing`. Defaults to all supported scanners. | -| `severity_levels` | Array | no | The severity levels the `Vulnerability-Check` approval rule considers. The supported severity levels are: `info`, `unknown`, `low`, `medium`, `high`, `critical`. Defaults to `unknown`, `high`, and `critical`. | -| `vulnerabilities_allowed` | integer | no | The number of vulnerabilities allowed for the `Vulnerability-Check` approval rule. Defaults to `0`. | -| `vulnerability_states` | Array | no | The vulnerability states the `Vulnerability-Check` approval rule considers. The supported vulnerability states are: `newly_detected` (default), `detected`, `confirmed`, `resolved`, `dismissed`. | +| `report_type` | string | no | The report type required when the rule type is `report_approver`. The supported report types are: `vulnerability`, `license_scanning`, `code_coverage`. The `vulnerability` report type is part of the Vulnerability-Check feature, which deprecated in GitLab 14.8, and planned for removal in GitLab 15.0. | +| `scanners` | Array | no | The security scanners the Vulnerability-Check approval rule considers. The supported scanners are: `sast`, `secret_detection`, `dependency_scanning`, `container_scanning`, `dast`, `coverage_fuzzing`, `api_fuzzing`. Defaults to all supported scanners. Deprecated in GitLab 14.8, and planned for removal in GitLab 15.0. | +| `severity_levels` | Array | no | The severity levels the Vulnerability-Check approval rule considers. The supported severity levels are: `info`, `unknown`, `low`, `medium`, `high`, `critical`. Defaults to `unknown`, `high`, and `critical`. Deprecated in GitLab 14.8, and planned for removal in GitLab 15.0. | +| `vulnerabilities_allowed` | integer | no | The number of vulnerabilities allowed for the Vulnerability-Check approval rule. Defaults to `0`. Deprecated in GitLab 14.8, and planned for removal in GitLab 15.0. | +| `vulnerability_states` | Array | no | The vulnerability states the Vulnerability-Check approval rule considers. The supported vulnerability states are: `newly_detected` (default), `detected`, `confirmed`, `resolved`, `dismissed`. Deprecated in GitLab 14.8, and planned for removal in GitLab 15.0. | ```json { @@ -404,6 +410,12 @@ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11877) in GitLab 12.3. > - Moved to GitLab Premium in 13.9. +WARNING: +The Vulnerability-Check feature, including the Vulnerability-Check attributes listed here, is in its +end-of-life process. It is [deprecated](../update/deprecations.md#vulnerability-check) +for use in GitLab 14.8, and is planned for removal in GitLab 15.0. Users should migrate to the new +[Security Approval Policies](../user/application_security/policies/#scan-result-policy-editor). + You can update project approval rules using the following endpoint: ```plaintext @@ -423,10 +435,10 @@ PUT /projects/:id/approval_rules/:approval_rule_id | `user_ids` | Array | no | The ids of users as approvers | | `group_ids` | Array | no | The ids of groups as approvers | | `protected_branch_ids` | Array | no | The IDs of protected branches to scope the rule by. To identify the ID, [use the API](protected_branches.md#list-protected-branches). | -| `scanners` | Array | no | The security scanners the `Vulnerability-Check` approval rule considers. The supported scanners are: `sast`, `secret_detection`, `dependency_scanning`, `container_scanning`, `dast`, `coverage_fuzzing`, `api_fuzzing`. Defaults to all supported scanners. | -| `severity_levels` | Array | no | The severity levels the `Vulnerability-Check` approval rule considers. The supported severity levels are: `info`, `unknown`, `low`, `medium`, `high`, `critical`. Defaults to `unknown`, `high`, and `critical`. | -| `vulnerabilities_allowed` | integer | no | The number of vulnerabilities allowed for the `Vulnerability-Check` approval rule. Defaults to `0`. | -| `vulnerability_states` | Array | no | The vulnerability states the `Vulnerability-Check` approval rule considers. The supported vulnerability states are: `newly_detected` (default), `detected`, `confirmed`, `resolved`, `dismissed`. | +| `scanners` | Array | no | The security scanners the Vulnerability-Check approval rule considers. The supported scanners are: `sast`, `secret_detection`, `dependency_scanning`, `container_scanning`, `dast`, `coverage_fuzzing`, `api_fuzzing`. Defaults to all supported scanners. Deprecated in GitLab 14.8, and planned for removal in GitLab 15.0. | +| `severity_levels` | Array | no | The severity levels the Vulnerability-Check approval rule considers. The supported severity levels are: `info`, `unknown`, `low`, `medium`, `high`, `critical`. Defaults to `unknown`, `high`, and `critical`. Deprecated in GitLab 14.8, and planned for removal in GitLab 15.0. | +| `vulnerabilities_allowed` | integer | no | The number of vulnerabilities allowed for the Vulnerability-Check approval rule. Defaults to `0`. Deprecated in GitLab 14.8, and planned for removal in GitLab 15.0. | +| `vulnerability_states` | Array | no | The vulnerability states the Vulnerability-Check approval rule considers. The supported vulnerability states are: `newly_detected` (default), `detected`, `confirmed`, `resolved`, `dismissed`. Deprecated in GitLab 14.8, and planned for removal in GitLab 15.0. | ```json { @@ -527,9 +539,9 @@ DELETE /projects/:id/approval_rules/:approval_rule_id | `id` | integer or string | yes | The ID or [URL-encoded path of a project](index.md#namespaced-path-encoding) | | `approval_rule_id` | integer | yes | The ID of a approval rule -## Merge Request-level MR approvals +## Merge request-level MR approvals -Configuration for approvals on a specific Merge Request. Must be authenticated for all endpoints. +Configuration for approvals on a specific merge request. Must be authenticated for all endpoints. ### Get Configuration @@ -957,7 +969,7 @@ These are system generated rules. | `merge_request_iid` | integer | yes | The IID of the merge request | | `approval_rule_id` | integer | yes | The ID of an approval rule | -## Approve Merge Request +## Approve merge request > Moved to GitLab Premium in 13.9. @@ -978,7 +990,7 @@ POST /projects/:id/merge_requests/:merge_request_iid/approve | `approval_password` | string | no | Current user's password. Required if [**Require user password to approve**](../user/project/merge_requests/approvals/settings.md#require-user-password-to-approve) is enabled in the project settings. | The `sha` parameter works in the same way as -when [accepting a merge request](merge_requests.md#accept-mr): if it is passed, then it must +when [accepting a merge request](merge_requests.md#merge-a-merge-request): if it is passed, then it must match the current HEAD of the merge request for the approval to be added. If it does not match, the response code is `409`. @@ -1020,7 +1032,7 @@ does not match, the response code is `409`. } ``` -## Unapprove Merge Request +## Unapprove merge request > Moved to GitLab Premium in 13.9. diff --git a/doc/api/merge_requests.md b/doc/api/merge_requests.md index 905ecd05d52..a54ea33aca8 100644 --- a/doc/api/merge_requests.md +++ b/doc/api/merge_requests.md @@ -13,41 +13,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w Every API call to merge requests must be authenticated. -**Important notes:** - -- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/29984) in GitLab 12.8, the mergeability (`merge_status`) -of each merge request is checked asynchronously when a request is made to this endpoint. Poll this API endpoint -to get updated status. This affects the `has_conflicts` property as it is dependent on the `merge_status`. It returns -`false` unless `merge_status` is `cannot_be_merged`. -- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/31890) in GitLab 13.0, listing merge requests may -not proactively update `merge_status` (which also affects the `has_conflicts`), as this can be an expensive operation. -If you need the value of these fields from this endpoint, set the `with_merge_status_recheck` parameter to -`true` in the query. -- `references.relative` is relative to the group or project that the merge request is being requested. When the merge request -is fetched from its project, `relative` format would be the same as `short` format, and when requested across groups or projects, it is expected to be the same as `full` format. -- If `approvals_before_merge` is not provided, it inherits the value from the target project. If provided, the following conditions must hold for it to take effect: - - - The target project's `approvals_before_merge` must be greater than zero. A - value of zero disables approvals for that project. - - The provided value of `approvals_before_merge` must be greater than the - target project's `approvals_before_merge`. - - This API returns `HTTP 201 Created` for a successful response. - -- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/46190) in GitLab 13.6, -diffs associated with the set of changes have the same size limitations applied as other diffs -returned by the API or viewed via the UI. When these limits impact the results, the `overflow` -field contains a value of `true`. Diff data without these limits applied can be retrieved by -adding the `access_raw_diffs` parameter, accessing diffs not from the database but from Gitaly directly. -This approach is generally slower and more resource-intensive, but isn't subject to size limits -placed on database-backed diffs. [Limits inherent to Gitaly](../development/diffs.md#diff-limits) -still apply. - -- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/349031) in GitLab 14.7, -field `merge_user` can be either user who merged this merge request, -user who set it to merge when pipeline succeeds or `null`. -Field `merged_by` (user who merged this merge request or `null`) has been deprecated. - ## List merge requests Get all merge requests the authenticated user has access to. By @@ -78,7 +43,7 @@ Parameters: | Attribute | Type | Required | Description | | ------------------------------- | -------------- | -------- | ---------------------------------------------------------------------------------------------------------------------- | | `state` | string | no | Return all merge requests or just those that are `opened`, `closed`, `locked`, or `merged`. | -| `order_by` | string | no | Return requests ordered by `created_at` or `updated_at` fields. Default is `created_at`. | +| `order_by` | string | no | Return requests ordered by `created_at`, `title`, or `updated_at` fields. Default is `created_at`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/331625) in GitLab 14.8.| | `sort` | string | no | Return requests sorted in `asc` or `desc` order. Default is `desc`. | | `milestone` | string | no | Return merge requests for a specific milestone. `None` returns merge requests with no milestone. `Any` returns merge requests that have an assigned milestone. | | `view` | string | no | If `simple`, returns the `iid`, URL, title, description, and basic state of merge request. | @@ -241,6 +206,14 @@ the `approvals_before_merge` parameter: ] ``` +### Merge requests list response notes + +- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/31890) in GitLab 13.0, listing merge requests may + not proactively update `merge_status` (which also affects the `has_conflicts`), as this can be an expensive operation. + If you need the value of these fields from this endpoint, set the `with_merge_status_recheck` parameter to + `true` in the query. +- For notes on merge request object fields, read [Single merge request response notes](#single-merge-request-response-notes). + ## List project merge requests Get all merge requests for this project. @@ -273,7 +246,7 @@ Parameters: | `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | | `iids[]` | integer array | no | Return the request having the given `iid`. | | `state` | string | no | Return all merge requests or just those that are `opened`, `closed`, `locked`, or `merged`. | -| `order_by` | string | no | Return requests ordered by `created_at` or `updated_at` fields. Default is `created_at`. | +| `order_by` | string | no | Return requests ordered by `created_at`, `title` or `updated_at` fields. Default is `created_at`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/331625) in GitLab 14.8. | | `sort` | string | no | Return requests sorted in `asc` or `desc` order. Default is `desc`. | | `milestone` | string | no | Return merge requests for a specific milestone. `None` returns merge requests with no milestone. `Any` returns merge requests that have an assigned milestone. | | `view` | string | no | If `simple`, returns the `iid`, URL, title, description, and basic state of merge request. | @@ -421,16 +394,6 @@ Parameters: ] ``` -The `merge_status` field may hold one of the following values: - -| Value | Interpretation | -|----------------------------|-----------------------------------------------------------------------| -| `unchecked` | We have not checked this yet | -| `checking` | We are currently checking if the merge request can be merged | -| `can_be_merged` | This merge request can be merged without conflict | -| `cannot_be_merged` | There are merge conflicts between the source and target branches | -| `cannot_be_merged_recheck` | Currently unchecked. Before the current changes, there were conflicts | - Users on [GitLab Premium or higher](https://about.gitlab.com/pricing/) also see the `approvals_before_merge` parameter: @@ -445,6 +408,8 @@ the `approvals_before_merge` parameter: ] ``` +For important notes on response data, read [Merge requests list response notes](#merge-requests-list-response-notes). + ## List group merge requests Get all merge requests for this group and its subgroups. @@ -468,7 +433,7 @@ Parameters: | ------------------------------- | -------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------ | | `id` | integer/string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) owned by the authenticated user. | | `state` | string | no | Return all merge requests or just those that are `opened`, `closed`, `locked`, or `merged`. | -| `order_by` | string | no | Return merge requests ordered by `created_at` or `updated_at` fields. Default is `created_at`. | +| `order_by` | string | no | Return merge requests ordered by `created_at`, `title` or `updated_at` fields. Default is `created_at`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/331625) in GitLab 14.8. | | `sort` | string | no | Return merge requests sorted in `asc` or `desc` order. Default is `desc`. | | `milestone` | string | no | Return merge requests for a specific milestone. `None` returns merge requests with no milestone. `Any` returns merge requests that have an assigned milestone. | | `view` | string | no | If `simple`, returns the `iid`, URL, title, description, and basic state of merge request. | @@ -627,6 +592,8 @@ the `approvals_before_merge` parameter: ] ``` +For important notes on response data, read [Merge requests list response notes](#merge-requests-list-response-notes). + ## Get single MR Shows information about a single merge request. @@ -805,7 +772,26 @@ the `approvals_before_merge` parameter: } ``` -The `diff_refs` in the response correspond to the latest diff version of the merge request. +### Single merge request response notes + +- The `merge_status` field may hold one of the following values: + - `unchecked`: We have not checked this yet. + - `checking`: We are currently checking if the merge request can be merged. + - `can_be_merged`: This merge request can be merged without conflict. + - `cannot_be_merged`: There are merge conflicts between the source and target branches. + - `cannot_be_merged_recheck`: Currently unchecked. Before the current changes, there were conflicts. +- The `diff_refs` in the response correspond to the latest diff version of the merge request. +- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/29984) in GitLab 12.8, the mergeability (`merge_status`) + of each merge request is checked asynchronously when a request is made to this endpoint. Poll this API endpoint + to get updated status. This affects the `has_conflicts` property as it is dependent on the `merge_status`. It returns + `false` unless `merge_status` is `cannot_be_merged`. +- `references.relative` is relative to the group or project that the merge request is being requested. When the merge + request is fetched from its project, `relative` format would be the same as `short` format, and when requested across + groups or projects, it is expected to be the same as `full` format. +- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/349031) in GitLab 14.7, + field `merge_user` can be either user who merged this merge request, + user who set it to merge when pipeline succeeds or `null`. + Field `merged_by` (user who merged this merge request or `null`) has been deprecated. ## Get single MR participants @@ -885,6 +871,15 @@ Parameters: Shows information about the merge request including its files and changes. +[Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/46190) in GitLab 13.6, +diffs associated with the set of changes have the same size limitations applied as other diffs +returned by the API or viewed via the UI. When these limits impact the results, the `overflow` +field contains a value of `true`. Diff data without these limits applied can be retrieved by +adding the `access_raw_diffs` parameter, accessing diffs not from the database but from Gitaly directly. +This approach is generally slower and more resource-intensive, but isn't subject to size limits +placed on database-backed diffs. [Limits inherent to Gitaly](../development/diffs.md#diff-limits) +still apply. + ```plaintext GET /projects/:id/merge_requests/:merge_request_iid/changes ``` @@ -1040,8 +1035,8 @@ It requires `.gitlab-ci.yml` to be configured with `only: [merge_requests]` to c The new pipeline can be: - A detached merge request pipeline. -- A [pipeline for merged results](../ci/pipelines/pipelines_for_merged_results.md) - if the [project setting is enabled](../ci/pipelines/pipelines_for_merged_results.md#enable-pipelines-for-merged-results). **(PREMIUM)** +- A [merged results pipeline](../ci/pipelines/merged_results_pipelines.md) + if the [project setting is enabled](../ci/pipelines/merged_results_pipelines.md#enable-merged-results-pipelines). ```plaintext POST /projects/:id/merge_requests/:merge_request_iid/pipelines @@ -1116,9 +1111,17 @@ POST /projects/:id/merge_requests | `milestone_id` | integer | no | The global ID of a milestone. | | `remove_source_branch` | boolean | no | Flag indicating if a merge request should remove the source branch when merging. | | `allow_collaboration` | boolean | no | Allow commits from members who can merge to the target branch. | -| `allow_maintainer_to_push` | boolean | no | [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/22665), see `allow_collaboration`. | +| `allow_maintainer_to_push` | boolean | no | Alias of `allow_collaboration`. | +| `approvals_before_merge` **(PREMIUM)** | integer | no | Number of approvals required before this can be merged (see below). | | `squash` | boolean | no | Squash commits into a single commit when merging. | +If `approvals_before_merge` is not provided, it inherits the value from the target project. If provided, the following conditions must hold for it to take effect: + +- The target project's `approvals_before_merge` must be greater than zero. A + value of zero disables approvals for that project. +- The provided value of `approvals_before_merge` must be greater than the + target project's `approvals_before_merge`. + ```json { "id": 1, @@ -1251,6 +1254,8 @@ the `approvals_before_merge` parameter: } ``` +For important notes on response data, read [Single merge request response notes](#single-merge-request-response-notes). + ## Update MR Updates an existing merge request. You can change the target branch, title, or even close the MR. @@ -1278,7 +1283,7 @@ PUT /projects/:id/merge_requests/:merge_request_iid | `squash` | boolean | no | Squash commits into a single commit when merging. | | `discussion_locked` | boolean | no | Flag indicating if the merge request's discussion is locked. If the discussion is locked only project members can add, edit or resolve comments. | | `allow_collaboration` | boolean | no | Allow commits from members who can merge to the target branch. | -| `allow_maintainer_to_push` | boolean | no | [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/22665), see `allow_collaboration`. | +| `allow_maintainer_to_push` | boolean | no | Alias of `allow_collaboration`. | Must include at least one non-required attribute from above. @@ -1430,6 +1435,8 @@ the `approvals_before_merge` parameter: } ``` +For important notes on response data, read [Single merge request response notes](#single-merge-request-response-notes). + ## Delete a merge request Only for administrators and project owners. Deletes the merge request in question. @@ -1447,17 +1454,9 @@ DELETE /projects/:id/merge_requests/:merge_request_iid curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/4/merge_requests/85" ``` -## Accept MR - -Merge changes submitted with MR using this API. - -If a merge request is unable to be accepted (such as Draft, Closed, Pipeline Pending Completion, or Failed while requiring Success) - you receive a `405` and the error message 'Method Not Allowed' - -If it has some conflicts and can not be merged - you receive a `406` and the error message 'Branch cannot be merged' - -If the `sha` parameter is passed and does not match the HEAD of the source - you receive a `409` and the error message 'SHA does not match HEAD of source branch' +## Merge a merge request -If you don't have permissions to accept this merge request - you receive a `401` +Accept and merge changes submitted with MR using this API. ```plaintext PUT /projects/:id/merge_requests/:merge_request_iid/merge @@ -1624,6 +1623,17 @@ the `approvals_before_merge` parameter: } ``` +This API returns specific HTTP status codes on failure: + +| HTTP Status | Message | Reason | +| :---: | ------- | ------ | +| `401` | `Unauthorized` | This user does not have permission to accept this merge request. | +| `405` | `Method Not Allowed` | The merge request cannot be accepted because it is `Draft`, `Closed`, `Pipeline Pending Completion`, or `Failed`. `Success` is required. | +| `406` | `Branch cannot be merged` | The branch has conflicts and cannot be merged. | +| `409` | `SHA does not match HEAD of source branch` | The provided `sha` parameter does not match the HEAD of the source. | + +For additional important notes on response data, read [Single merge request response notes](#single-merge-request-response-notes). + ## Merge to default merge ref path Merge the changes between the merge request source and target branches into `refs/merge-requests/:iid/merge` @@ -1821,6 +1831,8 @@ the `approvals_before_merge` parameter: } ``` +For important notes on response data, read [Single merge request response notes](#single-merge-request-response-notes). + ## Rebase a merge request Automatically rebase the `source_branch` of the merge request against its @@ -2130,6 +2142,8 @@ the `approvals_before_merge` parameter: } ``` +For important notes on response data, read [Single merge request response notes](#single-merge-request-response-notes). + ## Unsubscribe from a merge request Unsubscribes the authenticated user from a merge request to not receive @@ -2298,6 +2312,8 @@ the `approvals_before_merge` parameter: } ``` +For important notes on response data, read [Single merge request response notes](#single-merge-request-response-notes). + ## Create a to-do item Manually creates a to-do item for the current user on a merge request. @@ -2686,7 +2702,7 @@ Example response: ## Approvals -For approvals, see [Merge Request Approvals](merge_request_approvals.md) +For approvals, see [Merge request approvals](merge_request_approvals.md) ## List merge request state events diff --git a/doc/api/metrics_dashboard_annotations.md b/doc/api/metrics_dashboard_annotations.md index feba57a7ced..7732bf61d77 100644 --- a/doc/api/metrics_dashboard_annotations.md +++ b/doc/api/metrics_dashboard_annotations.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: concepts, howto --- diff --git a/doc/api/metrics_user_starred_dashboards.md b/doc/api/metrics_user_starred_dashboards.md index f615ddaaa71..3e54ec74b24 100644 --- a/doc/api/metrics_user_starred_dashboards.md +++ b/doc/api/metrics_user_starred_dashboards.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: concepts, howto --- diff --git a/doc/api/namespaces.md b/doc/api/namespaces.md index a02d44136d1..3972a46d7fc 100644 --- a/doc/api/namespaces.md +++ b/doc/api/namespaces.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/api/notes.md b/doc/api/notes.md index 879ffaca191..445940e02fc 100644 --- a/doc/api/notes.md +++ b/doc/api/notes.md @@ -320,7 +320,7 @@ Parameters: curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/snippets/52/notes/1659" ``` -## Merge Requests +## Merge requests ### List all merge request notes diff --git a/doc/api/oauth2.md b/doc/api/oauth2.md index ef7d133e907..59a929e30f4 100644 --- a/doc/api/oauth2.md +++ b/doc/api/oauth2.md @@ -1,8 +1,8 @@ --- type: reference, howto stage: Manage -group: Authentication & Authorization -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +group: Authentication and Authorization +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- # OAuth 2.0 identity provider API **(FREE)** @@ -32,7 +32,7 @@ GitLab supports the following authorization flows: hosted, first-party services. GitLab recommends against use of this flow. The draft specification for [OAuth 2.1](https://oauth.net/2.1/) specifically omits both the -Implicit grant and Resource Owner Password Credentials flows. It will be deprecated in the next OAuth specification version. +Implicit grant and Resource Owner Password Credentials flows. Refer to the [OAuth RFC](https://tools.ietf.org/html/rfc6749) to find out how all those flows work and pick the right one for your use case. @@ -41,7 +41,9 @@ Both **authorization code** (with or without PKCE) and **implicit grant** flows registered first via the `/profile/applications` page in your user's account. During registration, by enabling proper scopes, you can limit the range of resources which the `application` can access. Upon creation, you obtain the -`application` credentials: _Application ID_ and _Client Secret_ - **keep them secure**. +`application` credentials: _Application ID_ and _Client Secret_. The _Client Secret_ +**must be kept secure**. It is also advantageous to keep the _Application ID_ +secret when your application architecture allows. For a list of scopes in GitLab, see [the provider documentation](../integration/oauth_provider.md#authorized-applications). @@ -74,7 +76,10 @@ detailed flow description, from authorization request through access token. The following steps describe our implementation of the flow. The Authorization code with PKCE flow, PKCE for short, makes it possible to securely perform -the OAuth exchange of client credentials for access tokens on public clients. +the OAuth exchange of client credentials for access tokens on public clients without +requiring access to the _Client Secret_ at all. This makes the PKCE flow advantageous +for single page JavaScript applications or other client side apps where keeping secrets +from the user is a technical impossibility. Before starting the flow, generate the `STATE`, the `CODE_VERIFIER` and the `CODE_CHALLENGE`. @@ -113,7 +118,7 @@ Before starting the flow, generate the `STATE`, the `CODE_VERIFIER` and the `COD any HTTP client. The following example uses Ruby's `rest-client`: ```ruby - parameters = 'client_id=APP_ID&client_secret=APP_SECRET&code=RETURNED_CODE&grant_type=authorization_code&redirect_uri=REDIRECT_URI&code_verifier=CODE_VERIFIER' + parameters = 'client_id=APP_ID&code=RETURNED_CODE&grant_type=authorization_code&redirect_uri=REDIRECT_URI&code_verifier=CODE_VERIFIER' RestClient.post 'https://gitlab.example.com/oauth/token', parameters ``` @@ -135,7 +140,7 @@ Before starting the flow, generate the `STATE`, the `CODE_VERIFIER` and the `COD - Sends new tokens in the response. ```ruby - parameters = 'client_id=APP_ID&client_secret=APP_SECRET&refresh_token=REFRESH_TOKEN&grant_type=refresh_token&redirect_uri=REDIRECT_URI&code_verifier=CODE_VERIFIER' + parameters = 'client_id=APP_ID&refresh_token=REFRESH_TOKEN&grant_type=refresh_token&redirect_uri=REDIRECT_URI&code_verifier=CODE_VERIFIER' RestClient.post 'https://gitlab.example.com/oauth/token', parameters ``` @@ -239,19 +244,13 @@ You can now make requests to the API with the access token returned. ### Implicit grant flow -NOTE: -For a detailed flow diagram, see the [RFC specification](https://tools.ietf.org/html/rfc6749#section-4.2). - WARNING: Implicit grant flow is inherently insecure and the IETF has removed it in [OAuth 2.1](https://oauth.net/2.1/). -For this reason, [support for it is deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/288516). -In GitLab 14.0, new applications can't be created using it. In GitLab 14.4, support for it is -scheduled to be removed for existing applications. +It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/288516) for use in GitLab 14.0, and is planned for +[removal](https://gitlab.com/gitlab-org/gitlab/-/issues/344609) in GitLab 15.0. -We recommend that you use [Authorization code with PKCE](#authorization-code-with-proof-key-for-code-exchange-pkce) instead. If you choose to use Implicit flow, be sure to verify the -`application id` (or `client_id`) associated with the access token before granting -access to the data. To learn more, read -[Retrieving the token information](#retrieve-the-token-information)). +We recommend that you use [Authorization code with PKCE](#authorization-code-with-proof-key-for-code-exchange-pkce) +instead. Unlike the authorization code flow, the client receives an `access token` immediately as a result of the authorization request. The flow does not use the @@ -415,7 +414,7 @@ The following is an example response: The fields `scopes` and `expires_in_seconds` are included in the response. -These are aliases for `scope` and `expires_in` respectively, and have been included to +These fields are aliases for `scope` and `expires_in` respectively, and have been included to prevent breaking changes introduced in [doorkeeper 5.0.2](https://github.com/doorkeeper-gem/doorkeeper/wiki/Migration-from-old-versions#from-4x-to-5x). Don't rely on these fields as they are slated for removal in a later release. diff --git a/doc/api/pages.md b/doc/api/pages.md index a115f0b0a0f..7316d225dbc 100644 --- a/doc/api/pages.md +++ b/doc/api/pages.md @@ -12,7 +12,7 @@ The GitLab Pages feature must be enabled to use these endpoints. Find out more a ## Unpublish pages -Remove pages. The user must have the Administrator role. +Remove pages. The user must have administrator access. ```plaintext DELETE /projects/:id/pages diff --git a/doc/api/pages_domains.md b/doc/api/pages_domains.md index 624bdf29e5d..c1f81ffa361 100644 --- a/doc/api/pages_domains.md +++ b/doc/api/pages_domains.md @@ -12,7 +12,7 @@ The GitLab Pages feature must be enabled to use these endpoints. Find out more a ## List all Pages domains -Get a list of all Pages domains. The user must have the administrator role. +Get a list of all Pages domains. The user must have administrator access. ```plaintext GET /pages/domains diff --git a/doc/api/plan_limits.md b/doc/api/plan_limits.md index 8d37189ef2a..75c8d241513 100644 --- a/doc/api/plan_limits.md +++ b/doc/api/plan_limits.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -13,8 +13,7 @@ The plan limits API allows you to maintain the application limits for the existi The existing plans depend on the GitLab edition. In the Community Edition, only the plan `default` is available. In the Enterprise Edition, additional plans are available as well. -NOTE: -The Administrator role is required to use this API. +Administrator access is required to use this API. ## Get current plan limits diff --git a/doc/api/project_access_tokens.md b/doc/api/project_access_tokens.md index 125797a802f..f6eced4f08a 100644 --- a/doc/api/project_access_tokens.md +++ b/doc/api/project_access_tokens.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/api/project_clusters.md b/doc/api/project_clusters.md index 129b064c650..c1f59520bd7 100644 --- a/doc/api/project_clusters.md +++ b/doc/api/project_clusters.md @@ -12,7 +12,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. -Users need at least the [Maintainer](../user/permissions.md) role to use these endpoints. +Users need at least the Maintainer role to use these endpoints. ## List project clusters diff --git a/doc/api/project_import_export.md b/doc/api/project_import_export.md index e7609d34998..218036b1ee0 100644 --- a/doc/api/project_import_export.md +++ b/doc/api/project_import_export.md @@ -188,9 +188,9 @@ As an administrator, you can modify the maximum import file size. To do so, use ## Import a file from a remote object storage -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/282503) in GitLab 13.12 in [Beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta). +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/282503) in GitLab 13.12 in [Beta](../policy/alpha-beta-support.md#beta-features). -This endpoint is behind a feature flag that is disabled by default. +This endpoint is behind a feature flag that is enabled by default. To enable this endpoint: @@ -243,8 +243,8 @@ curl --request POST \ } ``` -The `ContentType` header must return a valid number. The maximum file size is 10 gigabytes. -The `ContentLength` header must be `application/gzip`. +The `Content-Length` header must return a valid number. The maximum file size is 10 gigabytes. +The `Content-Type` header must be `application/gzip`. ## Import status diff --git a/doc/api/project_relations_export.md b/doc/api/project_relations_export.md index 2016dcbd141..74fdc91e420 100644 --- a/doc/api/project_relations_export.md +++ b/doc/api/project_relations_export.md @@ -18,7 +18,7 @@ With the Project Relations Export API, you can partially export project structur similar to [project export](project_import_export.md), but it exports each top-level relation (for example, milestones/boards/labels) as a separate file instead of one archive. The project relations export API is primarily used in -[group migration](../user/group/import/index.md#enable-or-disable-gitlab-group-migration) +[group migration](../user/group/import/index.md) to support group project import. ## Schedule new export diff --git a/doc/api/project_snippets.md b/doc/api/project_snippets.md index c5f317f7540..efbd8bf9431 100644 --- a/doc/api/project_snippets.md +++ b/doc/api/project_snippets.md @@ -246,7 +246,7 @@ curl "https://gitlab.com/api/v4/projects/1/snippets/2/files/master/snippet%2Erb/ ## Get user agent details -Available only for users with the Administrator [role](../user/permissions.md). +Available only for users with administrator access. ```plaintext GET /projects/:id/snippets/:snippet_id/user_agent_detail diff --git a/doc/api/projects.md b/doc/api/projects.md index 791be613c71..db8d2361439 100644 --- a/doc/api/projects.md +++ b/doc/api/projects.md @@ -10,8 +10,8 @@ Interact with [projects](../user/project/index.md) using the REST API. ## Project visibility level -Project in GitLab can be either private, internal or public. -This is determined by the `visibility` field in the project. +A project in GitLab can be private, internal, or public. +The visibility level is determined by the `visibility` field in the project. Values for the project visibility level are: @@ -55,7 +55,7 @@ GET /projects | `min_access_level` | integer | **{dotted-circle}** No | Limit by current user minimal [access level](members.md#valid-access-levels). | | `order_by` | string | **{dotted-circle}** No | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, `last_activity_at`, or `similarity` fields. `repository_size`, `storage_size`, `packages_size` or `wiki_size` fields are only allowed for administrators. `similarity` ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332890) in GitLab 14.1) is only available when searching and is limited to projects that the current user is a member of. Default is `created_at`. | | `owned` | boolean | **{dotted-circle}** No | Limit by projects explicitly owned by the current user. | -| `repository_checksum_failed` **(PREMIUM)** | boolean | **{dotted-circle}** No | Limit projects where the repository checksum calculation has failed ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/6137) in GitLab 11.2). | +| `repository_checksum_failed` **(PREMIUM)** | boolean | **{dotted-circle}** No | Limit projects where the repository checksum calculation has failed. | | `repository_storage` | string | **{dotted-circle}** No | Limit results to projects stored on `repository_storage`. _(administrators only)_ | | `search_namespaces` | boolean | **{dotted-circle}** No | Include ancestor namespaces when matching search criteria. Default is `false`. | | `search` | string | **{dotted-circle}** No | Return list of projects matching the search criteria. | @@ -65,7 +65,7 @@ GET /projects | `statistics` | boolean | **{dotted-circle}** No | Include project statistics. Only available to Reporter or higher level role members. | | `topic` | string | **{dotted-circle}** No | Comma-separated topic names. Limit results to projects that match all of given topics. See `topics` attribute. | | `visibility` | string | **{dotted-circle}** No | Limit by visibility `public`, `internal`, or `private`. | -| `wiki_checksum_failed` **(PREMIUM)** | boolean | **{dotted-circle}** No | Limit projects where the wiki checksum calculation has failed ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/6137) in GitLab 11.2). | +| `wiki_checksum_failed` **(PREMIUM)** | boolean | **{dotted-circle}** No | Limit projects where the wiki checksum calculation has failed. | | `with_custom_attributes` | boolean | **{dotted-circle}** No | Include [custom attributes](custom_attributes.md) in response. _(administrator only)_ | | `with_issues_enabled` | boolean | **{dotted-circle}** No | Limit by enabled issues feature. | | `with_merge_requests_enabled` | boolean | **{dotted-circle}** No | Limit by enabled merge requests feature. | @@ -383,6 +383,7 @@ GET /users/:user_id/projects | Attribute | Type | Required | Description | |-------------------------------|---------|------------------------|-------------| +| `user_id` | string | **{check-circle}** Yes | The ID or username of the user. | | `archived` | boolean | **{dotted-circle}** No | Limit by archived status. | | `id_after` | integer | **{dotted-circle}** No | Limit results to projects with IDs greater than the specified ID. | | `id_before` | integer | **{dotted-circle}** No | Limit results to projects with IDs less than the specified ID. | @@ -395,7 +396,6 @@ GET /users/:user_id/projects | `sort` | string | **{dotted-circle}** No | Return projects sorted in `asc` or `desc` order. Default is `desc`. | | `starred` | boolean | **{dotted-circle}** No | Limit by projects starred by the current user. | | `statistics` | boolean | **{dotted-circle}** No | Include project statistics. Only available to Reporter or higher level role members. | -| `user_id` | string | **{check-circle}** Yes | The ID or username of the user. | | `visibility` | string | **{dotted-circle}** No | Limit by visibility `public`, `internal`, or `private`. | | `with_custom_attributes` | boolean | **{dotted-circle}** No | Include [custom attributes](custom_attributes.md) in response. _(administrator only)_ | | `with_issues_enabled` | boolean | **{dotted-circle}** No | Limit by enabled issues feature. | @@ -628,17 +628,17 @@ GET /users/:user_id/starred_projects | Attribute | Type | Required | Description | |-------------------------------|---------|------------------------|-------------| +| `user_id` | string | **{check-circle}** Yes | The ID or username of the user. | | `archived` | boolean | **{dotted-circle}** No | Limit by archived status. | | `membership` | boolean | **{dotted-circle}** No | Limit by projects that the current user is a member of. | | `min_access_level` | integer | **{dotted-circle}** No | Limit by current user minimal [access level](members.md#valid-access-levels). | | `order_by` | string | **{dotted-circle}** No | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, or `last_activity_at` fields. Default is `created_at`. | | `owned` | boolean | **{dotted-circle}** No | Limit by projects explicitly owned by the current user. | | `search` | string | **{dotted-circle}** No | Return list of projects matching the search criteria. | -| `simple` | boolean | **{dotted-circle}** No | Return only limited fields for each project. This is a no-op without authentication as then _only_ simple fields are returned.. | +| `simple` | boolean | **{dotted-circle}** No | Return only limited fields for each project. This is a no-op without authentication as then _only_ simple fields are returned. | | `sort` | string | **{dotted-circle}** No | Return projects sorted in `asc` or `desc` order. Default is `desc`. | | `starred` | boolean | **{dotted-circle}** No | Limit by projects starred by the current user. | | `statistics` | boolean | **{dotted-circle}** No | Include project statistics. Only available to Reporter or higher level role members. | -| `user_id` | string | **{check-circle}** Yes | The ID or username of the user. | | `visibility` | string | **{dotted-circle}** No | Limit by visibility `public`, `internal`, or `private`. | | `with_custom_attributes` | boolean | **{dotted-circle}** No | Include [custom attributes](custom_attributes.md) in response. _(administrator only)_ | | `with_issues_enabled` | boolean | **{dotted-circle}** No | Limit by enabled issues feature. | @@ -1045,10 +1045,6 @@ can also see the `approvals_before_merge` parameter: } ``` -The `web_url` and `avatar_url` attributes on `namespace` were -[introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/27427) -in GitLab 11.11. - If the project is a fork, the `forked_from_project` field appears in the response. For this field, if the upstream project is private, a valid token for authentication must be provided. The field `mr_default_target_self` appears as well. If this value is `false`, then all merge requests @@ -1164,12 +1160,12 @@ GET /projects/:id/groups | Attribute | Type | Required | Description | |-----------------------------|-------------------|------------------------|-------------| -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding). | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding). | | `search` | string | **{dotted-circle}** No | Search for specific groups. | -| `skip_groups` | array of integers | **{dotted-circle}** No | Skip the group IDs passed. | -| `with_shared` | boolean | **{dotted-circle}** No | Include projects shared with this group. Default is `false`. | | `shared_min_access_level` | integer | **{dotted-circle}** No | Limit to shared groups with at least this [access level](members.md#valid-access-levels). | | `shared_visible_only` | boolean | **{dotted-circle}** No | Limit to shared groups user has access to. | +| `skip_groups` | array of integers | **{dotted-circle}** No | Skip the group IDs passed. | +| `with_shared` | boolean | **{dotted-circle}** No | Include projects shared with this group. Default is `false`. | ```json [ @@ -1226,8 +1222,8 @@ POST /projects | `builds_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `ci_config_path` | string | **{dotted-circle}** No | The path to CI configuration file. | | `container_expiration_policy_attributes` | hash | **{dotted-circle}** No | Update the image cleanup policy for this project. Accepts: `cadence` (string), `keep_n` (integer), `older_than` (string), `name_regex` (string), `name_regex_delete` (string), `name_regex_keep` (string), `enabled` (boolean). Valid values for `cadence` are: `1d` (every day), `7d` (every week), `14d` (every two weeks), `1month` (every month), or `3month` (every quarter). | -| `container_registry_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable container registry for this project. Use `container_registry_access_level` instead. | | `container_registry_access_level` | string | **{dotted-circle}** No | Set visibility of container registry, for this project, to one of `disabled`, `private` or `enabled`. | +| `container_registry_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable container registry for this project. Use `container_registry_access_level` instead. | | `default_branch` | string | **{dotted-circle}** No | The [default branch](../user/project/repository/branches/default.md) name. Requires `initialize_with_readme` to be `true`. | | `description` | string | **{dotted-circle}** No | Short project description. | | `emails_disabled` | boolean | **{dotted-circle}** No | Disable email notifications. | @@ -1248,21 +1244,20 @@ POST /projects | `mirror_trigger_builds` **(PREMIUM)** | boolean | **{dotted-circle}** No | Pull mirroring triggers builds. | | `mirror` **(PREMIUM)** | boolean | **{dotted-circle}** No | Enables pull mirroring in a project. | | `namespace_id` | integer | **{dotted-circle}** No | Namespace for the new project (defaults to the current user's namespace). | -| `operations_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `only_allow_merge_if_all_discussions_are_resolved` | boolean | **{dotted-circle}** No | Set whether merge requests can only be merged when all the discussions are resolved. | | `only_allow_merge_if_pipeline_succeeds` | boolean | **{dotted-circle}** No | Set whether merge requests can only be merged with successful pipelines. This setting is named [**Pipelines must succeed**](../user/project/merge_requests/merge_when_pipeline_succeeds.md#only-allow-merge-requests-to-be-merged-if-the-pipeline-succeeds) in the project settings. | +| `operations_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `packages_enabled` | boolean | **{dotted-circle}** No | Enable or disable packages repository feature. | | `pages_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, `enabled`, or `public`. | -| `requirements_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, `enabled` or `public` | | `printing_merge_request_link_enabled` | boolean | **{dotted-circle}** No | Show link to create/view merge request when pushing from the command line. | | `public_builds` | boolean | **{dotted-circle}** No | If `true`, jobs can be viewed by non-project members. | | `remove_source_branch_after_merge` | boolean | **{dotted-circle}** No | Enable `Delete source branch` option by default for all new merge requests. | | `repository_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `repository_storage` | string | **{dotted-circle}** No | Which storage shard the repository is on. _(administrator only)_ | | `request_access_enabled` | boolean | **{dotted-circle}** No | Allow users to request member access. | +| `requirements_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private` or `enabled` | | `resolve_outdated_diff_discussions` | boolean | **{dotted-circle}** No | Automatically resolve merge request diffs discussions on lines changed with a push. | | `shared_runners_enabled` | boolean | **{dotted-circle}** No | Enable shared runners for this project. | -| `show_default_award_emojis` | boolean | **{dotted-circle}** No | Show default award emojis. | | `snippets_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `snippets_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable snippets for this project. Use `snippets_access_level` instead. | | `squash_option` | string | **{dotted-circle}** No | One of `never`, `always`, `default_on`, or `default_off`. | @@ -1304,10 +1299,10 @@ POST /projects/user/:user_id | `build_timeout` | integer | **{dotted-circle}** No | The maximum amount of time, in seconds, that a job can run. | | `builds_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `ci_config_path` | string | **{dotted-circle}** No | The path to CI configuration file. | -| `container_registry_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable container registry for this project. Use `container_registry_access_level` instead. | | `container_registry_access_level` | string | **{dotted-circle}** No | Set visibility of container registry, for this project, to one of `disabled`, `private` or `enabled`. | -| `description` | string | **{dotted-circle}** No | Short project description. | +| `container_registry_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable container registry for this project. Use `container_registry_access_level` instead. | | `default_branch` | string | **{dotted-circle}** No | The [default branch](../user/project/repository/branches/default.md) name. Requires `initialize_with_readme` to be `true`. | +| `description` | string | **{dotted-circle}** No | Short project description. | | `emails_disabled` | boolean | **{dotted-circle}** No | Disable email notifications. | | `external_authorization_classification_label` **(PREMIUM)** | string | **{dotted-circle}** No | The classification label for the project. | | `forking_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | @@ -1319,19 +1314,17 @@ POST /projects/user/:user_id | `jobs_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable jobs for this project. Use `builds_access_level` instead. | | `lfs_enabled` | boolean | **{dotted-circle}** No | Enable LFS. | | `merge_commit_template` | string | **{dotted-circle}** No | [Template](../user/project/merge_requests/commit_templates.md) used to create merge commit message in merge requests. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20263) in GitLab 14.5.)_ | -| `squash_commit_template` | string | **{dotted-circle}** No | [Template](../user/project/merge_requests/commit_templates.md) used to create squash commit message in merge requests. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345275) in GitLab 14.6.)_ | | `merge_method` | string | **{dotted-circle}** No | Set the [merge method](#project-merge-method) used. | | `merge_requests_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `merge_requests_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable merge requests for this project. Use `merge_requests_access_level` instead. | | `mirror_trigger_builds` **(PREMIUM)** | boolean | **{dotted-circle}** No | Pull mirroring triggers builds. | | `mirror` **(PREMIUM)** | boolean | **{dotted-circle}** No | Enables pull mirroring in a project. | | `namespace_id` | integer | **{dotted-circle}** No | Namespace for the new project (defaults to the current user's namespace). | -| `operations_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `only_allow_merge_if_all_discussions_are_resolved` | boolean | **{dotted-circle}** No | Set whether merge requests can only be merged when all the discussions are resolved. | | `only_allow_merge_if_pipeline_succeeds` | boolean | **{dotted-circle}** No | Set whether merge requests can only be merged with successful jobs. | +| `operations_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `packages_enabled` | boolean | **{dotted-circle}** No | Enable or disable packages repository feature. | | `pages_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, `enabled`, or `public`. | -| `requirements_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, `enabled` or `public` | | `path` | string | **{dotted-circle}** No | Custom repository name for new project. By default generated based on name. | | `printing_merge_request_link_enabled` | boolean | **{dotted-circle}** No | Show link to create/view merge request when pushing from the command line. | | `public_builds` | boolean | **{dotted-circle}** No | If `true`, jobs can be viewed by non-project-members. | @@ -1339,11 +1332,12 @@ POST /projects/user/:user_id | `repository_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `repository_storage` | string | **{dotted-circle}** No | Which storage shard the repository is on. _(administrators only)_ | | `request_access_enabled` | boolean | **{dotted-circle}** No | Allow users to request member access. | +| `requirements_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, `enabled` or `public` | | `resolve_outdated_diff_discussions` | boolean | **{dotted-circle}** No | Automatically resolve merge request diffs discussions on lines changed with a push. | | `shared_runners_enabled` | boolean | **{dotted-circle}** No | Enable shared runners for this project. | -| `show_default_award_emojis` | boolean | **{dotted-circle}** No | Show default award emojis. | | `snippets_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `snippets_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable snippets for this project. Use `snippets_access_level` instead. | +| `squash_commit_template` | string | **{dotted-circle}** No | [Template](../user/project/merge_requests/commit_templates.md) used to create squash commit message in merge requests. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345275) in GitLab 14.6.)_ | | `squash_option` | string | **{dotted-circle}** No | One of `never`, `always`, `default_on`, or `default_off`. | | `suggestion_commit_message` | string | **{dotted-circle}** No | The commit message used to apply merge request [suggestions](../user/project/merge_requests/reviews/suggestions.md). | | `tag_list` | array | **{dotted-circle}** No | _([Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0)_ The list of tags for a project; put array of tags, that should be finally assigned to a project. Use `topics` instead. | @@ -1379,6 +1373,7 @@ Supported attributes: | Attribute | Type | Required | Description | |-------------------------------------------------------------|----------------|------------------------|-------------| +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding). | | `allow_merge_on_skipped_pipeline` | boolean | **{dotted-circle}** No | Set whether or not merge requests can be merged with skipped jobs. | | `analytics_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private` or `enabled` | | `approvals_before_merge` **(PREMIUM)** | integer | **{dotted-circle}** No | How many approvers should approve merge request by default. To configure approval rules, see [Merge request approvals API](merge_request_approvals.md). | @@ -1395,38 +1390,39 @@ Supported attributes: | `ci_default_git_depth` | integer | **{dotted-circle}** No | Default number of revisions for [shallow cloning](../ci/pipelines/settings.md#limit-the-number-of-changes-fetched-during-clone). | | `ci_forward_deployment_enabled` | boolean | **{dotted-circle}** No | When a new deployment job starts, [skip older deployment jobs](../ci/pipelines/settings.md#skip-outdated-deployment-jobs) that are still pending | | `container_expiration_policy_attributes` | hash | **{dotted-circle}** No | Update the image cleanup policy for this project. Accepts: `cadence` (string), `keep_n` (integer), `older_than` (string), `name_regex` (string), `name_regex_delete` (string), `name_regex_keep` (string), `enabled` (boolean). | -| `container_registry_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable container registry for this project. Use `container_registry_access_level` instead. | | `container_registry_access_level` | string | **{dotted-circle}** No | Set visibility of container registry, for this project, to one of `disabled`, `private` or `enabled`. | +| `container_registry_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable container registry for this project. Use `container_registry_access_level` instead. | | `default_branch` | string | **{dotted-circle}** No | The [default branch](../user/project/repository/branches/default.md) name. | | `description` | string | **{dotted-circle}** No | Short project description. | | `emails_disabled` | boolean | **{dotted-circle}** No | Disable email notifications. | | `external_authorization_classification_label` **(PREMIUM)** | string | **{dotted-circle}** No | The classification label for the project. | | `forking_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding). | | `import_url` | string | **{dotted-circle}** No | URL to import repository from. | | `issues_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `issues_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable issues for this project. Use `issues_access_level` instead. | +| `issues_template` **(PREMIUM)** | string | **{dotted-circle}** No | Default description for Issues. Description is parsed with GitLab Flavored Markdown. See [Templates for issues and merge requests](#templates-for-issues-and-merge-requests). | | `jobs_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable jobs for this project. Use `builds_access_level` instead. | +| `keep_latest_artifact` | boolean | **{dotted-circle}** No | Disable or enable the ability to keep the latest artifact for this project. | | `lfs_enabled` | boolean | **{dotted-circle}** No | Enable LFS. | | `merge_commit_template` | string | **{dotted-circle}** No | [Template](../user/project/merge_requests/commit_templates.md) used to create merge commit message in merge requests. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20263) in GitLab 14.5.)_ | | `merge_method` | string | **{dotted-circle}** No | Set the [merge method](#project-merge-method) used. | | `merge_pipelines_enabled` | boolean | **{dotted-circle}** No | Enable or disable merge pipelines. | | `merge_requests_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `merge_requests_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable merge requests for this project. Use `merge_requests_access_level` instead. | +| `merge_requests_template` **(PREMIUM)** | string | **{dotted-circle}** No | Default description for merge requests. Description is parsed with GitLab Flavored Markdown. See [Templates for issues and merge requests](#templates-for-issues-and-merge-requests). | | `merge_trains_enabled` | boolean | **{dotted-circle}** No | Enable or disable merge trains. | | `mirror_overwrites_diverged_branches` **(PREMIUM)** | boolean | **{dotted-circle}** No | Pull mirror overwrites diverged branches. | | `mirror_trigger_builds` **(PREMIUM)** | boolean | **{dotted-circle}** No | Pull mirroring triggers builds. | | `mirror_user_id` **(PREMIUM)** | integer | **{dotted-circle}** No | User responsible for all the activity surrounding a pull mirror event. _(administrators only)_ | | `mirror` **(PREMIUM)** | boolean | **{dotted-circle}** No | Enables pull mirroring in a project. | +| `mr_default_target_self` | boolean | **{dotted-circle}** No | For forked projects, target merge requests to this project. If `false`, the target will be the upstream project. | | `name` | string | **{dotted-circle}** No | The name of the project. | -| `operations_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `only_allow_merge_if_all_discussions_are_resolved` | boolean | **{dotted-circle}** No | Set whether merge requests can only be merged when all the discussions are resolved. | | `only_allow_merge_if_pipeline_succeeds` | boolean | **{dotted-circle}** No | Set whether merge requests can only be merged with successful jobs. | | `only_mirror_protected_branches` **(PREMIUM)** | boolean | **{dotted-circle}** No | Only mirror protected branches. | +| `operations_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `packages_enabled` | boolean | **{dotted-circle}** No | Enable or disable packages repository feature. | | `pages_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, `enabled`, or `public`. | -| `requirements_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, `enabled` or `public` | -| `restrict_user_defined_variables` | boolean | **{dotted-circle}** No | Allow only users with the [Maintainer role](../user/permissions.md) to pass user-defined variables when triggering a pipeline. For example when the pipeline is triggered in the UI, with the API, or by a trigger token. | | `path` | string | **{dotted-circle}** No | Custom repository name for the project. By default generated based on name. | | `printing_merge_request_link_enabled` | boolean | **{dotted-circle}** No | Show link to create/view merge request when pushing from the command line. | | `public_builds` | boolean | **{dotted-circle}** No | If `true`, jobs can be viewed by non-project members. | @@ -1434,10 +1430,11 @@ Supported attributes: | `repository_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `repository_storage` | string | **{dotted-circle}** No | Which storage shard the repository is on. _(administrators only)_ | | `request_access_enabled` | boolean | **{dotted-circle}** No | Allow users to request member access. | +| `requirements_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, `enabled` or `public` | | `resolve_outdated_diff_discussions` | boolean | **{dotted-circle}** No | Automatically resolve merge request diffs discussions on lines changed with a push. | +| `restrict_user_defined_variables` | boolean | **{dotted-circle}** No | Allow only users with the Maintainer role to pass user-defined variables when triggering a pipeline. For example when the pipeline is triggered in the UI, with the API, or by a trigger token. | | `service_desk_enabled` | boolean | **{dotted-circle}** No | Enable or disable Service Desk feature. | | `shared_runners_enabled` | boolean | **{dotted-circle}** No | Enable shared runners for this project. | -| `show_default_award_emojis` | boolean | **{dotted-circle}** No | Show default award emojis. | | `snippets_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `snippets_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable snippets for this project. Use `snippets_access_level` instead. | | `squash_commit_template` | string | **{dotted-circle}** No | [Template](../user/project/merge_requests/commit_templates.md) used to create squash commit message in merge requests. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345275) in GitLab 14.6.)_ | @@ -1448,10 +1445,6 @@ Supported attributes: | `visibility` | string | **{dotted-circle}** No | See [project visibility level](#project-visibility-level). | | `wiki_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `wiki_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable wiki for this project. Use `wiki_access_level` instead. | -| `issues_template` **(PREMIUM)** | string | **{dotted-circle}** No | Default description for Issues. Description is parsed with GitLab Flavored Markdown. See [Templates for issues and merge requests](#templates-for-issues-and-merge-requests). | -| `merge_requests_template` **(PREMIUM)** | string | **{dotted-circle}** No | Default description for Merge Requests. Description is parsed with GitLab Flavored Markdown. See [Templates for issues and merge requests](#templates-for-issues-and-merge-requests). | -| `keep_latest_artifact` | boolean | **{dotted-circle}** No | Disable or enable the ability to keep the latest artifact for this project. | -| `mr_default_target_self` | boolean | **{dotted-circle}** No | For forked projects, target merge requests to this project. If `false`, the target will be the upstream project. | ## Fork project @@ -1468,18 +1461,16 @@ POST /projects/:id/fork | Attribute | Type | Required | Description | |------------------|----------------|------------------------|-------------| | `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding). | +| `description` | string | **{dotted-circle}** No | The description assigned to the resultant project after forking. | +| `mr_default_target_self` | boolean | **{dotted-circle}** No | For forked projects, target merge requests to this project. If `false`, the target will be the upstream project. | | `name` | string | **{dotted-circle}** No | The name assigned to the resultant project after forking. | | `namespace_id` | integer | **{dotted-circle}** No | The ID of the namespace that the project is forked to. | | `namespace_path` | string | **{dotted-circle}** No | The path of the namespace that the project is forked to. | | `namespace` | integer or string | **{dotted-circle}** No | _(Deprecated)_ The ID or path of the namespace that the project is forked to. | | `path` | string | **{dotted-circle}** No | The path assigned to the resultant project after forking. | -| `description` | string | **{dotted-circle}** No | The description assigned to the resultant project after forking. | | `visibility` | string | **{dotted-circle}** No | The [visibility level](#project-visibility-level) assigned to the resultant project after forking. | -| `mr_default_target_self` | boolean | **{dotted-circle}** No | For forked projects, target merge requests to this project. If `false`, the target will be the upstream project. | - -## List Forks of a project -> Introduced in GitLab 10.1. +## List forks of a project List the projects accessible to the calling user that have an established, forked relationship with the specified project @@ -1490,8 +1481,8 @@ GET /projects/:id/forks | Attribute | Type | Required | Description | |-------------------------------|----------------|------------------------|-------------| -| `archived` | boolean | **{dotted-circle}** No | Limit by archived status. | | `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding). | +| `archived` | boolean | **{dotted-circle}** No | Limit by archived status. | | `membership` | boolean | **{dotted-circle}** No | Limit by projects that the current user is a member of. | | `min_access_level` | integer | **{dotted-circle}** No | Limit by current user minimal [access level](members.md#valid-access-levels). | | `order_by` | string | **{dotted-circle}** No | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, or `last_activity_at` fields. Default is `created_at`. | @@ -1789,7 +1780,7 @@ Example response: } ``` -## List Starrers of a project +## List starrers of a project List the users who starred the specified project. @@ -2267,10 +2258,10 @@ POST /projects/:id/share | Attribute | Type | Required | Description | |----------------|----------------|------------------------|-------------| -| `expires_at` | string | **{dotted-circle}** No | Share expiration date in ISO 8601 format: 2016-09-26 | | `group_access` | integer | **{check-circle}** Yes | The [access level](members.md#valid-access-levels) to grant the group. | | `group_id` | integer | **{check-circle}** Yes | The ID of the group to share with. | | `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding). | +| `expires_at` | string | **{dotted-circle}** No | Share expiration date in ISO 8601 format: 2016-09-26 | ## Delete a shared project link within a group @@ -2375,11 +2366,12 @@ POST /projects/:id/hooks | Attribute | Type | Required | Description | |------------------------------|----------------|------------------------|-------------| +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding). | +| `url` | string | **{check-circle}** Yes | The hook URL. | | `confidential_issues_events` | boolean | **{dotted-circle}** No | Trigger hook on confidential issues events. | | `confidential_note_events` | boolean | **{dotted-circle}** No | Trigger hook on confidential note events. | | `deployment_events` | boolean | **{dotted-circle}** No | Trigger hook on deployment events. | | `enable_ssl_verification` | boolean | **{dotted-circle}** No | Do SSL verification when triggering the hook. | -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding). | | `issues_events` | boolean | **{dotted-circle}** No | Trigger hook on issues events. | | `job_events` | boolean | **{dotted-circle}** No | Trigger hook on job events. | | `merge_requests_events` | boolean | **{dotted-circle}** No | Trigger hook on merge requests events. | @@ -2387,11 +2379,10 @@ POST /projects/:id/hooks | `pipeline_events` | boolean | **{dotted-circle}** No | Trigger hook on pipeline events. | | `push_events_branch_filter` | string | **{dotted-circle}** No | Trigger hook on push events for matching branches only. | | `push_events` | boolean | **{dotted-circle}** No | Trigger hook on push events. | +| `releases_events` | boolean | **{dotted-circle}** No | Trigger hook on release events. | | `tag_push_events` | boolean | **{dotted-circle}** No | Trigger hook on tag push events. | | `token` | string | **{dotted-circle}** No | Secret token to validate received payloads; this isn't returned in the response. | -| `url` | string | **{check-circle}** Yes | The hook URL. | | `wiki_page_events` | boolean | **{dotted-circle}** No | Trigger hook on wiki events. | -| `releases_events` | boolean | **{dotted-circle}** No | Trigger hook on release events. | ### Edit project hook @@ -2403,12 +2394,13 @@ PUT /projects/:id/hooks/:hook_id | Attribute | Type | Required | Description | |------------------------------|----------------|------------------------|-------------| +| `hook_id` | integer | **{check-circle}** Yes | The ID of the project hook. | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding). | +| `url` | string | **{check-circle}** Yes | The hook URL. | | `confidential_issues_events` | boolean | **{dotted-circle}** No | Trigger hook on confidential issues events. | | `confidential_note_events` | boolean | **{dotted-circle}** No | Trigger hook on confidential note events. | | `deployment_events` | boolean | **{dotted-circle}** No | Trigger hook on deployment events. | | `enable_ssl_verification` | boolean | **{dotted-circle}** No | Do SSL verification when triggering the hook. | -| `hook_id` | integer | **{check-circle}** Yes | The ID of the project hook. | -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding). | | `issues_events` | boolean | **{dotted-circle}** No | Trigger hook on issues events. | | `job_events` | boolean | **{dotted-circle}** No | Trigger hook on job events. | | `merge_requests_events` | boolean | **{dotted-circle}** No | Trigger hook on merge requests events. | @@ -2416,11 +2408,10 @@ PUT /projects/:id/hooks/:hook_id | `pipeline_events` | boolean | **{dotted-circle}** No | Trigger hook on pipeline events. | | `push_events_branch_filter` | string | **{dotted-circle}** No | Trigger hook on push events for matching branches only. | | `push_events` | boolean | **{dotted-circle}** No | Trigger hook on push events. | +| `releases_events` | boolean | **{dotted-circle}** No | Trigger hook on release events. | | `tag_push_events` | boolean | **{dotted-circle}** No | Trigger hook on tag push events. | | `token` | string | **{dotted-circle}** No | Secret token to validate received payloads; this isn't returned in the response. | -| `url` | string | **{check-circle}** Yes | The hook URL. | | `wiki_page_events` | boolean | **{dotted-circle}** No | Trigger hook on wiki page events. | -| `releases_events` | boolean | **{dotted-circle}** No | Trigger hook on release events. | ### Delete project hook @@ -2478,8 +2469,8 @@ GET /projects | Attribute | Type | Required | Description | |------------|--------|------------------------|-------------| -| `order_by` | string | **{dotted-circle}** No | Return requests ordered by `id`, `name`, `created_at` or `last_activity_at` fields. | | `search` | string | **{check-circle}** Yes | A string contained in the project name. | +| `order_by` | string | **{dotted-circle}** No | Return requests ordered by `id`, `name`, `created_at` or `last_activity_at` fields. | | `sort` | string | **{dotted-circle}** No | Return requests sorted in `asc` or `desc` order. | ```shell @@ -2496,11 +2487,11 @@ POST /projects/:id/housekeeping |-----------|----------------|------------------------|-------------| | `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding). | -## Push Rules **(PREMIUM)** +## Push rules **(PREMIUM)** ### Get project push rules -Get the [push rules](../push_rules/push_rules.md#enabling-push-rules) of a +Get the [push rules](../user/project/repository/push_rules.md#enabling-push-rules) of a project. ```plaintext @@ -2554,6 +2545,7 @@ POST /projects/:id/push_rule | Attribute | Type | Required | Description | |-----------------------------------------|----------------|------------------------|-------------| +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding). | | `author_email_regex` | string | **{dotted-circle}** No | All commit author emails must match this, for example `@my-company.com$`. | | `branch_name_regex` | string | **{dotted-circle}** No | All branch names must match this, for example `(feature|hotfix)\/*`. | | `commit_committer_check` | boolean | **{dotted-circle}** No | Users can only push commits to this repository that were committed with one of their own verified emails. | @@ -2561,7 +2553,6 @@ POST /projects/:id/push_rule | `commit_message_regex` | string | **{dotted-circle}** No | All commit messages must match this, for example `Fixed \d+\..*`. | | `deny_delete_tag` | boolean | **{dotted-circle}** No | Deny deleting a tag. | | `file_name_regex` | string | **{dotted-circle}** No | All committed filenames must **not** match this, for example `(jar|exe)$`. | -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding), | | `max_file_size` | integer | **{dotted-circle}** No | Maximum file size (MB). | | `member_check` | boolean | **{dotted-circle}** No | Restrict commits by author (email) to existing GitLab users. | | `prevent_secrets` | boolean | **{dotted-circle}** No | GitLab rejects any files that are likely to contain secrets. | @@ -2577,6 +2568,7 @@ PUT /projects/:id/push_rule | Attribute | Type | Required | Description | |-----------------------------------------|----------------|------------------------|-------------| +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding). | | `author_email_regex` | string | **{dotted-circle}** No | All commit author emails must match this, for example `@my-company.com$`. | | `branch_name_regex` | string | **{dotted-circle}** No | All branch names must match this, for example `(feature|hotfix)\/*`. | | `commit_committer_check` | boolean | **{dotted-circle}** No | Users can only push commits to this repository that were committed with one of their own verified emails. | @@ -2584,7 +2576,6 @@ PUT /projects/:id/push_rule | `commit_message_regex` | string | **{dotted-circle}** No | All commit messages must match this, for example `Fixed \d+\..*`. | | `deny_delete_tag` | boolean | **{dotted-circle}** No | Deny deleting a tag. | | `file_name_regex` | string | **{dotted-circle}** No | All committed filenames must **not** match this, for example `(jar|exe)$`. | -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding). | | `max_file_size` | integer | **{dotted-circle}** No | Maximum file size (MB). | | `member_check` | boolean | **{dotted-circle}** No | Restrict commits by author (email) to existing GitLab users. | | `prevent_secrets` | boolean | **{dotted-circle}** No | GitLab rejects any files that are likely to contain secrets. | @@ -2607,7 +2598,8 @@ DELETE /projects/:id/push_rule ## Transfer a project to a new namespace -> Introduced in GitLab 11.1. +See the [Project documentation](../user/project/settings/index.md#transferring-an-existing-project-into-another-namespace) +for prerequisites to transfer a project. ```plaintext PUT /projects/:id/transfer @@ -2736,7 +2728,7 @@ Example response: Read more in the [Branches](branches.md) documentation. -## Project Import/Export +## Project import/export Read more in the [Project import/export](project_import_export.md) documentation. @@ -2750,8 +2742,7 @@ Read more in the [Project vulnerabilities](project_vulnerabilities.md) documenta ## Configure pull mirroring for a project **(PREMIUM)** -> - Introduced in GitLab 11. -> - Moved to GitLab Premium in 13.9. +> Moved to GitLab Premium in 13.9. Configure pull mirroring while [creating a new project](#create-project) or [updating an existing project](#edit-project) using the API @@ -2791,8 +2782,6 @@ Read more in the [Project Badges](project_badges.md) documentation. ## Download snapshot of a Git repository -> Introduced in GitLab 10.7 - This endpoint may only be accessed by an administrative user. Download a snapshot of the project (or wiki, if requested) Git repository. This diff --git a/doc/api/repository_files.md b/doc/api/repository_files.md index fd024240e90..23acebf5aab 100644 --- a/doc/api/repository_files.md +++ b/doc/api/repository_files.md @@ -185,17 +185,17 @@ This allows you to create a single file. For creating multiple files with a sing POST /projects/:id/repository/files/:file_path ``` -| Attribute | Type | Required | Description | -|------------------|----------------|----------|-----------------------------------------------------------------------------------------------------------------| -| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | -| `file_path` | string | yes | URL encoded full path to new file. Ex. `lib%2Fclass%2Erb`. | -| `branch` | string | yes | Name of the branch | -| `start_branch` | string | no | Name of the branch to start the new commit from | -| `encoding` | string | no | Change encoding to `base64`. Default is `text`. | -| `author_email` | string | no | Specify the commit author's email address | -| `author_name` | string | no | Specify the commit author's name | -| `content` | string | yes | File content | -| `commit_message` | string | yes | Commit message | +| Attribute | Type | Required | Description | +| ---------------- | -------------- | -------- | ----------- | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | +| `file_path` | string | yes | URL-encoded full path to new file. For example: `lib%2Fclass%2Erb`. | +| `branch` | string | yes | Name of the new branch to create. The commit is added to this branch. | +| `start_branch` | string | no | Name of the base branch to create the new branch from. | +| `encoding` | string | no | Change encoding to `base64`. Default is `text`. | +| `author_email` | string | no | The commit author's email address. | +| `author_name` | string | no | The commit author's name. | +| `content` | string | yes | The file's content. | +| `commit_message` | string | yes | The commit message. | ```shell curl --request POST --header 'PRIVATE-TOKEN: <your_access_token>' \ @@ -222,18 +222,18 @@ This allows you to update a single file. For updating multiple files with a sing PUT /projects/:id/repository/files/:file_path ``` -| Attribute | Type | Required | Description | -|------------------|----------------|----------|-----------------------------------------------------------------------------------------------------------------| +| Attribute | Type | Required | Description | +| ---------------- | -------------- | -------- | ----------- | | `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | -| `file_path` | string | yes | URL encoded full path to new file. Ex. `lib%2Fclass%2Erb`. | -| `branch` | string | yes | Name of the branch | -| `start_branch` | string | no | Name of the branch to start the new commit from | -| `encoding` | string | no | Change encoding to `base64`. Default is `text`. | -| `author_email` | string | no | Specify the commit author's email address | -| `author_name` | string | no | Specify the commit author's name | -| `content` | string | yes | File content | -| `commit_message` | string | yes | Commit message | -| `last_commit_id` | string | no | Last known file commit ID | +| `file_path` | string | yes | URL-encoded full path to new file. For example: `lib%2Fclass%2Erb`. | +| `branch` | string | yes | Name of the new branch to create. The commit is added to this branch. | +| `start_branch` | string | no | Name of the base branch to create the new branch from. | +| `encoding` | string | no | Change encoding to `base64`. Default is `text`. | +| `author_email` | string | no | The commit author's email address. | +| `author_name` | string | no | The commit author's name. | +| `content` | string | yes | The file's content. | +| `commit_message` | string | yes | The commit message. | +| `last_commit_id` | string | no | Last known file commit ID. | ```shell curl --request PUT --header 'PRIVATE-TOKEN: <your_access_token>' \ @@ -270,16 +270,16 @@ This allows you to delete a single file. For deleting multiple files with a sing DELETE /projects/:id/repository/files/:file_path ``` -| Attribute | Type | Required | Description | -|------------------|----------------|----------|-----------------------------------------------------------------------------------------------------------------| -| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | -| `file_path` | string | yes | URL encoded full path to new file. Ex. `lib%2Fclass%2Erb`. | -| `branch` | string | yes | Name of the branch | -| `start_branch` | string | no | Name of the branch to start the new commit from | -| `author_email` | string | no | Specify the commit author's email address. | -| `author_name` | string | no | Specify the commit author's name. | -| `commit_message` | string | yes | Commit message. | -| `last_commit_id` | string | no | Last known file commit ID. | +| Attribute | Type | Required | Description | +| ---------------- | -------------- | -------- | ----------- | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | +| `file_path` | string | yes | URL-encoded full path to new file. For example: `lib%2Fclass%2Erb`. | +| `branch` | string | yes | Name of the new branch to create. The commit is added to this branch. | +| `start_branch` | string | no | Name of the base branch to create the new branch from. | +| `author_email` | string | no | The commit author's email address. | +| `author_name` | string | no | The commit author's name. | +| `commit_message` | string | yes | The commit message. | +| `last_commit_id` | string | no | Last known file commit ID. | ```shell curl --request DELETE --header 'PRIVATE-TOKEN: <your_access_token>' \ diff --git a/doc/api/runners.md b/doc/api/runners.md index e53062ce074..3423a02c078 100644 --- a/doc/api/runners.md +++ b/doc/api/runners.md @@ -39,27 +39,38 @@ Get a list of specific runners available to the user. GET /runners GET /runners?scope=active GET /runners?type=project_type -GET /runners?status=active +GET /runners?status=online +GET /runners?paused=true GET /runners?tag_list=tag1,tag2 ``` -| Attribute | Type | Required | Description | -|-------------|----------------|----------|---------------------| -| `scope` | string | no | Deprecated: Use `type` or `status` instead. The scope of specific runners to show, one of: `active`, `paused`, `online`, `offline`; showing all runners if none provided | -| `type` | string | no | The type of runners to show, one of: `instance_type`, `group_type`, `project_type` | -| `status` | string | no | The status of runners to show, one of: `active`, `paused`, `online`, `offline` | -| `tag_list` | string array | no | List of the runner's tags | +| Attribute | Type | Required | Description | +|------------|--------------|----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `scope` | string | no | Deprecated: Use `type` or `status` instead. The scope of specific runners to show, one of: `active`, `paused`, `online` and `offline`; showing all runners if none provided | +| `type` | string | no | The type of runners to show, one of: `instance_type`, `group_type`, `project_type` | +| `status` | string | no | The status of runners to show, one of: `online` and `offline`. `active` and `paused` are also possible values which were deprecated in GitLab 14.8 and will be removed in GitLab 15.0 | +| `paused` | boolean | no | Whether to include only runners that are accepting or ignoring new jobs | +| `tag_list` | string array | no | List of the runner's tags | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/runners" ``` +NOTE: +The `active` and `paused` values in the `status` query parameter were deprecated [in GitLab 14.8](https://gitlab.com/gitlab-org/gitlab/-/issues/347211). +and will be removed in [GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/351109). They are replaced by the `paused` query parameter. + +NOTE: +The `active` attribute in the response was deprecated [in GitLab 14.8](https://gitlab.com/gitlab-org/gitlab/-/issues/347211). +and will be removed in [GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/351109). It is replaced by the `paused` attribute. + Example response: ```json [ { "active": true, + "paused": false, "description": "test-1-20150125", "id": 6, "ip_address": "127.0.0.1", @@ -71,6 +82,7 @@ Example response: }, { "active": true, + "paused": false, "description": "test-2-20150125", "id": 8, "ip_address": "127.0.0.1", @@ -86,33 +98,44 @@ Example response: ## List all runners **(FREE SELF)** Get a list of all runners in the GitLab instance (specific and shared). Access -is restricted to users with the administrator role. +is restricted to users with administrator access. ```plaintext GET /runners/all GET /runners/all?scope=online GET /runners/all?type=project_type -GET /runners/all?status=active +GET /runners/all?status=online +GET /runners/all?paused=true GET /runners/all?tag_list=tag1,tag2 ``` -| Attribute | Type | Required | Description | -|-------------|----------------|----------|---------------------| -| `scope` | string | no | Deprecated: Use `type` or `status` instead. The scope of runners to show, one of: `specific`, `shared`, `active`, `paused`, `online`, `offline`; showing all runners if none provided | -| `type` | string | no | The type of runners to show, one of: `instance_type`, `group_type`, `project_type` | -| `status` | string | no | The status of runners to show, one of: `active`, `paused`, `online`, `offline` | -| `tag_list` | string array | no | List of the runner's tags | +| Attribute | Type | Required | Description | +|------------|--------------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `scope` | string | no | Deprecated: Use `type` or `status` instead. The scope of runners to show, one of: `specific`, `shared`, `active`, `paused`, `online` and `offline`; showing all runners if none provided | +| `type` | string | no | The type of runners to show, one of: `instance_type`, `group_type`, `project_type` | +| `status` | string | no | The status of runners to show, one of: `online` and `offline`. `active` and `paused` are also possible values which were deprecated in GitLab 14.8 and will be removed in GitLab 15.0 | +| `paused` | boolean | no | Whether to include only runners that are accepting or ignoring new jobs | +| `tag_list` | string array | no | List of the runner's tags | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/runners/all" ``` +NOTE: +The `active` and `paused` values in the `status` query parameter were deprecated [in GitLab 14.8](https://gitlab.com/gitlab-org/gitlab/-/issues/347211). +and will be removed in [GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/351109). They are replaced by the `paused` query parameter. + +NOTE: +The `active` attribute in the response was deprecated [in GitLab 14.8](https://gitlab.com/gitlab-org/gitlab/-/issues/347211). +and will be removed in [GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/351109). It is replaced by the `paused` attribute. + Example response: ```json [ { "active": true, + "paused": false, "description": "shared-runner-1", "id": 1, "ip_address": "127.0.0.1", @@ -124,6 +147,7 @@ Example response: }, { "active": true, + "paused": false, "description": "shared-runner-2", "id": 3, "ip_address": "127.0.0.1", @@ -135,6 +159,7 @@ Example response: }, { "active": true, + "paused": false, "description": "test-1-20150125", "id": 6, "ip_address": "127.0.0.1", @@ -146,6 +171,7 @@ Example response: }, { "active": true, + "paused": false, "description": "test-2-20150125", "id": 8, "ip_address": "127.0.0.1", @@ -164,7 +190,7 @@ To view more than the first 20 runners, use [pagination](index.md#pagination). Get details of a runner. -At least the [Maintainer role](../user/permissions.md) is required to get runner details at the +At least the Maintainer role is required to get runner details at the project and group level. Instance-level runner details via this endpoint are available to all signed in users. @@ -185,11 +211,16 @@ NOTE: The `token` attribute in the response was deprecated [in GitLab 12.10](https://gitlab.com/gitlab-org/gitlab/-/issues/214320). and removed in [GitLab 13.0](https://gitlab.com/gitlab-org/gitlab/-/issues/214322). +NOTE: +The `active` attribute in the response was deprecated [in GitLab 14.8](https://gitlab.com/gitlab-org/gitlab/-/issues/347211). +and will be removed in [GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/351109). It is replaced by the `paused` attribute. + Example response: ```json { "active": true, + "paused": false, "architecture": null, "description": "test-1-20150125", "id": 6, @@ -229,16 +260,17 @@ Update details of a runner. PUT /runners/:id ``` -| Attribute | Type | Required | Description | -|---------------|---------|----------|---------------------| -| `id` | integer | yes | The ID of a runner | -| `description` | string | no | The description of a runner | -| `active` | boolean | no | The state of a runner; can be set to `true` or `false` | -| `tag_list` | array | no | The list of tags for a runner; put array of tags, that should be finally assigned to a runner | -| `run_untagged`| boolean | no | Flag indicating the runner can execute untagged jobs | -| `locked` | boolean | no | Flag indicating the runner is locked | -| `access_level` | string | no | The access_level of the runner; `not_protected` or `ref_protected` | -| `maximum_timeout` | integer | no | Maximum timeout set when this runner handles the job | +| Attribute | Type | Required | Description | +|-------------------|---------|----------|--------------------------------------------------------------------------------------------------| +| `id` | integer | yes | The ID of a runner | +| `description` | string | no | The description of a runner | +| `active` | boolean | no | Deprecated: Use `:paused` instead. Flag indicating whether the runner is allowed to receive jobs | +| `paused` | boolean | no | Flag indicating whether the runner should ignore new jobs | +| `tag_list` | array | no | The list of tags for a runner; put array of tags, that should be finally assigned to a runner | +| `run_untagged` | boolean | no | Flag indicating the runner can execute untagged jobs | +| `locked` | boolean | no | Flag indicating the runner is locked | +| `access_level` | string | no | The access_level of the runner; `not_protected` or `ref_protected` | +| `maximum_timeout` | integer | no | Maximum timeout set when this runner handles the job | ```shell curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/runners/6" \ @@ -249,6 +281,10 @@ NOTE: The `token` attribute in the response was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/214320) in GitLab 12.10. and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/214322) in GitLab 13.0. +NOTE: +The `active` query parameter was deprecated [in GitLab 14.8](https://gitlab.com/gitlab-org/gitlab/-/issues/347211). +and will be removed in [GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/351109). It is replaced by the `paused` attribute. + Example response: ```json @@ -292,7 +328,12 @@ Example response: Pause a specific runner. ```plaintext -PUT --form "active=false" /runners/:runner_id +PUT --form "paused=true" /runners/:runner_id + +# --or-- + +# Deprecated: removal planned in 15.0 +PUT --form "active=false" /runners/:runner_id ``` | Attribute | Type | Required | Description | @@ -301,9 +342,19 @@ PUT --form "active=false" /runners/:runner_id ```shell curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" \ + --form "paused=true" "https://gitlab.example.com/api/v4/runners/6" + +# --or-- + +# Deprecated: removal planned in 15.0 +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" \ --form "active=false" "https://gitlab.example.com/api/v4/runners/6" ``` +NOTE: +The `active` form attribute was deprecated [in GitLab 14.8](https://gitlab.com/gitlab-org/gitlab/-/issues/347211). +and will be removed in [GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/351109). It is replaced by the `paused` attribute. + ## List runner's jobs > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/15432) in GitLab 10.3. @@ -397,35 +448,45 @@ Example response: ## List project's runners -List all runners (specific and shared) available in the project. Shared runners -are listed if at least one shared runner is defined. +List all runners available in the project, including from ancestor groups and [any allowed shared runners](../ci/runners/runners_scope.md#enable-shared-runners). ```plaintext GET /projects/:id/runners GET /projects/:id/runners?scope=active GET /projects/:id/runners?type=project_type -GET /projects/:id/runners?status=active +GET /projects/:id/runners/all?status=online +GET /projects/:id/runners/all?paused=true GET /projects/:id/runners?tag_list=tag1,tag2 ``` -| Attribute | Type | Required | Description | -|------------|----------------|----------|---------------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | -| `scope` | string | no | Deprecated: Use `type` or `status` instead. The scope of specific runners to show, one of: `active`, `paused`, `online`, `offline`; showing all runners if none provided | -| `type` | string | no | The type of runners to show, one of: `instance_type`, `group_type`, `project_type` | -| `status` | string | no | The status of runners to show, one of: `active`, `paused`, `online`, `offline` | -| `tag_list` | string array | no | List of the runner's tags | +| Attribute | Type | Required | Description | +|------------|----------------|----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `scope` | string | no | Deprecated: Use `type` or `status` instead. The scope of specific runners to show, one of: `active`, `paused`, `online` and `offline`; showing all runners if none provided | +| `type` | string | no | The type of runners to show, one of: `instance_type`, `group_type`, `project_type` | +| `status` | string | no | The status of runners to show, one of: `online` and `offline`. `active` and `paused` are also possible values which were deprecated in GitLab 14.8 and will be removed in GitLab 15.0 | +| `paused` | boolean | no | Whether to include only runners that are accepting or ignoring new jobs | +| `tag_list` | string array | no | List of the runner's tags | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/9/runners" ``` +NOTE: +The `active` and `paused` values in the `status` query parameter were deprecated [in GitLab 14.8](https://gitlab.com/gitlab-org/gitlab/-/issues/347211). +and will be removed in [GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/351109). They are replaced by the `paused` query parameter. + +NOTE: +The `active` attribute in the response was deprecated [in GitLab 14.8](https://gitlab.com/gitlab-org/gitlab/-/issues/347211). +and will be removed in [GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/351109). It is replaced by the `paused` attribute. + Example response: ```json [ { "active": true, + "paused": false, "description": "test-2-20150125", "id": 8, "ip_address": "127.0.0.1", @@ -437,6 +498,7 @@ Example response: }, { "active": true, + "paused": false, "description": "development_runner", "id": 5, "ip_address": "127.0.0.1", @@ -444,7 +506,7 @@ Example response: "runner_type": "instance_type", "name": null, "online": true, - "status": "paused" + "status": "online" } ] ``` @@ -504,27 +566,36 @@ curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://git ## List group's runners -List all runners (specific and shared) available in the group as well it's ancestor groups. -Shared runners are listed if at least one shared runner is defined. +List all runners available in the group as well as its ancestor groups, including [any allowed shared runners](../ci/runners/runners_scope.md#enable-shared-runners). ```plaintext GET /groups/:id/runners GET /groups/:id/runners?type=group_type -GET /groups/:id/runners?status=active +GET /groups/:id/runners/all?status=online +GET /groups/:id/runners/all?paused=true GET /groups/:id/runners?tag_list=tag1,tag2 ``` -| Attribute | Type | Required | Description | -|------------|----------------|----------|---------------------| -| `id` | integer | yes | The ID of the group owned by the authenticated user | -| `type` | string | no | The type of runners to show, one of: `instance_type`, `group_type`, `project_type` | -| `status` | string | no | The status of runners to show, one of: `active`, `paused`, `online`, `offline` | -| `tag_list` | string array | no | List of the runner's tags | +| Attribute | Type | Required | Description | +|------------|----------------|----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `id` | integer | yes | The ID of the group owned by the authenticated user | +| `type` | string | no | The type of runners to show, one of: `instance_type`, `group_type`, `project_type`. The `project_type` value is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/351466) and will be removed in GitLab 15.0 | +| `status` | string | no | The status of runners to show, one of: `online` and `offline`. `active` and `paused` are also possible values which were deprecated in GitLab 14.8 and will be removed in GitLab 15.0 | +| `paused` | boolean | no | Whether to include only runners that are accepting or ignoring new jobs | +| `tag_list` | string array | no | List of the runner's tags | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/9/runners" ``` +NOTE: +The `active` and `paused` values in the `status` query parameter were deprecated [in GitLab 14.8](https://gitlab.com/gitlab-org/gitlab/-/issues/347211). +and will be removed in [GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/351109). They are replaced by the `paused` query parameter. + +NOTE: +The `active` attribute in the response was deprecated [in GitLab 14.8](https://gitlab.com/gitlab-org/gitlab/-/issues/347211). +and will be removed in [GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/351109). It is replaced by the `paused` attribute. + Example response: ```json @@ -534,6 +605,7 @@ Example response: "description": "Shared", "ip_address": "127.0.0.1", "active": true, + "paused": false, "is_shared": true, "runner_type": "instance_type", "name": "gitlab-runner", @@ -545,6 +617,7 @@ Example response: "description": "Test", "ip_address": "127.0.0.1", "active": true, + "paused": false, "is_shared": true, "runner_type": "instance_type", "name": "gitlab-runner", @@ -556,6 +629,7 @@ Example response: "description": "Test 2", "ip_address": "127.0.0.1", "active": true, + "paused": false, "is_shared": false, "runner_type": "group_type", "name": "gitlab-runner", @@ -573,18 +647,20 @@ Register a new runner for the instance. POST /runners ``` -| Attribute | Type | Required | Description | -|-------------------|----------------|----------|----------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `token` | string | yes | [Registration token](#registration-and-authentication-tokens). | -| `description` | string | no | Runner's description | -| `info` | hash | no | Runner's metadata. You can include `name`, `version`, `revision`, `platform`, and `architecture`, but only `version` is displayed in the Admin area of the UI. | -| `active` | boolean | no | Whether the runner is active | -| `locked` | boolean | no | Whether the runner should be locked for current project | -| `run_untagged` | boolean | no | Whether the runner should handle untagged jobs | -| `tag_list` | string array | no | List of runner's tags | -| `access_level` | string | no | The access_level of the runner; `not_protected` or `ref_protected` | -| `maximum_timeout` | integer | no | Maximum timeout set when this runner handles the job | -| `maintainer_note` | string | no | Free-form maintainer notes for the runner (255 characters) | +| Attribute | Type | Required | Description | +|--------------------|--------------|----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `token` | string | yes | [Registration token](#registration-and-authentication-tokens) | +| `description` | string | no | Runner's description | +| `info` | hash | no | Runner's metadata. You can include `name`, `version`, `revision`, `platform`, and `architecture`, but only `version` is displayed in the Admin area of the UI | +| `active` | boolean | no | Deprecated: Use `:paused` instead. Whether the runner is allowed to receive jobs | +| `paused` | boolean | no | Whether the runner should ignore new jobs | +| `locked` | boolean | no | Whether the runner should be locked for current project | +| `run_untagged` | boolean | no | Whether the runner should handle untagged jobs | +| `tag_list` | string array | no | List of runner's tags | +| `access_level` | string | no | The access_level of the runner; `not_protected` or `ref_protected` | +| `maximum_timeout` | integer | no | Maximum timeout set when this runner handles the job | +| `maintainer_note` | string | no | [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/350730), see `maintenance_note` | +| `maintenance_note` | string | no | Free-form maintenance notes for the runner (255 characters) | ```shell curl --request POST "https://gitlab.example.com/api/v4/runners" \ @@ -603,7 +679,8 @@ Example response: ```json { "id": 12345, - "token": "6337ff461c94fd3fa32ba3b1ff4125" + "token": "6337ff461c94fd3fa32ba3b1ff4125", + "token_expires_at": "2021-09-27T21:05:03.203Z" } ``` @@ -743,6 +820,7 @@ Example response: ```json { - "token": "6337ff461c94fd3fa32ba3b1ff4125" + "token": "6337ff461c94fd3fa32ba3b1ff4125", + "token_expires_at": "2021-09-27T21:05:03.203Z" } ``` diff --git a/doc/api/scim.md b/doc/api/scim.md index acc6a6ae686..ab3a181f5be 100644 --- a/doc/api/scim.md +++ b/doc/api/scim.md @@ -1,7 +1,7 @@ --- type: reference, howto stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/api/settings.md b/doc/api/settings.md index 2ed841b885c..4246c7a46f1 100644 --- a/doc/api/settings.md +++ b/doc/api/settings.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -65,6 +65,8 @@ Example response: "dsa_key_restriction": 0, "ecdsa_key_restriction": 0, "ed25519_key_restriction": 0, + "ecdsa_sk_key_restriction": 0, + "ed25519_sk_key_restriction": 0, "first_day_of_week": 0, "enforce_terms": true, "terms": "Hello world!", @@ -166,6 +168,8 @@ Example response: "dsa_key_restriction": 0, "ecdsa_key_restriction": 0, "ed25519_key_restriction": 0, + "ecdsa_sk_key_restriction": 0, + "ed25519_sk_key_restriction": 0, "first_day_of_week": 0, "enforce_terms": true, "terms": "Hello world!", @@ -268,7 +272,9 @@ listed in the descriptions of the relevant settings. | `domain_allowlist` | array of strings | no | Force people to use only corporate emails for sign-up. Default is `null`, meaning there is no restriction. | | `dsa_key_restriction` | integer | no | The minimum allowed bit length of an uploaded DSA key. Default is `0` (no restriction). `-1` disables DSA keys. | | `ecdsa_key_restriction` | integer | no | The minimum allowed curve size (in bits) of an uploaded ECDSA key. Default is `0` (no restriction). `-1` disables ECDSA keys. | +| `ecdsa_sk_key_restriction` | integer | no | The minimum allowed curve size (in bits) of an uploaded ECDSA_SK key. Default is `0` (no restriction). `-1` disables ECDSA_SK keys. | | `ed25519_key_restriction` | integer | no | The minimum allowed curve size (in bits) of an uploaded ED25519 key. Default is `0` (no restriction). `-1` disables ED25519 keys. | +| `ed25519_sk_key_restriction` | integer | no | The minimum allowed curve size (in bits) of an uploaded ED25519_SK key. Default is `0` (no restriction). `-1` disables ED25519_SK keys. | | `eks_access_key_id` | string | no | AWS IAM access key ID. | | `eks_account_id` | string | no | Amazon account ID. | | `eks_integration_enabled` | boolean | no | Enable integration with Amazon EKS. | @@ -370,6 +376,7 @@ listed in the descriptions of the relevant settings. | `push_event_hooks_limit` | integer | no | Number of changes (branches or tags) in a single push to determine whether webhooks and services fire or not. Webhooks and services aren't submitted if it surpasses that value. | | `rate_limiting_response_text` | string | no | When rate limiting is enabled via the `throttle_*` settings, send this plain text response when a rate limit is exceeded. 'Retry later' is sent if this is blank. | | `raw_blob_request_limit` | integer | no | Max number of requests per minute for each raw path. Default: 300. To disable throttling set to 0.| +| `user_email_lookup_limit` | integer | no | Max number of requests per minute for email lookup. Default: 60. To disable throttling set to 0.| | `recaptcha_enabled` | boolean | no | (**If enabled, requires:** `recaptcha_private_key` and `recaptcha_site_key`) Enable reCAPTCHA. | | `recaptcha_private_key` | string | required by: `recaptcha_enabled` | Private key for reCAPTCHA. | | `recaptcha_site_key` | string | required by: `recaptcha_enabled` | Site key for reCAPTCHA. | diff --git a/doc/api/statistics.md b/doc/api/statistics.md index 59197260988..17daf2fc71f 100644 --- a/doc/api/statistics.md +++ b/doc/api/statistics.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/api/status_checks.md b/doc/api/status_checks.md index 7de188ad185..a3a342854dd 100644 --- a/doc/api/status_checks.md +++ b/doc/api/status_checks.md @@ -31,7 +31,7 @@ GET /projects/:id/merge_requests/:merge_request_iid/status_checks "id": 2, "name": "Rule 1", "external_url": "https://gitlab.com/test-endpoint", - "status": "approved" + "status": "pass" }, { "id": 1, @@ -47,6 +47,8 @@ GET /projects/:id/merge_requests/:merge_request_iid/status_checks For a single merge request, use the API to inform GitLab that a merge request has passed a check by an external service. To set the status of an external check, the personal access token used must belong to a user with at least the developer role on the target project of the merge request. +Execute this API call as any user with rights to approve the merge request itself. + ```plaintext POST /projects/:id/merge_requests/:merge_request_iid/status_check_responses ``` @@ -59,6 +61,7 @@ POST /projects/:id/merge_requests/:merge_request_iid/status_check_responses | `merge_request_iid` | integer | yes | IID of a merge request | | `sha` | string | yes | SHA at `HEAD` of the source branch | | `external_status_check_id` | integer | yes | ID of an external status check | +| `status` | string | no | Set to `pass` to pass the check | NOTE: `sha` must be the SHA at the `HEAD` of the merge request's source branch. diff --git a/doc/api/suggestions.md b/doc/api/suggestions.md index ea9baa79b4a..0a3e7d54f88 100644 --- a/doc/api/suggestions.md +++ b/doc/api/suggestions.md @@ -13,8 +13,8 @@ Every API call to suggestions must be authenticated. ## Applying suggestions -Applies a suggested patch in a merge request. Users must be -at least [Developer](../user/permissions.md) to perform such action. +Applies a suggested patch in a merge request. Users must have +at least the Developer role to perform such action. ```plaintext PUT /suggestions/:id/apply diff --git a/doc/api/tags.md b/doc/api/tags.md index 6aa40cf476d..903cb361587 100644 --- a/doc/api/tags.md +++ b/doc/api/tags.md @@ -22,7 +22,7 @@ Parameters: | `id` | integer/string| yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user| | `order_by` | string | no | Return tags ordered by `name` or `updated` fields. Default is `updated` | | `sort` | string | no | Return tags sorted in `asc` or `desc` order. Default is `desc` | -| `search` | string | no | Return list of tags matching the search criteria. You can use `^term` and `term$` to find tags that begin and end with `term` respectively. | +| `search` | string | no | Return list of tags matching the search criteria. You can use `^term` and `term$` to find tags that begin and end with `term` respectively. No other regular expressions are supported. | ```json [ diff --git a/doc/api/usage_data.md b/doc/api/usage_data.md index 4809166f357..c7064ebf65b 100644 --- a/doc/api/usage_data.md +++ b/doc/api/usage_data.md @@ -13,7 +13,7 @@ The Service Data API is associated with [Service Ping](../development/service_pi > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/57270) in GitLab 13.11. -Export all metric definitions as a single YAML file, similar to the [Metrics Dictionary](https://metrics.gitlab.com/index.html), for easier importing. +Export all metric definitions as a single YAML file, similar to the [Metrics Dictionary](https://metrics.gitlab.com/), for easier importing. ```plaintext GET /usage_data/metric_definitions diff --git a/doc/api/users.md b/doc/api/users.md index 9b79a249220..925563aeb1f 100644 --- a/doc/api/users.md +++ b/doc/api/users.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -78,13 +78,14 @@ GET /users?external=true GitLab supports bot users such as the [alert bot](../operations/incident_management/integrations.md) or the [support bot](../user/project/service_desk.md#support-bot-user). You can exclude the following types of [internal users](../development/internal_users.md#internal-users) -from the users' list, with the `exclude_internal=true` parameter, -([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241144) in GitLab 13.4). +from the users' list with the `exclude_internal=true` parameter +([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241144) in GitLab 13.4): - Alert bot - Support bot -However, this action does not exclude [project bot users](../user/project/settings/project_access_tokens.md#project-bot-users). +However, this action does not exclude [bot users for projects](../user/project/settings/project_access_tokens.md#bot-users-for-projects) +or [bot users for groups](../user/group/settings/group_access_tokens.md#bot-users-for-groups). ```plaintext GET /users?exclude_internal=true @@ -108,7 +109,7 @@ GET /users | `sort` | string | no | Return users sorted in `asc` or `desc` order. Default is `desc` | | `two_factor` | string | no | Filter users by Two-factor authentication. Filter values are `enabled` or `disabled`. By default it returns all users | | `without_projects` | boolean | no | Filter users without projects. Default is `false`, which means that all users are returned, with and without projects. | -| `admins` | boolean | no | Return only admin users. Default is `false` | +| `admins` | boolean | no | Return only administrators. Default is `false` | | `saml_provider_id` **(PREMIUM)** | number | no | Return only users created by the specified SAML provider ID. If not included, it returns all users. | ```json @@ -532,7 +533,7 @@ Parameters: | Attribute | Type | Required | Description | |---------------|---------|----------|----------------------------------------------| | `id` | integer | yes | The ID of a user | -| `hard_delete` | boolean | no | If true, contributions that would usually be [moved to the ghost user](../user/profile/account/delete_account.md#associated-records) are deleted instead, as well as groups owned solely by this user. | +| `hard_delete` | boolean | no | If true, contributions that would usually be [moved to Ghost User](../user/profile/account/delete_account.md#associated-records) are deleted instead, as well as groups owned solely by this user. | ## List current user (for normal users) @@ -696,6 +697,38 @@ Example response: } ``` +## Set user status + +Set the status of the current user. + +```plaintext +PUT /user/status +``` + +| Attribute | Type | Required | Description | +| -------------------- | ------ | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `emoji` | string | no | The name of the emoji to use as status. If omitted `speech_balloon` is used. Emoji name can be one of the specified names in the [Gemojione index](https://github.com/bonusly/gemojione/blob/master/config/index.json). | +| `message` | string | no | The message to set as a status. It can also contain emoji codes. | +| `clear_status_after` | string | no | Automatically clean up the status after a given time interval, allowed values: `30_minutes`, `3_hours`, `8_hours`, `1_day`, `3_days`, `7_days`, `30_days` + +When both parameters `emoji` and `message` are empty, the status is cleared. When the `clear_status_after` parameter is missing from the request, the previously set value for `"clear_status_after` is cleared. + +```shell +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --data "clear_status_after=1_day" --data "emoji=coffee" \ + --data "message=I crave coffee" "https://gitlab.example.com/api/v4/user/status" +``` + +Example responses + +```json +{ + "emoji":"coffee", + "message":"I crave coffee", + "message_html": "I crave coffee", + "clear_status_at":"2021-02-15T10:49:01.311Z" +} +``` + ## Get user preferences Get a list of currently authenticated user's preferences. @@ -743,38 +776,6 @@ Parameters: | `view_diffs_file_by_file` | Yes | Flag indicating the user sees only one file diff per page. | | `show_whitespace_in_diffs` | Yes | Flag indicating the user sees whitespace changes in diffs. | -## Set user status - -Set the status of the current user. - -```plaintext -PUT /user/status -``` - -| Attribute | Type | Required | Description | -| -------------------- | ------ | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `emoji` | string | no | The name of the emoji to use as status. If omitted `speech_balloon` is used. Emoji name can be one of the specified names in the [Gemojione index](https://github.com/bonusly/gemojione/blob/master/config/index.json). | -| `message` | string | no | The message to set as a status. It can also contain emoji codes. | -| `clear_status_after` | string | no | Automatically clean up the status after a given time interval, allowed values: `30_minutes`, `3_hours`, `8_hours`, `1_day`, `3_days`, `7_days`, `30_days` - -When both parameters `emoji` and `message` are empty, the status is cleared. When the `clear_status_after` parameter is missing from the request, the previously set value for `"clear_status_after` is cleared. - -```shell -curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --data "clear_status_after=1_day" --data "emoji=coffee" \ - --data "message=I crave coffee" "https://gitlab.example.com/api/v4/user/status" -``` - -Example responses - -```json -{ - "emoji":"coffee", - "message":"I crave coffee", - "message_html": "I crave coffee", - "clear_status_at":"2021-02-15T10:49:01.311Z" -} -``` - ## User Follow ### Follow and unfollow users @@ -1545,7 +1546,7 @@ Please refer to the [Events API documentation](events.md#get-user-contribution-e ## Get all impersonation tokens of a user -> Requires administrator permissions. +Requires administrator access. It retrieves every impersonation token of the user. Use the pagination parameters `page` and `per_page` to restrict the list of impersonation tokens. @@ -1719,8 +1720,8 @@ Example response: ## Create an impersonation token -> Requires administrator permissions. -> Token values are returned once. Make sure you save it - you can't access it again. +Requires administrator access. Token values are returned once. Make sure you save it because you can't access +it again. It creates a new impersonation token. Only administrators can do this. You are only able to create impersonation tokens to impersonate the user and perform @@ -1764,7 +1765,7 @@ Example response: ## Revoke an impersonation token -> Requires administrator permissions. +Requires administrator access. It revokes an impersonation token. @@ -1837,7 +1838,7 @@ The activities that update the timestamp are: - Git HTTP/SSH activities (such as clone, push) - User logging in to GitLab -- User visiting pages related to Dashboards, Projects, Issues, and Merge Requests ([introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/54947) in GitLab 11.8) +- User visiting pages related to dashboards, projects, issues, and merge requests ([introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/54947) in GitLab 11.8) - User using the API - User using the GraphQL API diff --git a/doc/api/v3_to_v4.md b/doc/api/v3_to_v4.md index da269073905..9c2170e0709 100644 --- a/doc/api/v3_to_v4.md +++ b/doc/api/v3_to_v4.md @@ -1,18 +1,9 @@ --- -stage: Ecosystem -group: Integrations -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'https://gitlab.com/gitlab-org/gitlab-foss/-/blob/11-0-stable/doc/api/v3_to_v4.md' +remove_date: '2023-01-31' --- -# API V3 to API V4 +This document was moved to [another location](https://gitlab.com/gitlab-org/gitlab-foss/-/blob/11-0-stable/doc/api/v3_to_v4.md). -WARNING: -The GitLab API v3 was [removed](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/36819) in GitLab 11.0. - -For information about the current version of the GitLab API, read the [API documentation](index.md). - -## Related topics - -- [GitLab v3 API documentation](https://gitlab.com/gitlab-org/gitlab-foss/-/blob/8-16-stable/doc/api/index.md) -- [Migration guide](https://gitlab.com/gitlab-org/gitlab-foss/-/blob/11-0-stable/doc/api/v3_to_v4.md) from - v3 to v4 +<!-- This redirect file can be deleted after <2023-01-31>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/api/visual_review_discussions.md b/doc/api/visual_review_discussions.md index 8457f63fd38..7f91b829061 100644 --- a/doc/api/visual_review_discussions.md +++ b/doc/api/visual_review_discussions.md @@ -1,8 +1,7 @@ --- stage: Verify -group: Testing +group: Pipeline Insights info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference, api --- # Visual Review discussions API **(PREMIUM)** @@ -10,7 +9,7 @@ type: reference, api > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18710) in GitLab 12.5. > - [Moved](https://about.gitlab.com/blog/2021/01/26/new-gitlab-product-subscription-model/) to GitLab Premium in 13.9. -Visual Review discussions are notes on Merge Requests sent as +Visual Review discussions are notes on merge requests sent as feedback from [Visual Reviews](../ci/review_apps/index.md#visual-reviews). ## Create new merge request thread diff --git a/doc/architecture/blueprints/ci_data_decay/index.md b/doc/architecture/blueprints/ci_data_decay/index.md index 155c781b04a..cbd9f5dea7a 100644 --- a/doc/architecture/blueprints/ci_data_decay/index.md +++ b/doc/architecture/blueprints/ci_data_decay/index.md @@ -74,13 +74,13 @@ we might want to follow these three tracks described below. <!-- markdownlint-disable MD029 --> -1. Partition builds queuing tables -2. Archive CI/CD data into partitioned database schema -3. Migrate archived builds metadata out of primary database +1. Partition CI/CD builds queuing database tables +2. Partition CI/CD pipelines database tables +3. Reduce the rate of builds metadata table growth <!-- markdownlint-enable MD029 --> -### Migrate archived builds metadata out of primary database +### Reduce the rate of builds metadata table growth Once a build (or a pipeline) gets archived, it is no longer possible to resume pipeline processing in such pipeline. It means that all the metadata, we store @@ -98,15 +98,16 @@ be able to use de-duplication of metadata entries and other normalization strategies to consume less storage while retaining ability to query this dataset. Technical evaluation will be required to find the best solution here. -Epic: [Migrate archived builds metadata out of primary database](https://gitlab.com/groups/gitlab-org/-/epics/7216). +Epic: [Reduce the rate of builds metadata table growth](https://gitlab.com/groups/gitlab-org/-/epics/7434). -### Archive CI/CD data into partitioned database schema +### Partition CI/CD pipelines database tables -After we move CI/CD metadata to a different store, the problem of having -billions of rows describing pipelines, builds and artifacts, remains. We still -need to keep reference to the metadata we store in object storage and we still -do need to be able to retrieve this information reliably in bulk (or search -through it). +After we move CI/CD metadata to a different store, or reduce the rate of +metadata growth in a different way, the problem of having billions of rows +describing pipelines, builds and artifacts, remains. We still need to keep +reference to the metadata we might store in object storage and we still do need +to be able to retrieve this information reliably in bulk (or search through +it). It means that by moving data to object storage we might not be able to reduce the number of rows in CI/CD tables. Moving data to object storage should help @@ -132,9 +133,9 @@ partitioning on the application level. Partitioning rarely accessed data should also follow the policy defined for builds archival, to make it consistent and reliable. -Epic: [Archive CI/CD data into partitioned database schema](https://gitlab.com/groups/gitlab-org/-/epics/5417). +Epic: [Partition CI/CD pipelines database tables](https://gitlab.com/groups/gitlab-org/-/epics/5417). -### Partition builds queuing tables +### Partition CI/CD builds queuing database tables While working on the [CI/CD Scale](../ci_scale/index.md) blueprint, we have introduced a [new architecture for queuing CI/CD builds](https://gitlab.com/groups/gitlab-org/-/epics/5909#note_680407908) @@ -156,7 +157,7 @@ for builds archival. Instead we should leverage a long-standing policy saying that builds created more 24 hours ago need to be removed from the queue. This business rule is present in the product since the inception of GitLab CI. -Epic: [Partition builds queuing tables](https://gitlab.com/gitlab-org/gitlab/-/issues/347027). +Epic: [Partition CI/CD builds queuing database tables](https://gitlab.com/groups/gitlab-org/-/epics/7438). ## Principles @@ -215,9 +216,9 @@ pipelines data, although a user provided partition identifier may be required. All three tracks can be worked on in parallel: -1. [Migrate archived build metadata to object storage](https://gitlab.com/groups/gitlab-org/-/epics/7216). -1. [Partition CI/CD data that have been archived](https://gitlab.com/groups/gitlab-org/-/epics/5417). -1. [Partition CI/CD queuing tables using list partitioning](https://gitlab.com/gitlab-org/gitlab/-/issues/347027) +1. [Reduce the rate of builds metadata table growth](https://gitlab.com/groups/gitlab-org/-/epics/7434). +1. [Partition CI/CD pipelines database tables](https://gitlab.com/groups/gitlab-org/-/epics/5417). +1. [Partition CI/CD queuing tables using list partitioning](https://gitlab.com/groups/gitlab-org/-/epics/7438) ## Status diff --git a/doc/architecture/blueprints/consolidating_groups_and_projects/index.md b/doc/architecture/blueprints/consolidating_groups_and_projects/index.md index 345160dc77f..6040ac1e50f 100644 --- a/doc/architecture/blueprints/consolidating_groups_and_projects/index.md +++ b/doc/architecture/blueprints/consolidating_groups_and_projects/index.md @@ -131,7 +131,7 @@ epic. The initial iteration will provide a framework to house features under `Namespaces`. Stage groups will eventually need to migrate their own features and functionality over to `Namespaces`. This may impact these features in unexpected ways. Therefore, to minimize UX debt and maintain product consistency, stage groups will have to consider a number of factors when migrating their features over to `Namespaces`: -1. **Conceptual model**: What are the current and future state conceptual models of these features ([see object modeling for designers](https://hpadkisson.medium.com/object-modeling-for-designers-an-introduction-7871bdcf8baf))? These should be documented in Pajamas (example: [Merge Requests](https://design.gitlab.com/objects/merge-request)). +1. **Conceptual model**: What are the current and future state conceptual models of these features ([see object modeling for designers](https://hpadkisson.medium.com/object-modeling-for-designers-an-introduction-7871bdcf8baf))? These should be documented in Pajamas (example: [merge requests](https://design.gitlab.com/objects/merge-request)). 1. **Merge conflicts**: What inconsistencies are there across project, group, and admin levels? How might these be addressed? For an example of how we rationalized this for labels, please see [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/338820). 1. **Inheritance & information flow**: How is information inherited across our container hierarchy currently? How might this be impacted if complying with the new [inheritance behavior](https://gitlab.com/gitlab-org/gitlab/-/issues/343316) framework? 1. **Settings**: Where can settings for this feature be found currently? How will these be impacted by `Namespaces`? diff --git a/doc/architecture/blueprints/database_testing/index.md b/doc/architecture/blueprints/database_testing/index.md index 4676caab85d..8c0cb550d61 100644 --- a/doc/architecture/blueprints/database_testing/index.md +++ b/doc/architecture/blueprints/database_testing/index.md @@ -100,7 +100,7 @@ The short-term goal is detailed in [this epic](https://gitlab.com/groups/gitlab- ### Mid-term - Improved feedback, query testing and background migration testing -Mid-term, we plan to expand the level of detail the testing pipeline reports back to the Merge Request and expand its scope to cover query testing, too. By doing so, we use our experience from database code reviews and using thin-clone technology and bring this back closer to the GitLab workflow. Instead of reaching out to different tools (`postgres.ai`, `joe`, Slack, plan visualizations, and so on) we bring this back to GitLab and working directly on the Merge Request. +Mid-term, we plan to expand the level of detail the testing pipeline reports back to the merge requet and expand its scope to cover query testing, too. By doing so, we use our experience from database code reviews and using thin-clone technology and bring this back closer to the GitLab workflow. Instead of reaching out to different tools (`postgres.ai`, `joe`, Slack, plan visualizations, and so on) we bring this back to GitLab and working directly on the merge request. Secondly, we plan to cover background migrations testing, too. These are typically data migrations that are scheduled to run over a long period of time. The success of both the scheduling phase and the job execution phase typically depends a lot on data distribution - which only surfaces when running these migrations on actual production data. In order to become confident about a background migration, we plan to provide the following feedback: @@ -109,7 +109,7 @@ Secondly, we plan to cover background migrations testing, too. These are typical ### Long-term - incorporate into GitLab product -There are opportunities to discuss for extracting features from this into GitLab itself. For example, annotating the Merge Request with query examples and attaching feedback gathered from the testing run can become a first-class citizen instead of using Merge Request description and comments for it. We plan to evaluate those ideas as we see those being used in earlier phases and bring our experience back into the product. +There are opportunities to discuss for extracting features from this into GitLab itself. For example, annotating the merge request with query examples and attaching feedback gathered from the testing run can become a first-class citizen instead of using merge request description and comments for it. We plan to evaluate those ideas as we see those being used in earlier phases and bring our experience back into the product. ## An alternative discussed: Anonymization diff --git a/doc/architecture/blueprints/object_storage/index.md b/doc/architecture/blueprints/object_storage/index.md index a79374d60bd..7864c951eca 100644 --- a/doc/architecture/blueprints/object_storage/index.md +++ b/doc/architecture/blueprints/object_storage/index.md @@ -89,20 +89,23 @@ replaced by a mock implementation. Furthermore, the presence of a shared disk, both in CI and in local development, often hides broken implementations until we deploy on an HA environment. -Shipping MinIO as part of the product will reduce the differences +One consideration we can take is to investigate shipping MinIO as part of the product. This could reduce the differences between a cloud and a local installation, standardizing our file storage on a single technology. -The removal of local disk operations will reduce the complexity of +The removal of local disk operations would reduce the complexity of development as well as mitigate several security attack vectors as we no longer write user-provided data on the local storage. -It will also reduce human errors as we will always run a local object +It would also reduce human errors as we will always run a local object storage in development mode and any local file disk access should raise a red flag during the merge request review. This effort is described in [this epic](https://gitlab.com/groups/gitlab-org/-/epics/6099). +Before considering any specific third-party technology, the +open source software licensing implications should be considered. As of 23 April 2021, [MinIO is subject to the AGPL v3 license](https://github.com/minio/minio/commit/069432566fcfac1f1053677cc925ddafd750730a). GitLab Legal must be consulted before any decision is taken to ship MinIO as proposed in this blueprint. + ### Enable direct upload by default on every upload Because every group of features requires its own bucket, we don't have diff --git a/doc/architecture/blueprints/runner_scaling/index.md b/doc/architecture/blueprints/runner_scaling/index.md index 8e47b5fda8c..174fe191cc7 100644 --- a/doc/architecture/blueprints/runner_scaling/index.md +++ b/doc/architecture/blueprints/runner_scaling/index.md @@ -44,7 +44,7 @@ and the documentation for it has been removed from the official page. This means that the original reason to use Docker Machine is no longer valid too. To keep supporting our customers and the wider community we need to design a -new mechanism for GitLab Runner autoscaling. It not only needs to support +new mechanism for GitLab Runner auto-scaling. It not only needs to support auto-scaling, but it also needs to do that in the way to enable us to build on top of it to improve efficiency, reliability and availability. @@ -144,7 +144,7 @@ on a single machine bring. It is difficult to predict that, so ideally we should build a PoC that will help us to better understand what we can expect from this. -To run this experiement we most likely we will need to build an experimental +To run this experiment we most likely we will need to build an experimental plugin, that not only allows us to schedule running multiple builds on a single machine, but also has a set of comprehensive metrics built into it, to make it easier to understand how it performs. @@ -204,6 +204,44 @@ document, define requirements and score the solution accordingly. This will allow us to choose a solution that will work best for us and the wider community. +### Design principles + +Our goal is to design a GitLab Runner plugin system interface that is flexible +and simple for the wider community to consume. As we cannot build plugins for +all cloud platforms, we want to ensure a low entry barrier for anyone who needs +to develop a plugin. We want to allow everyone to contribute. + +To achieve this goal, we will follow a few critical design principles. These +principles will guide our development process for the new plugin system +abstraction. + +#### General high-level principles + +1. Design the new auto-scaling architecture aiming for having more choices and + flexibility in the future, instead of imposing new constraints. +1. Design the new auto-scaling architecture to experiment with running multiple + jobs in parallel, on a single machine. +1. Design the new provisioning architecture to replace Docker Machine in a way + that the wider community can easily build on top of the new abstractions. + +#### Principles for the new plugin system + +1. Make the entry barrier for writing a new plugin low. +1. Developing a new plugin should be simple and require only basic knowledge of + a programming language and a cloud provider's API. +1. Strive for a balance between the plugin system's simplicity and flexibility. + These are not mutually exclusive. +1. Abstract away as many technical details as possible but do not hide them completely. +1. Build an abstraction that serves our community well but allows us to ship it quickly. +1. Invest in a flexible solution, avoid one-way-door decisions, foster iteration. +1. When in doubts err on the side of making things more simple for the wider community. + +#### The most important technical details + +1. Favor gRPC communication between a plugin and GitLab Runner. +1. Make it possible to version communication interface and support many versions. +1. Make Go a primary language for writing plugins but accept other languages too. + ## Status Status: RFC. diff --git a/doc/ci/chatops/index.md b/doc/ci/chatops/index.md index 698b467a813..c785e2f29ea 100644 --- a/doc/ci/chatops/index.md +++ b/doc/ci/chatops/index.md @@ -43,7 +43,7 @@ job completes: [Slack API](https://api.slack.com/) to send data to the channel. To use the `run` command, you must have at least the -[Developer role](../../user/permissions.md#project-members-permissions). +Developer role. If a job shouldn't be able to be triggered from chat, you can set the job to `except: [chat]`. ## Best practices for ChatOps CI jobs @@ -55,7 +55,7 @@ functions available. Consider these best practices when creating ChatOps jobs: of the standard CI pipeline. - If the job is set to `when: manual`, ChatOps creates the pipeline, but the job waits to be started. - ChatOps provides limited support for access control. If the user triggering the - slash command has at least the [Developer role](../../user/permissions.md#project-members-permissions) + slash command has at least the Developer role in the project, the job runs. The job itself can use existing [CI/CD variables](../variables/index.md#predefined-cicd-variables) like `GITLAB_USER_ID` to perform additional rights validation, but diff --git a/doc/ci/ci_cd_for_external_repos/github_integration.md b/doc/ci/ci_cd_for_external_repos/github_integration.md index 4e4b7420697..a906f213626 100644 --- a/doc/ci/ci_cd_for_external_repos/github_integration.md +++ b/doc/ci/ci_cd_for_external_repos/github_integration.md @@ -16,7 +16,7 @@ Watch a video on [Using GitLab CI/CD pipelines with GitHub repositories](https:/ NOTE: Because of [GitHub limitations](https://gitlab.com/gitlab-org/gitlab/-/issues/9147), -[GitHub OAuth](../../integration/github.md#enabling-github-oauth) +[GitHub OAuth](../../integration/github.md#enable-github-oauth-in-gitlab) cannot be used to authenticate with GitHub as an external CI/CD repository. ## Connect with Personal Access Token diff --git a/doc/ci/cloud_services/aws/index.md b/doc/ci/cloud_services/aws/index.md index f09b23db2c5..aa38562c866 100644 --- a/doc/ci/cloud_services/aws/index.md +++ b/doc/ci/cloud_services/aws/index.md @@ -1,6 +1,6 @@ --- -stage: Configure -group: Configure +stage: Verify +group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -30,7 +30,7 @@ Include the following information: After you create the identity provider, configure a [web identity role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-idp_oidc.html) with conditions for limiting access to GitLab resources. Temporary credentials are obtained using [AWS Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html), so set the `Action` to [sts:AssumeRoleWithWebIdentity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRoleWithWebIdentity.html). -For the full list of supported filtering types, see [Connect to cloud services](../index.md). +For the full list of supported filtering types, see [Connect to cloud services](../index.md#configure-a-conditional-role-with-oidc-claims). ```json { diff --git a/doc/ci/cloud_services/google_cloud/index.md b/doc/ci/cloud_services/google_cloud/index.md new file mode 100644 index 00000000000..14928f91816 --- /dev/null +++ b/doc/ci/cloud_services/google_cloud/index.md @@ -0,0 +1,182 @@ +--- +stage: Verify +group: Pipeline Authoring +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Configure OpenID Connect with GCP Workload Identity Federation + +WARNING: +The `CI_JOB_JWT_V2` variable is under development [(alpha)](../../../policy/alpha-beta-support.md#alpha-features) and is not yet suitable for production use. + +This tutorial demonstrates authenticating to Google Cloud from a GitLab CI/CD job +using a JSON Web Token (JWT) token and Workload Identity Federation. This configuration +generates on-demand, short-lived credentials without needing to store any secrets. + +To get started, configure OpenID Connect (OIDC) for identity federation between GitLab +and Google Cloud. For more information on using OIDC with GitLab, read +[Connect to cloud services](../index.md). + +This tutorial assumes you have a Google Cloud account and a Google Cloud project. +Your account must have at least the **Workload Identity Pool Admin** permission +on the Google Cloud project. + +To complete this tutorial: + +1. [Create the Google Cloud Workload Identity Pool](#create-the-google-cloud-workload-identity-pool). +1. [Create a Workload Identity Provider](#create-a-workload-identity-provider). +1. [Grant permissions for service account impersonation](#grant-permissions-for-service-account-impersonation). +1. [Retrieve a temporary credential](#retrieve-a-temporary-credential). + +## Create the Google Cloud Workload Identity Pool + +[Create a new Google Cloud Workload Identity Pool](https://cloud.google.com/iam/docs/configuring-workload-identity-federation#oidc) with the following options: + +- **Name**: Human-friendly name for the Workload Identity Pool, such as `GitLab`. +- **Pool ID**: Unique ID in the Google Cloud project for the Workload Identity Pool, + such as `gitlab`. This value is used to refer to the pool. and appears in URLs. +- **Description**: Optional. A description of the pool. +- **Enabled Pool**: Ensure this option is `true`. + +We recommend creating a single _pool_ per GitLab installation per Google Cloud project. If you have multiple GitLab repositories and CI/CD jobs on the same GitLab instance, they can authenticate using different _providers_ against the same _pool_. + +## Create a Workload Identity Provider + +[Create a new Google Cloud Workload Identity Provider](https://cloud.google.com/iam/docs/configuring-workload-identity-federation#create_the_workload_identity_pool_and_provider) +inside the Workload Identity Pool created in the previous step, using the following options: + +- **Provider type**: OpenID Connect (OIDC). +- **Provider name**: Human-friendly name for the Workload Identity Provider, + such as `gitlab/gitlab`. +- **Provider ID**: Unique ID in the pool for the Workload Identity Provider, + such as `gitlab-gitlab`. This value is used to refer to the provider, and appears in URLs. +- **Issuer (URL)**: The address of your GitLab instance, such as `https://gitlab.com` or + `https://gitlab.example.com`. + - The address must use the `https://` protocol. + - The address must end in a trailing slash. +- **Audiences**: Manually set the allowed audiences list to the address of your + GitLab instance, such as `https://gitlab.com` or `https://gitlab.example.com`. + - The address must use the `https://` protocol. + - The address must not end in a trailing slash. +- **Provider attributes mapping**: Create the following mappings, where `attribute.X` is the + name of the attribute you would like to be present on Google's claims, and `assertion.X` + is the value to extract from the [GitLab claim](../index.md#how-it-works): + + | Attribute (on Google) | Assertion (from GitLab) | + | --- | --- | + | `google.subject` | `assertion.sub` | + | `attribute.X` | `assertion.X` | + + You can also [build complex attributes](https://cloud.google.com/iam/help/workload-identity/attribute-mapping) + using Common Expression Language (CEL). + + You must map every attribute that you want to use for permission granting. For example, if you want to map permissions in the next step based on the user's email address, you must map `attribute.user_email` to `assertion.user_email`. + +## Grant permissions for Service Account impersonation + +Creating the Workload Identity Pool and Workload Identity Provider defines the _authentication_ +into Google Cloud. At this point, you can authenticate from GitLab CI/CD job into Google Cloud. +However, you have no permissions on Google Cloud (_authorization_). + +To grant your GitLab CI/CD job permissions on Google Cloud, you must: + +1. [Create a Google Cloud Service Account](https://www.google.com/search?q=google+cloud+create+service+account). + You can use whatever name and ID you prefer. +1. [Grant IAM permissions](https://cloud.google.com/iam/docs/granting-changing-revoking-access) to your + service account on Google Cloud resources. These permissions vary significantly based on + your use case. In general, grant this service account the permissions on your Google Cloud + project and resources you want your GitLab CI/CD job to be able to use. For example, if you needed to upload a file to a Google Cloud Storage bucket in your GitLab CI/CD job, you would grant this Service Account the `roles/storage.objectCreator` role on your Cloud Storage bucket. +1. [Grant the external identity permissions](https://cloud.google.com/iam/docs/using-workload-identity-federation#impersonate) + to impersonate that Service Account. This step enables a GitLab CI/CD job to _authorize_ + to Google Cloud, via Service Account impersonation. This step grants an IAM permission + _on the Service Account itself_, giving the external identity permissions to act as that + service account. External identities are expressed using the `principalSet://` protocol. + +Much like the previous step, this step depends heavily on your desired configuration. +For example, to allow a GitLab CI/CD job to impersonate a Service Account named +`my-service-account` if the GitLab CI/CD job was initiated by a GitLab user with the +username `chris`, you would grant the `roles/iam.workloadIdentityUser` IAM role to the +external identity on `my-service-account`. The external identity takes the format: + +```plaintext +principalSet://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_ID/attribute.user_login/chris +``` + +where `PROJECT_NUMBER` is your Google Cloud project number, and `POOL_ID` is the +ID (not name) of the Workload Identity Pool created in the first section. + +This configuration also assumes you added `user_login` as an attribute mapped from +the assertion in the previous section. + +## Retrieve a temporary credential + +After you configure the OIDC and role, the GitLab CI/CD job can retrieve a temporary credential from the +[Google Cloud Security Token Service (STS)](https://cloud.google.com/iam/docs/reference/sts/rest). + +```shell +PAYLOAD="$(cat <<EOF +{ + "audience": "//iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_ID/providers/PROVIDER_ID", + "grantType": "urn:ietf:params:oauth:grant-type:token-exchange", + "requestedTokenType": "urn:ietf:params:oauth:token-type:access_token", + "scope": "https://www.googleapis.com/auth/cloud-platform", + "subjectTokenType": "urn:ietf:params:oauth:token-type:jwt", + "subjectToken": "${CI_JOB_JWT_V2}" +} +EOF +)" +``` + +```shell +FEDERATED_TOKEN="$(curl --fail "https://sts.googleapis.com/v1/token" \ + --header "Accept: application/json" \ + --header "Content-Type: application/json" \ + --data "${PAYLOAD}" \ + | jq -r '.access_token' +)" +``` + +Where: + +- `PROJECT_NUMBER` is your Google Cloud project number (not name). +- `POOL_ID` is the ID of the Workload Identity Pool created in the first section. +- `PROVIDER_ID` is the ID of the Workload Identity Provider created in the second section. +- `CI_JOB_JWT_V2` is injected into the CI/CD job by GitLab. For more information about + this variable, read [Connect to cloud services](../index.md). + +You can then use the resulting federated token to impersonate the service account created +in the previous section: + +```shell +ACCESS_TOKEN="$(curl --fail "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/SERVICE_ACCOUNT_EMAIL:generateAccessToken" \ + --header "Accept: application/json" \ + --header "Content-Type: application/json" \ + --header "Authorization: Bearer FEDERATED_TOKEN" \ + --data '{"scope": ["https://www.googleapis.com/auth/cloud-platform"]}' \ + | jq -r '.accessToken' +)" +``` + +Where: + +- `SERVICE_ACCOUNT_EMAIL` is the full email address of the service account to impersonate, + created in the previous section. +- `FEDERATED_TOKEN` is the federated token retrieved from the previous step. + +The result is a Google Cloud OAuth 2.0 access token, which you can use to authenticate to +most Google Cloud APIs and services when used as a bearer token. You can also pass this +value to the `gcloud` CLI by setting the environment variable `CLOUDSDK_AUTH_ACCESS_TOKEN`. + +## Working example + +Review this +[reference project](https://gitlab.com/guided-explorations/gcp/configure-openid-connect-in-gcp) +for provisioning OIDC in GCP using Terraform and a sample script to retrieve temporary credentials. + +## Troubleshooting + +- When debugging `curl` responses, install the latest version of curl. Use `--fail-with-body` + instead of `-f`. This command prints the entire body, which can contain helpful error messages. + +- Review Google Cloud's documentation for + [Troubleshooting Workload Identity Federation](https://cloud.google.com/iam/docs/troubleshooting-workload-identity-federation). diff --git a/doc/ci/cloud_services/index.md b/doc/ci/cloud_services/index.md index 73e726ea8a9..a80231a04c2 100644 --- a/doc/ci/cloud_services/index.md +++ b/doc/ci/cloud_services/index.md @@ -1,6 +1,6 @@ --- -stage: Configure -group: Configure +stage: Verify +group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -19,7 +19,7 @@ GitLab CI/CD supports [OpenID Connect (OIDC)](https://openid.net/connect/faq/) t The original implementation of `CI_JOB_JWT` supports [HashiCorp Vault integration](../examples/authenticating-with-hashicorp-vault/). The updated implementation of `CI_JOB_JWT_V2` supports additional cloud providers with OIDC including AWS, GCP, and Vault. WARNING: -The `CI_JOB_JWT_V2` variable is under development [(alpha)](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha) and is not yet suitable for production use. +The `CI_JOB_JWT_V2` variable is under development [(alpha)](../../policy/alpha-beta-support.md#alpha-features) and is not yet suitable for production use. ## Use cases @@ -99,9 +99,9 @@ sequenceDiagram Note right of Cloud: Decode & verify JWT with public key (https://gitlab/-/jwks) Note right of Cloud: Validate audience defined in OIDC Note right of Cloud: Validate conditional (sub, aud) role - Note right of Cloud: Generate credential or fetch secret + Note right of Cloud: Generate credential or fetch secret Cloud->>GitLab: Return temporary credential - Note left of GitLab: Perform operation + Note left of GitLab: Perform operation ``` @@ -131,3 +131,4 @@ To configure the trust between GitLab and OIDC, you must create a conditional ro To connect with your cloud provider, see the following tutorials: - [Configure OpenID Connect in AWS](aws/index.md) +- [Configure OpenID Connect in Google Cloud](google_cloud/index.md) diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index 5bd9293924d..7edff334134 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -450,3 +450,26 @@ To configure access for `<aws_account_id>.dkr.ecr.<region>.amazonaws.com`, follo You can add configuration for as many registries as you want, adding more registries to the `"credHelpers"` hash. + +### Use checksum to keep your image secure + +We recommend using the image checksum in your job definition in your `.gitlab-ci.yml` file to verify the integrity of the image. A failed image integrity verification will prevent you from using a modified container. + +To use the image checksum you have to append the checksum at the end: + +```yaml +image: ruby:2.6.8@sha256:d1dbaf9665fe8b2175198e49438092fdbcf4d8934200942b94425301b17853c7 +``` + +To get the image checksum, on the image `TAG` tab, view the `DIGEST` column. +For example, view the [Ruby image](https://hub.docker.com/_/ruby?tab=tags). +The checksum is a random string, like `6155f0235e95`. + +You can also get the checksum of any image on your system with the command `docker images --digests`: + +```shell +❯ docker images --digests +REPOSITORY TAG DIGEST (...) +gitlab/gitlab-ee latest sha256:723aa6edd8f122d50cae490b1743a616d54d4a910db892314d68470cc39dfb24 (...) +gitlab/gitlab-runner latest sha256:4a18a80f5be5df44cb7575f6b89d1fdda343297c6fd666c015c0e778b276e726 (...) +``` diff --git a/doc/ci/environments/deployment_approvals.md b/doc/ci/environments/deployment_approvals.md index d60e5704877..e320118c191 100644 --- a/doc/ci/environments/deployment_approvals.md +++ b/doc/ci/environments/deployment_approvals.md @@ -7,7 +7,8 @@ description: Require approvals prior to deploying to a Protected Environment # Deployment approvals **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/343864) in GitLab 14.7 with a flag named `deployment_approvals`. Disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/343864) in GitLab 14.7 with a flag named `deployment_approvals`. Disabled by default. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/347342) in GitLab 14.8. WARNING: This feature is in an alpha stage and subject to change without prior notice. @@ -49,12 +50,19 @@ Example: name: ${CI_JOB_NAME} ``` -### Require approvals for a protected environment +### Require approvals for a protected environment NOTE: -At this time, only API-based configuration is available. UI-based configuration is planned for the near future. See [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/344675). +At this time, it is not possible to require approvals for an existing protected environment. The workaround is to unprotect the environment and configure approvals when re-protecting the environment. -Use the [Protected Environments API](../../api/protected_environments.md#protect-repository-environments) to create an environment with `required_approval_count` > 0. After this is set, all jobs deploying to this environment automatically go into a blocked state and wait for approvals before running. +There are two ways to configure approvals for a protected environment: + +1. Using the [UI](protected_environments.md#protecting-environments) + 1. Set the **Required approvals** field to 1 or more. +1. Using the [REST API](../../api/protected_environments.md#protect-repository-environments) + 2. Set the `required_approval_count` field to 1 or more. + +After this is configured, all jobs deploying to this environment automatically go into a blocked state and wait for approvals before running. Ensure that the number of required approvals is less than the number of users allowed to deploy. Example: @@ -65,8 +73,9 @@ curl --header 'Content-Type: application/json' --request POST \ "https://gitlab.example.com/api/v4/projects/22034114/protected_environments" ``` +NOTE: To protect, update, or unprotect an environment, you must have at least the -[Maintainer role](../../user/permissions.md). +Maintainer role. ## Approve or reject a deployment @@ -75,7 +84,7 @@ This functionality is currently only available through the API. UI is planned fo A blocked deployment is enqueued as soon as it receives the required number of approvals. A single rejection causes the deployment to fail. The creator of a deployment cannot approve it, even if they have permission to deploy. -Using the [Deployments API](../../api/deployments.md#approve-or-reject-a-blocked-deployment), users who are allowed to deploy to the protected environment can approve or reject a blocked deployment. +Using the [Deployments API](../../api/deployments.md#approve-or-reject-a-blocked-deployment), users who are allowed to deploy to the protected environment can approve or reject a blocked deployment. Example: @@ -95,7 +104,11 @@ curl --data "status=approved" \ #### Using the API -Use the [Deployments API](../../api/deployments.md) to see deployments. The `status` field indicates if a deployment is blocked. +Use the [Deployments API](../../api/deployments.md) to see deployments. + +- The `status` field indicates if a deployment is blocked. +- The `pending_approval_count` field indicates how many approvals are remaining to run a deployment. +- The `approvals` field contains the deployment's approvals. ## Related features diff --git a/doc/ci/environments/deployment_safety.md b/doc/ci/environments/deployment_safety.md index 55c3c83338d..0e73dc4f7cd 100644 --- a/doc/ci/environments/deployment_safety.md +++ b/doc/ci/environments/deployment_safety.md @@ -26,7 +26,7 @@ If you are using a continuous deployment workflow and want to ensure that concur ## Restrict write access to a critical environment By default, environments can be modified by any team member that has at least the -[Developer role](../../user/permissions.md#project-members-permissions). +Developer role. If you want to restrict write access to a critical environment (for example a `production` environment), you can set up [protected environments](protected_environments.md). @@ -93,7 +93,7 @@ If you want to prevent deployments for a particular period, for example during a vacation period when most employees are out, you can set up a [Deploy Freeze](../../user/project/releases/index.md#prevent-unintentional-releases-by-setting-a-deploy-freeze). During a deploy freeze period, no deployment can be executed. This is helpful to ensure that deployments do not happen unexpectedly. - + ## Setting appropriate roles to your project GitLab supports several different roles that can be assigned to your project members. See @@ -119,7 +119,7 @@ The other pipelines don't get the protected variable. You can also We recommend that you use protected variables on protected environments to make sure that the secrets aren't exposed unintentionally. You can also define production secrets on the [runner side](../runners/configure_runners.md#prevent-runners-from-revealing-sensitive-information). -This prevents other users with the [Maintainer role](../../user/permissions.md) from reading the secrets and makes sure +This prevents other users with the Maintainer role from reading the secrets and makes sure that the runner only runs on protected branches. For more information, see [pipeline security](../pipelines/index.md#pipeline-security-on-protected-branches). diff --git a/doc/ci/environments/img/environments_open_live_environment_v14_8.png b/doc/ci/environments/img/environments_open_live_environment_v14_8.png Binary files differnew file mode 100644 index 00000000000..daf531c83c4 --- /dev/null +++ b/doc/ci/environments/img/environments_open_live_environment_v14_8.png diff --git a/doc/ci/environments/index.md b/doc/ci/environments/index.md index 794e4320fc6..63bdd279927 100644 --- a/doc/ci/environments/index.md +++ b/doc/ci/environments/index.md @@ -27,7 +27,7 @@ You can even access a [web terminal](#web-terminals-deprecated) for your environ Prerequisites: -- You must have at least the Reporter [role](../../user/permissions.md#project-members-permissions). +- You must have at least the Reporter role. To view a list of environments and deployments: @@ -308,10 +308,18 @@ Note the following: - If the environment URL isn't valid (for example, the URL is malformed), the system doesn't update the environment URL. - If the script that runs in `stop_review` exists only in your repository and therefore can't use - `GIT_STRATEGY: none`, configure [pipelines for merge requests](../../ci/pipelines/merge_request_pipelines.md) + `GIT_STRATEGY: none`, configure [merge request pipelines](../../ci/pipelines/merge_request_pipelines.md) for these jobs. This ensures that runners can fetch the repository even after a feature branch is deleted. For more information, see [Ref Specs for Runners](../pipelines/index.md#ref-specs-for-runners). +NOTE: +For Windows runners, using `echo` to write to `.env` files may fail. Using the PowerShell `Add-Content`command +will help in such cases. For example: + +```powershell +Add-Content -Path deploy.env -Value "DYNAMIC_ENVIRONMENT_URL=$DYNAMIC_ENVIRONMENT_URL" +``` + ## Track newly included merge requests per deployment GitLab can track newly included merge requests per deployment. @@ -372,7 +380,7 @@ places in GitLab: - In a merge request as a link:  - In the Environments view as a button: -  +  - In the Deployments view as a button:  @@ -440,7 +448,7 @@ the `stop_review` job might not be included in all pipelines that include the The job with [`action: stop` might not run](#the-job-with-action-stop-doesnt-run) if it's in a later stage than the job that started the environment. -If you can't use [pipelines for merge requests](../pipelines/merge_request_pipelines.md), +If you can't use [merge request pipelines](../pipelines/merge_request_pipelines.md), set the [`GIT_STRATEGY`](../runners/configure_runners.md#git-strategy) to `none` in the `stop_review` job. Then the [runner](https://docs.gitlab.com/runner/) doesn't try to check out the code after the branch is deleted. diff --git a/doc/ci/environments/protected_environments.md b/doc/ci/environments/protected_environments.md index 78db2345de4..c7d1653aace 100644 --- a/doc/ci/environments/protected_environments.md +++ b/doc/ci/environments/protected_environments.md @@ -20,7 +20,7 @@ NOTE: GitLab administrators can use all environments, including protected environments. To protect, update, or unprotect an environment, you need to have at least the -[Maintainer role](../../user/permissions.md). +Maintainer role. ## Protecting environments @@ -38,6 +38,9 @@ To protect an environment: - You can select groups that are already associated with the project only. - Users must have at least the Developer role to appear in the **Allowed to deploy** list. +1. In the **Required approvals** list, select the number of approvals required to deploy to this environment. + - Ensure that this number is less than the number of users allowed to deploy. + - See [Deployment Approvals](deployment_approvals.md) for more information about this feature. 1. Select **Protect**. The protected environment now appears in the list of protected environments. @@ -94,7 +97,7 @@ Alternatively, you can use the API to protect an environment: 1. Use the API to add the group with protected environment access: ```shell - curl --header 'Content-Type: application/json' --request POST --data '{"name": "production", "deploy_access_levels": [{"group_id": 9899826}]}' \ + curl --header 'Content-Type: application/json' --request POST --data '{"name": "production", "deploy_access_levels": [{"group_id": 9899826}], "required_approval_count": 0}' \ --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.com/api/v4/projects/22034114/protected_environments" ``` @@ -103,12 +106,12 @@ The group now has access and can be seen in the UI. ## Environment access by group membership A user may be granted access to protected environments as part of [group membership](../../user/group/index.md). Users -with the Reporter [role](../../user/permissions.md) can only be granted access to protected environments with this +with the Reporter role can only be granted access to protected environments with this method. ## Deployment branch access -Users with the [Developer role](../../user/permissions.md) can be granted +Users with the Developer role can be granted access to a protected environment through any of these methods: - As an individual contributor, through a role. @@ -125,7 +128,7 @@ they have the following privileges: Users granted access to a protected environment, but not push or merge access to the branch deployed to it, are only granted access to deploy the environment. An individual in a -group with the Reporter [role](../../user/permissions.md), or in groups added to the project with the Reporter +group with the Reporter role, or in groups added to the project with the Reporter role, appears in the dropdown menu for deployment-only access. To add deployment-only access: @@ -136,7 +139,7 @@ To add deployment-only access: 1. Follow the steps in [Protecting Environments](#protecting-environments). Note that deployment-only access is the only possible access level for groups with the Reporter -[role](../../user/permissions.md). +role. ## Modifying and unprotecting environments @@ -194,14 +197,14 @@ and are protected at the same time. In an enterprise organization, with thousands of projects under a single group, ensuring that all of the [project-level protected environments](#protecting-environments) are properly configured is not a scalable solution. For example, a developer -might gain privileged access to a higher environment when they are given the [Maintainer role](../../user/permissions.md) +might gain privileged access to a higher environment when they are given the Maintainer role for a new project. Group-level protected environments can be a solution in this situation. To maximize the effectiveness of group-level protected environments, [group-level memberships](../../user/group/index.md) must be correctly configured: -- Operators should be given at least the [Maintainer role](../../user/permissions.md) +- Operators should be given at least the Maintainer role for the top-level group. They can maintain CI/CD configurations for the higher environments (such as production) in the group-level settings page, which includes group-level protected environments, @@ -210,8 +213,8 @@ configured: configurations are inherited to the child projects as read-only entries. This ensures that only operators can configure the organization-wide deployment ruleset. -- Developers should be given no more than the [Developer role](../../user/permissions.md) - for the top-level group, or explicitly given the [Maintainer role](../../user/permissions.md) for a child project +- Developers should be given no more than the Developer role + for the top-level group, or explicitly given the Maintainer role for a child project They do *NOT* have access to the CI/CD configurations in the top-level group, so operators can ensure that the critical configuration won't be accidentally changed by the developers. @@ -224,7 +227,7 @@ configured: environment configurations exist, to run a deployment job, the user must be allowed in **both** rulesets. - In a project or a subgroup of the top-level group, developers can be - safely assigned the [Maintainer role](../../user/permissions.md) to tune their lower environments (such + safely assigned the Maintainer role to tune their lower environments (such as `testing`). Having this configuration in place: diff --git a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md index aed45951239..edc58684057 100644 --- a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md +++ b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Verify +group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: tutorial --- diff --git a/doc/ci/examples/end_to_end_testing_webdriverio/index.md b/doc/ci/examples/end_to_end_testing_webdriverio/index.md index 9881c9657bc..4d247a4ff74 100644 --- a/doc/ci/examples/end_to_end_testing_webdriverio/index.md +++ b/doc/ci/examples/end_to_end_testing_webdriverio/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Testing +group: Pipeline Insights info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments author: Vincent Tunru author_gitlab: Vinnl diff --git a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md index be33e62b75c..6dbec0dfc8b 100644 --- a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md +++ b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md @@ -378,7 +378,7 @@ Now, we would need to deploy our app by running `envoy run deploy`, but it won't Now it's time to commit [Envoy.blade.php](https://gitlab.com/mehranrasulian/laravel-sample/blob/master/Envoy.blade.php) and push it to the `main` branch. To keep things simple, we commit directly to `main`, without using [feature-branches](../../../topics/gitlab_flow.md#github-flow-as-a-simpler-alternative) since collaboration is beyond the scope of this tutorial. -In a real world project, teams may use [Issue Tracker](../../../user/project/issues/index.md) and [Merge Requests](../../../user/project/merge_requests/index.md) to move their code across branches: +In a real world project, teams may use [Issue Tracker](../../../user/project/issues/index.md) and [merge requests](../../../user/project/merge_requests/index.md) to move their code across branches: ```shell git add Envoy.blade.php diff --git a/doc/ci/git_submodules.md b/doc/ci/git_submodules.md index cdc75fd2bec..23055514839 100644 --- a/doc/ci/git_submodules.md +++ b/doc/ci/git_submodules.md @@ -60,6 +60,15 @@ To make submodules work correctly in CI/CD jobs: GIT_SUBMODULE_STRATEGY: recursive ``` +1. You can provide additional flags to control advanced checkout behavior using + [`GIT_SUBMODULE_UPDATE_FLAGS`](runners/configure_runners.md#git-submodule-update-flags). + + ```yaml + variables: + GIT_SUBMODULE_STRATEGY: recursive + GIT_SUBMODULE_UPDATE_FLAGS: --jobs 4 + ``` + If you use the [`CI_JOB_TOKEN`](jobs/ci_job_token.md) to clone a submodule in a pipeline job, the user executing the job must be assigned to a role that has [permission](../user/permissions.md#gitlab-cicd-permissions) to trigger a pipeline diff --git a/doc/ci/img/ci_lint.png b/doc/ci/img/ci_lint.png Binary files differdeleted file mode 100644 index fdc3868cdce..00000000000 --- a/doc/ci/img/ci_lint.png +++ /dev/null diff --git a/doc/ci/img/ci_lint_dry_run.png b/doc/ci/img/ci_lint_dry_run.png Binary files differdeleted file mode 100644 index 61d6379f70e..00000000000 --- a/doc/ci/img/ci_lint_dry_run.png +++ /dev/null diff --git a/doc/ci/img/environments_available_13_10.png b/doc/ci/img/environments_available_13_10.png Binary files differdeleted file mode 100644 index 94ffb0032fa..00000000000 --- a/doc/ci/img/environments_available_13_10.png +++ /dev/null diff --git a/doc/ci/index.md b/doc/ci/index.md index c557e9e6f57..d0c2e1986b2 100644 --- a/doc/ci/index.md +++ b/doc/ci/index.md @@ -64,7 +64,7 @@ GitLab CI/CD supports numerous configuration options: | [Git submodules for CI/CD](git_submodules.md) | Configure jobs for using Git submodules. | | [SSH keys for CI/CD](ssh_keys/index.md) | Using SSH keys in your CI pipelines. | | [Pipeline triggers](triggers/index.md) | Trigger pipelines through the API. | -| [Pipelines for Merge Requests](pipelines/merge_request_pipelines.md) | Design a pipeline structure for running a pipeline in merge requests. | +| [Merge request pipelines](pipelines/merge_request_pipelines.md) | Design a pipeline structure for running a pipeline in merge requests. | | [Integrate with Kubernetes clusters](../user/infrastructure/clusters/index.md) | Connect your project to Google Kubernetes Engine (GKE) or an existing Kubernetes cluster. | | [Optimize GitLab and GitLab Runner for large repositories](large_repositories/index.md) | Recommended strategies for handling large repositories. | | [`.gitlab-ci.yml` full reference](yaml/index.md) | All the attributes you can use with GitLab CI/CD. | diff --git a/doc/ci/jobs/ci_job_token.md b/doc/ci/jobs/ci_job_token.md index 1906d9cdb6c..a62f670cb12 100644 --- a/doc/ci/jobs/ci_job_token.md +++ b/doc/ci/jobs/ci_job_token.md @@ -23,7 +23,6 @@ You can use a GitLab CI/CD job token to authenticate with specific API endpoints - [Releases](../../api/releases/index.md). - [Terraform plan](../../user/infrastructure/index.md). -The token has the same permissions to access the API as the user that executes the The token has the same permissions to access the API as the user that caused the job to run. A user can cause a job to run by pushing a commit, triggering a manual job, being the owner of a scheduled pipeline, and so on. Therefore, this user must be assigned to @@ -95,7 +94,7 @@ The job token scope is only for controlling access to private projects. 1. Expand **Token Access**. 1. Toggle **Limit CI_JOB_TOKEN access** to enabled. 1. Optional. Add existing projects to the token's access scope. The user adding a - project must have the [maintainer role](../../user/permissions.md) in both projects. + project must have the Maintainer role in both projects. There is [a proposal](https://gitlab.com/groups/gitlab-org/-/epics/3559) to improve the feature with more strategic control of the access permissions. diff --git a/doc/ci/jobs/index.md b/doc/ci/jobs/index.md index 104badb782c..39e14d0d20a 100644 --- a/doc/ci/jobs/index.md +++ b/doc/ci/jobs/index.md @@ -95,6 +95,9 @@ You can't use these keywords as job names: - `variables` - `cache` - `include` +- `true` +- `false` +- `nil` Job names must be 255 characters or less. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/342800) in GitLab 14.5, [with a feature flag](../../administration/feature_flags.md) named `ci_validate_job_length`. diff --git a/doc/ci/jobs/job_control.md b/doc/ci/jobs/job_control.md index a2406a68bb2..6523de0ed1e 100644 --- a/doc/ci/jobs/job_control.md +++ b/doc/ci/jobs/job_control.md @@ -220,7 +220,7 @@ check the value of the `$CI_PIPELINE_SOURCE` variable: | `chat` | For pipelines created by using a [GitLab ChatOps](../chatops/index.md) command. | | `external` | When you use CI services other than GitLab. | | `external_pull_request_event` | When an external pull request on GitHub is created or updated. See [Pipelines for external pull requests](../ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests). | -| `merge_request_event` | For pipelines created when a merge request is created or updated. Required to enable [merge request pipelines](../pipelines/merge_request_pipelines.md), [merged results pipelines](../pipelines/pipelines_for_merged_results.md), and [merge trains](../pipelines/merge_trains.md). | +| `merge_request_event` | For pipelines created when a merge request is created or updated. Required to enable [merge request pipelines](../pipelines/merge_request_pipelines.md), [merged results pipelines](../pipelines/merged_results_pipelines.md), and [merge trains](../pipelines/merge_trains.md). | | `parent_pipeline` | For pipelines triggered by a [parent/child pipeline](../pipelines/parent_child_pipelines.md) with `rules`. Use this pipeline source in the child pipeline configuration so that it can be triggered by the parent pipeline. | | `pipeline` | For [multi-project pipelines](../pipelines/multi_project_pipelines.md) created by [using the API with `CI_JOB_TOKEN`](../pipelines/multi_project_pipelines.md#create-multi-project-pipelines-by-using-the-api), or the [`trigger`](../yaml/index.md#trigger) keyword. | | `push` | For pipelines triggered by a `git push` event, including for branches and tags. | @@ -263,6 +263,9 @@ Other commonly used variables for `if` clauses: branch. Use when you want to have the same configuration in multiple projects with different default branches. - `if: '$CI_COMMIT_BRANCH =~ /regex-expression/'`: If the commit branch matches a regular expression. +- `if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_COMMIT_TITLE =~ /Merge branch.*/`: + If the commit branch is the default branch and the commit message title matches a regular expression. + For example, the default commit message for a merge commit starts with `Merge branch`. - `if: '$CUSTOM_VARIABLE !~ /regex-expression/'`: If the [custom variable](../variables/index.md#custom-cicd-variables) `CUSTOM_VARIABLE` does **not** match a regular expression. - `if: '$CUSTOM_VARIABLE == "value1"'`: If the custom variable `CUSTOM_VARIABLE` is @@ -406,9 +409,9 @@ the `build` job is still skipped. The job does not run for any of the files. With some configurations that use `changes`, [jobs or pipelines might run unexpectedly](#jobs-or-pipelines-run-unexpectedly-when-using-changes) -#### Use `only:changes` with pipelines for merge requests +#### Use `only:changes` with merge request pipelines -With [pipelines for merge requests](../pipelines/merge_request_pipelines.md), +With [merge request pipelines](../pipelines/merge_request_pipelines.md), it's possible to define a job to be created based on files modified in a merge request. @@ -704,9 +707,12 @@ deploystacks: [gcp, data] deploystacks: [vultr, data] ``` -In [GitLab 14.1 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/239737), you can -use the variables defined in `parallel: matrix` with the [`tags`](../yaml/index.md#tags) keyword for -dynamic runner selection. +### Select different runner tags for each parallel matrix job + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/239737) in GitLab 14.1. + +You can use variables defined in `parallel: matrix` with the [`tags`](../yaml/index.md#tags) +keyword for dynamic runner selection: ```yaml deploystacks: @@ -931,7 +937,7 @@ For example: You might have jobs or pipelines that run unexpectedly when using [`rules: changes`](../yaml/index.md#ruleschanges) or [`only: changes`](../yaml/index.md#onlychanges--exceptchanges) without -[pipelines for merge requests](../pipelines/merge_request_pipelines.md). +[merge request pipelines](../pipelines/merge_request_pipelines.md). Pipelines on branches or tags that don't have an explicit association with a merge request use a previous SHA to calculate the diff. This calculation is equivalent to `git diff HEAD~` @@ -944,3 +950,19 @@ and can cause unexpected behavior, including: Additionally, rules with `changes` always evaluate as true in [scheduled pipelines](../pipelines/schedules.md). All files are considered to have changed when a scheduled pipeline runs, so jobs might always be added to scheduled pipelines that use `changes`. + +### `You are not allowed to download code from this project.` error message + +You might see pipelines fail when a GitLab administrator runs a protected manual job +in a private project. + +CI/CD jobs usually clone the project when the job starts, and this uses [the permissions](../../user/permissions.md#job-permissions) +of the user that runs the job. All users, including administrators, must be direct members +of a private project to clone the source of that project. [An issue exists](https://gitlab.com/gitlab-org/gitlab/-/issues/23130) +to change this behavior. + +To run protected manual jobs: + +- Add the administrator as a direct member of the private project (any role) +- [Impersonate a user](../../user/admin_area/index.md#user-impersonation) who is a + direct member of the project. diff --git a/doc/ci/lint.md b/doc/ci/lint.md index e58907ee0bd..c0df0b2a439 100644 --- a/doc/ci/lint.md +++ b/doc/ci/lint.md @@ -4,48 +4,54 @@ group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Validate `.gitlab-ci.yml` syntax with the CI Lint tool **(FREE)** +# Validate GitLab CI/CD configuration **(FREE)** -If you want to test the validity of your GitLab CI/CD configuration before committing -the changes, you can use the CI Lint tool. This tool checks for syntax and logical -errors by default, and can simulate pipeline creation to try to find more complicated -issues as well. +Use the CI Lint tool to check the validity of GitLab CI/CD configuration. +You can validate the syntax from a `.gitlab-ci.yml` file or any other sample CI/CD configuration. +This tool checks for syntax and logic errors, and can simulate pipeline +creation to try to find more complicated configuration problems. -To access the CI Lint tool, navigate to **CI/CD > Pipelines** or **CI/CD > Jobs** -in your project and click **CI lint**. +If you use the [pipeline editor](pipeline_editor/index.md), it verifies configuration +syntax automatically. -If you use VS Code, you can also validate your CI/CD configuration with the +If you use VS Code, you can validate your CI/CD configuration with the [GitLab Workflow VS Code extension](../user/project/repository/vscode.md). -## Validate basic logic and syntax +## Check CI/CD syntax -By default, the CI lint checks the syntax of your CI YAML configuration and also runs -some basic logical validations. Configuration added with the [`includes` keyword](yaml/index.md#include), -is also validated. +The CI lint tool checks the syntax of GitLab CI/CD configuration, including +configuration added with the [`includes` keyword](yaml/index.md#include). -To use the CI lint, paste a complete CI configuration (`.gitlab-ci.yml` for example) -into the text box and click **Validate**: +To check CI/CD configuration with the CI lint tool: - +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select one of: + - **CI/CD > Pipelines** + - **CI/CD > Jobs** +1. In the top right, select **CI lint**. +1. Paste a copy of the CI/CD configuration you want to check into the text box. +1. Select **Validate**. -## Pipeline simulation +## Simulate a pipeline > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229794) in GitLab 13.3. -Not all pipeline configuration issues can be found by the [basic CI lint validation](#validate-basic-logic-and-syntax). -You can simulate the creation of a pipeline for deeper validation that can discover -more complicated issues. - -To validate the configuration by running a pipeline simulation: +You can simulate the creation of a GitLab CI/CD pipeline to find more complicated issues, +including problems with [`needs`](yaml/index.md#needs) and [`rules`](yaml/index.md#rules) +configuration. A simulation runs as a Git `push` event on the default branch. -1. Paste the GitLab CI configuration to verify into the text box. -1. Select the **Simulate pipeline creation for the default branch** checkbox. -1. Select **Validate**. +Prerequisites: - +- You must have [permissions](../user/permissions.md#project-members-permissions) + to create pipelines on this branch to validate with a simulation. -### Pipeline simulation limitations +To simulate a pipeline: -Simulations run as `git push` events against the default branch. You must have -[permissions](../user/permissions.md#project-members-permissions) to create pipelines -on this branch to validate with a simulation. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select one of: + - **CI/CD > Pipelines** + - **CI/CD > Jobs** +1. In the top right, select **CI lint**. +1. Paste a copy of the CI/CD configuration you want to check into the text box. +1. Select **Simulate pipeline creation for the default branch**. +1. Select **Validate**. diff --git a/doc/ci/metrics_reports.md b/doc/ci/metrics_reports.md index eb302b9ed7f..5b472eec7b5 100644 --- a/doc/ci/metrics_reports.md +++ b/doc/ci/metrics_reports.md @@ -1,8 +1,7 @@ --- stage: Verify -group: Testing +group: Pipeline Insights info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference --- # Metrics Reports **(PREMIUM)** diff --git a/doc/ci/migration/jenkins.md b/doc/ci/migration/jenkins.md index ef6f28e36e5..bfa2eb54806 100644 --- a/doc/ci/migration/jenkins.md +++ b/doc/ci/migration/jenkins.md @@ -196,7 +196,7 @@ can leverage. You can see the complete list of packaging features in the Where you may have used plugins to get things like code quality, unit tests, security scanning, and so on working in Jenkins, GitLab takes advantage of our connected ecosystem to automatically pull these kinds of results into -your Merge Requests, pipeline details pages, and other locations. You may find that you actually don't +your merge requests, pipeline details pages, and other locations. You may find that you actually don't need to configure anything to have these appear. If they aren't working as expected, or if you'd like to see what's available, our [CI feature index](../index.md#gitlab-cicd-features) has the full list diff --git a/doc/ci/pipelines/cicd_minutes.md b/doc/ci/pipelines/cicd_minutes.md index e0fb5b45986..8b10a74bd78 100644 --- a/doc/ci/pipelines/cicd_minutes.md +++ b/doc/ci/pipelines/cicd_minutes.md @@ -149,7 +149,7 @@ namespace. ## How CI/CD minutes are calculated -CI/CD minutes are calculated based on: +CI/CD minutes for individual jobs are calculated based on: - The duration the job runs. - The visibility of the projects where the job runs. @@ -174,6 +174,10 @@ For example: - If a pipeline runs for one of the personal projects for `alice`, the CI/CD minutes are added to the overall consumption for the `alice` namespace. +The CI/CD minutes used by one pipeline is the total CI/CD minutes used by all the jobs +that ran in the pipeline. The CI/CD minute usage for a pipeline can be higher than +the duration of the pipeline if many jobs ran at the same time. + ### Cost factor The cost factor for a job running on a shared runner is: diff --git a/doc/ci/pipelines/img/merged_result_pipeline.png b/doc/ci/pipelines/img/merged_result_pipeline.png Binary files differdeleted file mode 100644 index 2584cd4d38d..00000000000 --- a/doc/ci/pipelines/img/merged_result_pipeline.png +++ /dev/null diff --git a/doc/ci/pipelines/index.md b/doc/ci/pipelines/index.md index b873b2ae30f..5a5fd2b5540 100644 --- a/doc/ci/pipelines/index.md +++ b/doc/ci/pipelines/index.md @@ -50,14 +50,14 @@ Pipelines can be configured in many different ways: followed by the next stage. - [Directed Acyclic Graph Pipeline (DAG) pipelines](../directed_acyclic_graph/index.md) are based on relationships between jobs and can run more quickly than basic pipelines. -- [Pipelines for Merge Requests](../pipelines/merge_request_pipelines.md) run for merge +- [Merge request pipelines](../pipelines/merge_request_pipelines.md) run for merge requests only (rather than for every commit). -- [Pipelines for Merged Results](../pipelines/pipelines_for_merged_results.md) +- [Merged results pipelines](../pipelines/merged_results_pipelines.md) are merge request pipelines that act as though the changes from the source branch have already been merged into the target branch. -- [Merge Trains](../pipelines/merge_trains.md) - use pipelines for merged results to queue merges one after the other. -- [Parent-Child pipelines](parent_child_pipelines.md) break down complex pipelines +- [Merge trains](../pipelines/merge_trains.md) + use merged results pipelines to queue merges one after the other. +- [Parent-child pipelines](parent_child_pipelines.md) break down complex pipelines into one parent pipeline that can trigger multiple child sub-pipelines, which all run in the same project and with the same SHA. This pipeline architecture is commonly used for mono-repos. - [Multi-project pipelines](multi_project_pipelines.md) combine pipelines for different projects together. @@ -123,7 +123,7 @@ This table lists the refspecs injected for each pipeline type: |--------------- |---------------------------------------- | | pipeline for branches | `+<sha>:refs/pipelines/<id>` and `+refs/heads/<name>:refs/remotes/origin/<name>` | | pipeline for tags | `+<sha>:refs/pipelines/<id>` and `+refs/tags/<name>:refs/tags/<name>` | -| [pipeline for merge requests](../pipelines/merge_request_pipelines.md) | `+<sha>:refs/pipelines/<id>` | +| [merge request pipeline](../pipelines/merge_request_pipelines.md) | `+<sha>:refs/pipelines/<id>` | The refs `refs/heads/<name>` and `refs/tags/<name>` exist in your project repository. GitLab generates the special ref `refs/pipelines/<id>` during a @@ -281,7 +281,7 @@ pipelines. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24851) in GitLab 12.7. -Users with the [Owner role](../../user/permissions.md) in a project can delete a pipeline +Users with the Owner role for a project can delete a pipeline by clicking on the pipeline in the **CI/CD > Pipelines** to get to the **Pipeline Details** page, then using the **Delete** button. diff --git a/doc/ci/pipelines/job_artifacts.md b/doc/ci/pipelines/job_artifacts.md index 1710c57b36b..fa8041671dc 100644 --- a/doc/ci/pipelines/job_artifacts.md +++ b/doc/ci/pipelines/job_artifacts.md @@ -1,9 +1,8 @@ --- stage: Verify -group: Testing +group: Pipeline Insights info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments disqus_identifier: 'https://docs.gitlab.com/ee/user/project/pipelines/job_artifacts.html' -type: reference, howto --- # Job artifacts **(FREE)** @@ -194,7 +193,8 @@ artifacts: ### Add untracked files to artifacts Use [`artifacts:untracked`](../yaml/index.md#artifactsuntracked) to add all Git untracked -files as artifacts (along with the paths defined in [`artifacts:paths`](../yaml/index.md#artifactspaths)). +files as artifacts (along with the paths defined in [`artifacts:paths`](../yaml/index.md#artifactspaths)). Untracked +files are those that haven't been added to the repository but exist in the repository checkout. Save all Git untracked files and files in `binaries`: @@ -259,7 +259,7 @@ You can delete a single job, which also removes the job's artifacts and log. You must be: - The owner of the job. -- A [maintainer](../../user/permissions.md#gitlab-cicd-permissions) of the project. +- A user with at least the Maintainer role for the project. To delete a job: @@ -381,7 +381,7 @@ to the `expire_in` specification. If a new pipeline for the same ref completes successfully, the previous pipeline's artifacts are deleted according to the `expire_in` configuration. The artifacts -of the new pipeline are kept automatically. +of the new pipeline are kept automatically. Keeping the latest artifacts can use a large amount of storage space in projects with a lot of jobs or large artifacts. If the latest artifacts are not needed in diff --git a/doc/ci/pipelines/merge_request_pipelines.md b/doc/ci/pipelines/merge_request_pipelines.md index 4d7ebc09e6f..dcc3e7e6919 100644 --- a/doc/ci/pipelines/merge_request_pipelines.md +++ b/doc/ci/pipelines/merge_request_pipelines.md @@ -5,13 +5,15 @@ info: To determine the technical writer assigned to the Stage/Group associated w --- -# Pipelines for merge requests **(FREE)** +# Merge request pipelines **(FREE)** + +> [Renamed](https://gitlab.com/gitlab-org/gitlab/-/issues/351192) from `pipelines for merge requests` to `merge request pipelines` in GitLab 14.8. You can configure your [pipeline](index.md) to run every time you commit changes to a branch. This type of pipeline is called a *branch pipeline*. Alternatively, you can configure your pipeline to run every time you make changes to the -source branch for a merge request. This type of pipeline is called a *pipeline for merge requests*. +source branch for a merge request. This type of pipeline is called a *merge request pipeline*. Branch pipelines: @@ -20,28 +22,28 @@ Branch pipelines: - Have access to [some predefined variables](../variables/predefined_variables.md). - Have access to [protected variables](../variables/index.md#protect-a-cicd-variable). -Pipelines for merge requests: +Merge request pipelines: - Run when you: - Create a new merge request. - Push a new commit to the source branch for a merge request. - Select **Run pipeline** from the **Pipelines** tab in a merge request. This option - is only available when pipelines for merge requests are configured for the pipeline. + is only available when merge request pipelines are configured for the pipeline. - Do not run by default. The jobs in the CI/CD configuration file [must be configured](#prerequisites) - to run in pipelines for merge request. + to run in merge request pipelines. - Have access to [more predefined variables](#available-predefined-variables). - Do not have access to [protected variables](../variables/index.md#protect-a-cicd-variable). Both of these types of pipelines can appear on the **Pipelines** tab of a merge request. -## Types of pipelines for merge requests +## Types of merge request pipelines -The three types of pipelines for merge requests are: +The three types of merge request pipelines are: -- Pipelines for merge requests, which run on the changes in the merge request's +- Merge request pipelines, which run on the changes in the merge request's source branch. These pipelines display a `detached` label to indicate that the pipeline ran only on the contents of the source branch, ignoring the target branch. -- [Pipelines for merged results](pipelines_for_merged_results.md), which run on +- [Merged results pipelines](merged_results_pipelines.md), which run on the result of combining the source branch's changes with the target branch. - [Merge trains](merge_trains.md), which run when merging multiple merge requests at the same time. The changes from each merge request are combined into the @@ -50,31 +52,31 @@ The three types of pipelines for merge requests are: ## Prerequisites -To use pipelines for merge requests: +To use merge request pipelines: - Your project's [CI/CD configuration file](../yaml/index.md) must be configured with - jobs that run in pipelines for merge requests. To do this, you can use: + jobs that run in merge request pipelines. To do this, you can use: - [`rules`](#use-rules-to-add-jobs). - [`only/except`](#use-only-to-add-jobs). -- You must have at least the Developer [role](../../user/permissions.md) in the - source project to run a pipeline for merge requests. +- You must have at least the Developer role in the + source project to run a merge request pipeline. - Your repository must be a GitLab repository, not an [external repository](../ci_cd_for_external_repos/index.md). ## Use `rules` to add jobs You can use the [`rules`](../yaml/index.md#rules) keyword to configure jobs to run in -pipelines for merge requests. For example: +merge request pipelines. For example: ```yaml job1: script: - - echo "This job runs in pipelines for merge requests" + - echo "This job runs in merge request pipelines" rules: - if: $CI_PIPELINE_SOURCE == 'merge_request_event' ``` You can also use the [`workflow: rules`](../yaml/index.md#workflowrules) keyword -to configure the entire pipeline to run in pipelines for merge requests. For example: +to configure the entire pipeline to run in merge request pipelines. For example: ```yaml workflow: @@ -83,22 +85,22 @@ workflow: job1: script: - - echo "This job runs in pipelines for merge requests" + - echo "This job runs in merge request pipelines" job2: script: - - echo "This job also runs in pipelines for merge requests" + - echo "This job also runs in merge request pipelines" ``` ## Use `only` to add jobs You can use the [`only`](../yaml/index.md#onlyrefs--exceptrefs) keyword with `merge_requests` -to configure jobs to run in pipelines for merge requests. +to configure jobs to run in merge request pipelines. ```yaml job1: script: - - echo "This job runs in pipelines for merge requests" + - echo "This job runs in merge request pipelines" only: - merge_requests ``` @@ -122,7 +124,7 @@ Pipelines for forks display with the **fork** badge in the parent project: ### Run pipelines in the parent project **(PREMIUM)** -Project members in the parent project can choose to run a pipeline for merge requests +Project members in the parent project can choose to run a merge request pipeline for a merge request submitted from a fork project. This pipeline: - Is created and runs in the parent (target) project, not the fork (source) project. @@ -140,20 +142,26 @@ parent project when the pipeline runs, even before merge. As a reviewer, careful check the changes in the merge request before triggering the pipeline. GitLab shows a warning that you must accept before you can trigger the pipeline. -Parent project members with at least the [Developer role](../../user/permissions.md) -can create pipelines in the parent project for merge requests from a forked project: +Prerequisites: + +- You must be a member of the parent project and have at least the [Developer role](../../user/permissions.md). +- The fork project must be [visible](../../public_access/public_access.md) to the + user running the pipeline. Otherwise, the **Pipelines** tab does not display + in the merge request. + +To run a pipeline in the parent project for a merge request from a fork project: 1. In the merge request, go to the **Pipelines** tab. 1. Select **Run pipeline**. You must accept the warning, or the pipeline does not run. ## Available predefined variables -When you use pipelines for merge requests, you can use: +When you use merge request pipelines, you can use: - All the same [predefined variables](../variables/predefined_variables.md) that are available in branch pipelines. - [Additional predefined variables](../variables/predefined_variables.md#predefined-variables-for-merge-request-pipelines) - available only to jobs in pipelines for merge requests. These variables contain + available only to jobs in merge request pipelines. These variables contain information from the associated merge request, which can be when calling the [GitLab Merge Request API endpoint](../../api/merge_requests.md) from a job. @@ -166,14 +174,14 @@ to run for both branches and merge requests at the same time. Adjust your pipeli configuration to [avoid duplicate pipelines](../jobs/job_control.md#avoid-duplicate-pipelines). In [GitLab 13.7 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/201845), -you can add `workflow:rules` to [switch from branch pipelines to pipelines for merge requests](../yaml/workflow.md#switch-between-branch-pipelines-and-merge-request-pipelines). +you can add `workflow:rules` to [switch from branch pipelines to merge request pipelines](../yaml/workflow.md#switch-between-branch-pipelines-and-merge-request-pipelines). After a merge request is open on the branch, the pipeline switches to a merge request pipeline. ### Two pipelines when pushing an invalid CI/CD configuration file If you push an invalid CI/CD configuration to a merge request's branch, two failed pipelines appear in the pipelines tab. One pipeline is a failed branch pipeline, -the other is a failed pipeline for merge requests. +the other is a failed merge request pipeline. When the configuration syntax is fixed, no further failed pipelines should appear. To find and fix the configuration problem, you can use: @@ -183,7 +191,7 @@ To find and fix the configuration problem, you can use: ### The merge request's pipeline is marked as failed but the latest pipeline succeeded -It's possible to have both branch pipelines and pipelines for merge requests in the +It's possible to have both branch pipelines and merge request pipelines in the **Pipelines** tab of a single merge request. This might be [by configuration](../yaml/workflow.md#switch-between-branch-pipelines-and-merge-request-pipelines), or [by accident](#two-pipelines-when-pushing-to-a-branch). @@ -191,8 +199,8 @@ If both types of pipelines are in one merge request, the merge request's pipelin is not considered successful if: - The branch pipeline succeeds. -- The pipeline for merge request fails. +- The merge request pipeline fails. When using the [merge when pipeline succeeds](../../user/project/merge_requests/merge_when_pipeline_succeeds.md) -feature and both pipelines types are present, the pipelines for merge requests are checked, +feature and both pipelines types are present, the merge request pipelines are checked, not the branch pipelines. diff --git a/doc/ci/pipelines/merge_trains.md b/doc/ci/pipelines/merge_trains.md index d47cbf5f47c..ffcce06cfbd 100644 --- a/doc/ci/pipelines/merge_trains.md +++ b/doc/ci/pipelines/merge_trains.md @@ -11,7 +11,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w For more information about why you might want to use merge trains, read [How merge trains keep your master green](https://about.gitlab.com/blog/2020/01/30/all-aboard-merge-trains/). -When [pipelines for merged results](pipelines_for_merged_results.md) are +When [merged results pipelines](merged_results_pipelines.md) are enabled, the pipeline jobs run as if the changes from your source branch have already been merged into the target branch. @@ -67,7 +67,7 @@ branch](https://www.youtube.com/watch?v=D4qCqXgZkHQ). To enable merge trains: -- You must have the [Maintainer role](../../user/permissions.md). +- You must have the Maintainer role. - You must be using [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) 11.9 or later. - In GitLab 13.0 and later, you need [Redis](https://redis.io/) 5.0 or later. - Your repository must be a GitLab repository, not an @@ -93,7 +93,7 @@ In GitLab 13.5 and earlier, there is only one checkbox, named WARNING: If you select the check box but don't configure your CI/CD to use -pipelines for merge requests, your merge requests may become stuck in an +merge request pipelines, your merge requests may become stuck in an unresolved state or your pipelines may be dropped. ## Start a merge train @@ -145,8 +145,22 @@ This is the fastest option to get the change merged into the target branch.  WARNING: -Each time you merge a merge request immediately, the current merge train -is recreated and all pipelines restart. +Each time you merge a merge request immediately, the current merge train is recreated, +all pipelines restart, and [redundant pipelines are cancelled](#automatic-pipeline-cancellation). + +### Automatic pipeline cancellation + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12996) in GitLab 12.3. + +GitLab CI/CD can detect the presence of redundant pipelines, and cancels them +to conserve CI resources. + +When a user merges a merge request immediately in an ongoing merge +train, the train is reconstructed, because it recreates the expected +post-merge commit and pipeline. In this case, the merge train may already +have pipelines running against the previous expected post-merge commit. +These pipelines are considered redundant and are automatically +canceled. ## Troubleshooting @@ -179,8 +193,8 @@ for more information. ### Merge train pipeline cannot be retried -When a pipeline for merge trains fails the merge request is dropped from the train and the pipeline can't be retried. -Pipelines for merge trains run on the merged result of the changes in the merge request and +When a merge train pipeline fails, the merge request is dropped from the train and the pipeline can't be retried. +Merge train pipelines run on the merged result of the changes in the merge request and the changes from other merge requests already on the train. If the merge request is dropped from the train, the merged result is out of date and the pipeline can't be retried. @@ -197,11 +211,16 @@ is enabled in **Settings > General > Merge requests**. This option requires that run a new successful pipeline before you can re-add a merge request to a merge train. Merge trains ensure that each pipeline has succeeded before a merge happens, so -you can clear the **Pipelines must succeed** checkbox and keep -**Enable merge trains and pipelines for merged results** (merge trains) selected. +you can: + +- Clear the **Pipelines must succeed** checkbox. +- Select the **Enable merged results pipelines** and **Enable merge trains** checkboxes. + + In GitLab 13.5 and earlier, there is only one checkbox, named + **Enable merge trains and pipelines for merged results**. If you want to keep the **Pipelines must succeed** option selected along with merge -trains, create a new pipeline for merged results when this error occurs: +trains, create a new merged results pipeline when this error occurs: 1. On the **Pipelines** tab, select **Run pipeline**. 1. Select **Start/Add to merge train when pipeline succeeds**. @@ -215,8 +234,8 @@ In [GitLab 13.6 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/244831) you can [enable or disable merge trains in the project settings](#enable-merge-trains). In GitLab 13.5 and earlier, merge trains are automatically enabled when -[pipelines for merged results](pipelines_for_merged_results.md) are enabled. -To use pipelines for merged results without using merge trains, you can enable a +[merged results pipelines](merged_results_pipelines.md) are enabled. +To use merged results pipelines without using merge trains, you can enable a [feature flag](../../user/feature_flags.md) that blocks the merge trains feature. [GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) diff --git a/doc/ci/pipelines/merged_results_pipelines.md b/doc/ci/pipelines/merged_results_pipelines.md new file mode 100644 index 00000000000..4794107cc87 --- /dev/null +++ b/doc/ci/pipelines/merged_results_pipelines.md @@ -0,0 +1,78 @@ +--- +stage: Verify +group: Pipeline Execution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Merged results pipelines **(PREMIUM)** + +> [Renamed](https://gitlab.com/gitlab-org/gitlab/-/issues/351192) from `pipelines for merged results` to `merged results pipelines` in GitLab 14.8. + +A *merged results pipeline* is a type of [merge request pipeline](merge_request_pipelines.md). It is a pipeline that runs against the results of the source and target branches merged together. + +GitLab creates an internal commit with the merged results, so the pipeline can run +against it. This commit does not exist in either branch, +but you can view it in the pipeline details. + +The pipeline runs against the target branch as it exists at the moment you run the pipeline. +Over time, while you're working in the source branch, the target branch might change. +Any time you want to be sure the merged results are accurate, you should re-run the pipeline. + +Merged results pipelines can't run when: + +- The target branch has changes that conflict with the changes in the source branch. +- The merge request is a [**Draft** merge request](../../user/project/merge_requests/drafts.md). + +In these cases, the pipeline runs as a [merge request pipeline](merge_request_pipelines.md) +and is labeled as `detached`. + +## Prerequisites + +To use merged results pipelines: + +- Your project's [CI/CD configuration file](../yaml/index.md) must be configured to + [run jobs in merge request pipelines](merge_request_pipelines.md#prerequisites). +- Your repository must be a GitLab repository, not an + [external repository](../ci_cd_for_external_repos/index.md). +- You must not be using [fast forward merges](../../user/project/merge_requests/fast_forward_merge.md). + [An issue exits](https://gitlab.com/gitlab-org/gitlab/-/issues/26996) to change this behavior. + +## Enable merged results pipelines + +To enable merged results pipelines in a project, you must have at least the +Maintainer role: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > General**. +1. Expand **Merge requests**. +1. Select **Enable merged results pipelines**. +1. Select **Save changes**. + +WARNING: +If you select the checkbox but don't configure your pipeline to use +merge request pipelines, your merge requests may become stuck in an +unresolved state or your pipelines may be dropped. + +## Troubleshooting + +### Merged results pipelines are not created + +In GitLab 13.7 and earlier, merged results pipelines might not be created due +to a disabled [feature flag](../../user/feature_flags.md). This feature flag +[was removed](https://gitlab.com/gitlab-org/gitlab/-/issues/299115) in GitLab 13.8. +Upgrade to 13.8 or later, or make sure the `:merge_ref_auto_sync` +[feature flag is enabled](../../administration/feature_flags.md#check-if-a-feature-flag-is-enabled) +on your GitLab instance. + +### Pipelines fail intermittently with a `fatal: reference is not a tree:` error + +Merged results pipelines run on a merge ref for a merge request +(`refs/merge-requests/<iid>/merge`), so the Git reference could be overwritten at an +unexpected time. + +For example, when a source or target branch is advanced, the pipeline fails with +the `fatal: reference is not a tree:` error, which indicates that the checkout-SHA +is not found in the merge ref. + +This behavior was improved in GitLab 12.4 by introducing [persistent pipeline refs](../troubleshooting.md#fatal-reference-is-not-a-tree-error). +Upgrade to GitLab 12.4 or later to resolve the problem. diff --git a/doc/ci/pipelines/multi_project_pipelines.md b/doc/ci/pipelines/multi_project_pipelines.md index 8a83e7e31f4..2163527e3ca 100644 --- a/doc/ci/pipelines/multi_project_pipelines.md +++ b/doc/ci/pipelines/multi_project_pipelines.md @@ -283,7 +283,7 @@ tag in a different project. Prerequisites: - The upstream project must be [public](../../public_access/public_access.md). -- The user must have the [Developer role](../../user/permissions.md#project-members-permissions) +- The user must have the Developer role in the upstream project. To trigger the pipeline when the upstream project is rebuilt: diff --git a/doc/ci/pipelines/parent_child_pipelines.md b/doc/ci/pipelines/parent_child_pipelines.md index 5e4b707a38c..a3ded24e8c9 100644 --- a/doc/ci/pipelines/parent_child_pipelines.md +++ b/doc/ci/pipelines/parent_child_pipelines.md @@ -100,7 +100,7 @@ microservice_a: ## Merge request child pipelines -To trigger a child pipeline as a [Merge Request Pipeline](merge_request_pipelines.md) we need to: +To trigger a child pipeline as a [merge request pipeline](merge_request_pipelines.md) we need to: - Set the trigger job to run on merge requests: diff --git a/doc/ci/pipelines/pipeline_artifacts.md b/doc/ci/pipelines/pipeline_artifacts.md index b174b6af9f9..e9dd1b2a942 100644 --- a/doc/ci/pipelines/pipeline_artifacts.md +++ b/doc/ci/pipelines/pipeline_artifacts.md @@ -1,8 +1,7 @@ --- stage: Verify -group: Testing +group: Pipeline Insights info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference, howto --- # Pipeline artifacts **(FREE)** diff --git a/doc/ci/pipelines/pipelines_for_merged_results.md b/doc/ci/pipelines/pipelines_for_merged_results.md index 91a49a48882..457f5ce9c30 100644 --- a/doc/ci/pipelines/pipelines_for_merged_results.md +++ b/doc/ci/pipelines/pipelines_for_merged_results.md @@ -1,133 +1,9 @@ --- -stage: Verify -group: Pipeline Execution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'merged_results_pipelines.md' +remove_date: '2022-04-27' --- -# Pipelines for merged results **(PREMIUM)** +This document was moved to [another location](merged_results_pipelines.md). -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7380) in GitLab 11.10. - -When you submit a merge request, you are requesting to merge changes from a -source branch into a target branch. By default, the CI pipeline runs jobs -against the source branch. - -With *pipelines for merged results*, the pipeline runs as if the changes from -the source branch have already been merged into the target branch. The commit shown for the pipeline does not exist on the source or target branches but represents the combined target and source branches. - - - -If the pipeline fails due to a problem in the target branch, you can wait until the -target is fixed and re-run the pipeline. -This new pipeline runs as if the source is merged with the updated target, and you -don't need to rebase. - -The pipeline does not automatically run when the target branch changes. Only changes -to the source branch trigger a new pipeline. If a long time has passed since the last successful -pipeline, you may want to re-run it before merge, to ensure that the source changes -can still be successfully merged into the target. - -When the merge request can't be merged, the pipeline runs against the source branch only. For example, when: - -- The target branch has changes that conflict with the changes in the source branch. -- The merge request is a [**Draft** merge request](../../user/project/merge_requests/drafts.md). - -In these cases, the pipeline runs as a [pipeline for merge requests](merge_request_pipelines.md) -and is labeled as `detached`. If these cases no longer exist, new pipelines -again run against the merged results. - -Any user who has developer [permissions](../../user/permissions.md) can run a -pipeline for merged results. - -## Prerequisites - -To enable pipelines for merge results: - -- You must have the [Maintainer role](../../user/permissions.md). -- You must be using [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) 11.9 or later. -- You must not be using - [fast forward merges](../../user/project/merge_requests/fast_forward_merge.md) yet. - To follow progress, see [#26996](https://gitlab.com/gitlab-org/gitlab/-/issues/26996). -- Your repository must be a GitLab repository, not an - [external repository](../ci_cd_for_external_repos/index.md). - -## Enable pipelines for merged results - -To enable pipelines for merged results for your project: - -1. [Configure your CI/CD configuration file](merge_request_pipelines.md#prerequisites) - so that the pipeline or individual jobs run for merge requests. -1. Visit your project's **Settings > General** and expand **Merge requests**. -1. Check **Enable merged results pipelines**. -1. Click **Save changes**. - -WARNING: -If you select the checkbox but don't configure your CI/CD to use -pipelines for merge requests, your merge requests may become stuck in an -unresolved state or your pipelines may be dropped. - -## Using Merge Trains - -When you enable [Pipelines for merged results](#pipelines-for-merged-results), -GitLab [automatically displays](merge_trains.md#add-a-merge-request-to-a-merge-train) -a **Start/Add Merge Train button**. - -Generally, this is a safer option than merging merge requests immediately, because your -merge request is evaluated with an expected post-merge result before the actual -merge happens. - -For more information, read the [documentation on Merge Trains](merge_trains.md). - -## Automatic pipeline cancellation - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12996) in GitLab 12.3. - -GitLab CI/CD can detect the presence of redundant pipelines, and cancels them -to conserve CI resources. - -When a user merges a merge request immediately in an ongoing merge -train, the train is reconstructed, because it recreates the expected -post-merge commit and pipeline. In this case, the merge train may already -have pipelines running against the previous expected post-merge commit. -These pipelines are considered redundant and are automatically -canceled. - -## Troubleshooting - -### Pipelines for merged results not created even with new change pushed to merge request - -Can be caused by some disabled feature flags. Please make sure that -the following feature flags are enabled on your GitLab instance: - -- `:merge_ref_auto_sync` - -To check and set these feature flag values, please ask an administrator to: - -1. Log into the Rails console of the GitLab instance: - - ```shell - sudo gitlab-rails console - ``` - -1. Check if the flags are enabled or not: - - ```ruby - Feature.enabled?(:merge_ref_auto_sync) - ``` - -1. If needed, enable the feature flags: - - ```ruby - Feature.enable(:merge_ref_auto_sync) - ``` - -### Intermittently pipelines fail by `fatal: reference is not a tree:` error - -Since pipelines for merged results are a run on a merge ref of a merge request -(`refs/merge-requests/<iid>/merge`), the Git reference could be overwritten at an -unexpected timing. For example, when a source or target branch is advanced. -In this case, the pipeline fails because of `fatal: reference is not a tree:` error, -which indicates that the checkout-SHA is not found in the merge ref. - -This behavior was improved at GitLab 12.4 by introducing [Persistent pipeline refs](../troubleshooting.md#fatal-reference-is-not-a-tree-error). -You should be able to create pipelines at any timings without concerning the error. +<!-- This redirect file can be deleted after 2022-04-27. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/pipelines/schedules.md b/doc/ci/pipelines/schedules.md index 692460120fe..fe9db3306cd 100644 --- a/doc/ci/pipelines/schedules.md +++ b/doc/ci/pipelines/schedules.md @@ -17,7 +17,9 @@ Pipeline schedules can be used to also run [pipelines](index.md) at specific int - Every other Sunday at 0900 hours (cron example: `0 9 * * sun%2`). - Once every day (cron example: `0 0 * * *`). -Schedule timing is configured with cron notation, parsed by [Fugit](https://github.com/floraison/fugit). +Schedule timing is configured with [cron notation](../../topics/cron/index.md). +You can use any cron value, but scheduled pipelines cannot run more frequently +than the instance's [maximum frequency for scheduled pipelines](#advanced-configuration). In addition to using the GitLab UI, pipeline schedules can be maintained using the [Pipeline schedules API](../../api/pipeline_schedules.md). @@ -82,20 +84,24 @@ job: ### Advanced configuration **(FREE SELF)** -The pipelines are not executed exactly on schedule because schedules are handled by -Sidekiq, which runs according to its interval. +Scheduled pipelines can be configured with any [cron value](../../topics/cron/index.md), +but they do not always run exactly when scheduled. An internal process, called the +_pipeline schedule worker_, queues all the scheduled pipelines, but does not +run continuously. The worker runs on its own schedule, and scheduled pipelines that +are ready to start are only queued the next time the worker runs. Scheduled pipelines +can't run more frequently than the worker. -For example, only two pipelines are created per day if: +The default frequency of the pipeline schedule worker is `3-59/10 * * * *` (every ten minutes, +starting with `0:03`, `0:13`, `0:23`, and so on). The default frequency for GitLab.com +is listed in the [GitLab.com settings](../../user/gitlab_com/index.md#gitlab-cicd). -- You set a schedule to create a pipeline every minute (`* * * * *`). -- The Sidekiq worker runs on 00:00 and 12:00 every day (`0 */12 * * *`). - -To change the Sidekiq worker's frequency: +To change the frequency of the pipeline schedule worker: 1. Edit the `gitlab_rails['pipeline_schedule_worker_cron']` value in your instance's `gitlab.rb` file. 1. [Reconfigure GitLab](../../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -For GitLab.com, refer to the [dedicated settings page](../../user/gitlab_com/index.md#gitlab-cicd). +For example, to set the maximum frequency of pipelines to twice a day, set `pipeline_schedule_worker_cron` +to a cron value of `0 */12 * * *` (`00:00` and `12:00` every day). ## Working with scheduled pipelines diff --git a/doc/ci/pipelines/settings.md b/doc/ci/pipelines/settings.md index 85824dfb7c7..e22746dbfa0 100644 --- a/doc/ci/pipelines/settings.md +++ b/doc/ci/pipelines/settings.md @@ -108,6 +108,10 @@ To customize the path: - Is on an external site, enter the full URL. 1. Select **Save changes**. +NOTE: +You cannot use your project's [pipeline editor](../pipeline_editor/index.md) to +edit CI/CD configuration files in other projects or on an external site. + ### Custom CI/CD configuration file examples If the CI/CD configuration file is not in the root directory, the path must be relative to it. @@ -191,21 +195,18 @@ Jobs that exceed the timeout are marked as failed. You can override this value [for individual runners](../runners/configure_runners.md#set-maximum-job-timeout-for-a-runner). -## Add test coverage results to a merge request +## Add test coverage results to a merge request (DEPRECATED) + +> [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/17633) in GitLab 14.9. Replaced by [`coverage` keyword](../yaml/index.md#coverage). + +WARNING: +This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/17633) +for use in GitLab 14.9, and is planned for [removal](https://gitlab.com/gitlab-org/gitlab/-/issues/17633) in GitLab 15.0. If you use test coverage in your code, you can use a regular expression to find coverage results in the job log. You can then include these results in the merge request in GitLab. -1. On the top bar, select **Menu > Projects** and find your project. -1. On the left sidebar, select **Settings > CI/CD**. -1. Expand **General pipelines**. -1. In the **Test coverage parsing** field, enter a regular expression. - Leave blank to disable this feature. - -You can use <https://rubular.com> to test your regex. The regex returns the **last** -match found in the output. - If the pipeline succeeds, the coverage is shown in the merge request widget and in the jobs table. If multiple jobs in the pipeline have coverage reports, they are averaged. @@ -214,6 +215,31 @@ averaged.  +To define a coverage-parsing regular expression: + +- Using the project's `.gitlab-ci.yml`, provide a regular expression using the [`coverage`](../yaml/index.md#coverage) + keyword. Setting the regular expression this way takes precedence over the project's CI/CD settings. + +- Using the Project's CI/CD settings: + - Set using the GitLab UI: + + 1. On the top bar, select **Menu > Projects** and find your project. + 1. On the left sidebar, select **Settings > CI/CD**. + 1. Expand **General pipelines**. + 1. In the **Test coverage parsing** field, enter a regular expression. Leave blank to disable this feature. + + - Set when [editing a project](../../api/projects.md#edit-project) or [creating a project](../../api/projects.md#create-project) + using the GitLab API with the `build_coverage_regex` attribute: + + ```shell + curl --request PUT --header "PRIVATE-TOKEN: <your-token>" \ + --url 'https://gitlab.com/api/v4/projects/<your-project-ID>' \ + --data "build_coverage_regex=<your-regular-expression>" + ``` + +You can use <https://rubular.com> to test your regular expression. The regular expression returns the **last** +match found in the output. + ### Test coverage examples Use this regex for commonly used test tools. @@ -338,7 +364,7 @@ https://gitlab.example.com/<namespace>/<project>/badges/<branch>/pipeline.svg?ig ### Test coverage report badge -You can define the regular expression for the [coverage report](#add-test-coverage-results-to-a-merge-request) +You can define the regular expression for the [coverage report](#add-test-coverage-results-to-a-merge-request-deprecated) that each job log is matched against. This means that each job in the pipeline can have the test coverage percentage value defined. diff --git a/doc/ci/quick_start/index.md b/doc/ci/quick_start/index.md index 77a666c0cca..fbdf226181b 100644 --- a/doc/ci/quick_start/index.md +++ b/doc/ci/quick_start/index.md @@ -12,7 +12,7 @@ Use this document to get started with [GitLab CI/CD](../index.md). Before you start, make sure you have: - A project in GitLab that you would like to use CI/CD for. -- The [Maintainer or Owner role](../../user/permissions.md) for the project. +- The Maintainer or Owner role for the project. If you are migrating from another CI/CD tool, view this documentation: diff --git a/doc/ci/resource_groups/index.md b/doc/ci/resource_groups/index.md index d31fb5561e9..9312f4a8850 100644 --- a/doc/ci/resource_groups/index.md +++ b/doc/ci/resource_groups/index.md @@ -58,7 +58,7 @@ can still run `build` jobs concurrently for maximizing the pipeline efficiency. - The basic knowledge of the [GitLab CI/CD pipelines](../pipelines/index.md) - The basic knowledge of the [GitLab Environments and Deployments](../environments/index.md) -- [Developer role](../../user/permissions.md) (or above) in the project to configure CI/CD pipelines. +- At least the Developer role for the project to configure CI/CD pipelines. ### Limitations diff --git a/doc/ci/review_apps/index.md b/doc/ci/review_apps/index.md index 37005939eb7..ed29b33bc3a 100644 --- a/doc/ci/review_apps/index.md +++ b/doc/ci/review_apps/index.md @@ -1,8 +1,7 @@ --- stage: Verify -group: Testing +group: Pipeline Insights info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference --- # Review Apps **(FREE)** @@ -73,7 +72,7 @@ you can copy and paste into `.gitlab-ci.yml` as a starting point. Prerequisite: -- You need at least the Developer [role](../../user/permissions.md) for the project. +- You need at least the Developer role for the project. To use the Review Apps template: diff --git a/doc/ci/runners/configure_runners.md b/doc/ci/runners/configure_runners.md index d826b28acce..60e21653a45 100644 --- a/doc/ci/runners/configure_runners.md +++ b/doc/ci/runners/configure_runners.md @@ -160,7 +160,7 @@ the GitLab instance. To determine this: ### Determine the IP address of a specific runner To can find the IP address of a runner for a specific project, -you must have the [Owner role](../../user/permissions.md#project-members-permissions) for the +you must have the Owner role for the project. 1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. @@ -187,7 +187,7 @@ the appropriate dependencies to run Rails test suites. When you [register a runner](https://docs.gitlab.com/runner/register/), its default behavior is to **only pick** [tagged jobs](../yaml/index.md#tags). -To change this, you must have the [Owner role](../../user/permissions.md#project-members-permissions) for the project. +To change this, you must have the Owner role for the project. To make a runner pick untagged jobs: @@ -291,6 +291,7 @@ globally or for individual jobs: - [`GIT_CHECKOUT`](#git-checkout) - [`GIT_CLEAN_FLAGS`](#git-clean-flags) - [`GIT_FETCH_EXTRA_FLAGS`](#git-fetch-extra-flags) +- [`GIT_SUBMODULE_UPDATE_FLAGS`](#git-submodule-update-flags) - [`GIT_DEPTH`](#shallow-cloning) (shallow cloning) - [`GIT_CLONE_PATH`](#custom-build-directories) (custom build directories) - [`TRANSFER_METER_FREQUENCY`](#artifact-and-cache-settings) (artifact/cache meter update frequency) @@ -331,10 +332,6 @@ try to preserve worktrees and try to re-use them by default. This has limitations when using the [Docker Machine executor](https://docs.gitlab.com/runner/executors/docker_machine.html). -It does not work for [the `kubernetes` executor](https://docs.gitlab.com/runner/executors/kubernetes.html), -but a [feature proposal](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3847) exists. -The `kubernetes` executor always clones into an temporary directory. - A Git strategy of `none` also re-uses the local working copy, but skips all Git operations normally done by GitLab. GitLab Runner pre-clone scripts are also skipped, if present. This strategy could mean you need to add `fetch` and `checkout` commands @@ -382,6 +379,8 @@ For this feature to work correctly, the submodules must be configured - a relative path to another repository on the same GitLab server. See the [Git submodules](../git_submodules.md) documentation. +You can provide additional flags to control advanced behavior using [`GIT_SUBMODULE_UPDATE_FLAGS`](#git-submodule-update-flags). + ### Git checkout > Introduced in GitLab Runner 9.3. @@ -442,14 +441,14 @@ script: > [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4142) in GitLab Runner 13.1. -The `GIT_FETCH_EXTRA_FLAGS` variable is used to control the behavior of +Use the `GIT_FETCH_EXTRA_FLAGS` variable to control the behavior of `git fetch`. You can set it globally or per-job in the [`variables`](../yaml/index.md#variables) section. `GIT_FETCH_EXTRA_FLAGS` accepts all options of the [`git fetch`](https://git-scm.com/docs/git-fetch) command. However, `GIT_FETCH_EXTRA_FLAGS` flags are appended after the default flags that can't be modified. The default flags are: -- [GIT_DEPTH](#shallow-cloning). +- [`GIT_DEPTH`](#shallow-cloning). - The list of [refspecs](https://git-scm.com/book/en/v2/Git-Internals-The-Refspec). - A remote called `origin`. @@ -475,6 +474,47 @@ git fetch origin $REFSPECS --depth 50 --prune Where `$REFSPECS` is a value provided to the runner internally by GitLab. +### Git submodule update flags + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/3192) in GitLab Runner 14.8. + +Use the `GIT_SUBMODULE_UPDATE_FLAGS` variable to control the behavior of `git submodule update` +when [`GIT_SUBMODULE_STRATEGY`](#git-submodule-strategy) is set to either `normal` or `recursive`. +You can set it globally or per-job in the [`variables`](../yaml/index.md#variables) section. + +`GIT_SUBMODULE_UPDATE_FLAGS` accepts all options of the +[`git submodule update`](https://git-scm.com/docs/git-submodule#Documentation/git-submodule.txt-update--init--remote-N--no-fetch--no-recommend-shallow-f--force--checkout--rebase--merge--referenceltrepositorygt--depthltdepthgt--recursive--jobsltngt--no-single-branch--ltpathgt82308203) +subcommand. However, note that `GIT_SUBMODULE_UPDATE_FLAGS` flags are appended after a few default flags: + +- `--init`, if [`GIT_SUBMODULE_STRATEGY`](#git-submodule-strategy) was set to `normal` or `recursive`. +- `--recursive`, if [`GIT_SUBMODULE_STRATEGY`](#git-submodule-strategy) was set to `recursive`. +- [`GIT_DEPTH`](#shallow-cloning). See the default value below. + +Git honors the last occurrence of a flag in the list of arguments, so manually +providing them in `GIT_SUBMODULE_UPDATE_FLAGS` will also override these default flags. + +You can use this variable to fetch the latest remote `HEAD` instead of the commit tracked, +in the repository, or to speed up the checkout by fetching submodules in multiple parallel jobs: + +```yaml +variables: + GIT_SUBMODULE_STRATEGY: recursive + GIT_SUBMODULE_UPDATE_FLAGS: --remote --jobs 4 +script: + - ls -al .git/modules/ +``` + +The configuration above results in `git submodule update` being called this way: + +```shell +git submodule update --init --depth 50 --recursive --remote --jobs 4 +``` + +WARNING: +You should be aware of the implications for the security, stability, and reproducibility of +your builds when using the `--remote` flag. In most cases, it is better to explicitly track +submodule commits as designed, and update them using an auto-remediation/dependency bot. + ### Shallow cloning > Introduced in GitLab 8.9 as an experimental feature. diff --git a/doc/ci/runners/runners_scope.md b/doc/ci/runners/runners_scope.md index f76a767abec..7cfd8e50f6c 100644 --- a/doc/ci/runners/runners_scope.md +++ b/doc/ci/runners/runners_scope.md @@ -55,7 +55,7 @@ To enable shared runners: ### Disable shared runners You can disable shared runners for individual projects or for groups. -You must have the [Owner role](../../user/permissions.md#group-members-permissions) for the project +You must have the Owner role for the project or group. To disable shared runners for a project: @@ -144,7 +144,7 @@ Group runners process jobs by using a first in, first out ([FIFO](https://en.wik ### Create a group runner You can create a group runner for your self-managed GitLab instance or for GitLab.com. -You must have the [Owner role](../../user/permissions.md#group-members-permissions) for the group. +You must have the Owner role for the group. To create a group runner: @@ -160,7 +160,7 @@ To create a group runner: You can view and manage all runners for a group, its subgroups, and projects. You can do this for your self-managed GitLab instance or for GitLab.com. -You must have the [Owner role](../../user/permissions.md#group-members-permissions) for the group. +You must have the Owner role for the group. 1. Go to the group where you want to view the runners. 1. Go to **Settings > CI/CD** and expand the **Runners** section. @@ -183,7 +183,7 @@ From this page, you can edit, pause, and remove runners from the group, its subg ### Pause or remove a group runner You can pause or remove a group runner for your self-managed GitLab instance or for GitLab.com. -You must have the [Owner role](../../user/permissions.md#group-members-permissions) for the group. +You must have the Owner role for the group. 1. Go to the group you want to remove or pause the runner for. 1. Go to **Settings > CI/CD** and expand the **Runners** section. @@ -213,7 +213,7 @@ A fork *does* copy the CI/CD settings of the cloned repository. ### Create a specific runner You can create a specific runner for your self-managed GitLab instance or for GitLab.com. -You must have the [Owner role](../../user/permissions.md#project-members-permissions) for the project. +You must have the Owner role for the project. To create a specific runner: @@ -227,7 +227,7 @@ To create a specific runner: A specific runner is available in the project it was created for. An administrator can enable a specific runner to apply to additional projects. -- You must have the [Owner role](../../user/permissions.md#group-members-permissions) for the +- You must have the Owner role for the project. - The specific runner must not be [locked](#prevent-a-specific-runner-from-being-enabled-for-other-projects). diff --git a/doc/ci/runners/saas/macos_saas_runner.md b/doc/ci/runners/saas/macos_saas_runner.md index 40c4deb51aa..bfe408f4485 100644 --- a/doc/ci/runners/saas/macos_saas_runner.md +++ b/doc/ci/runners/saas/macos_saas_runner.md @@ -12,7 +12,7 @@ Use these runners to build, test, and deploy apps for the Apple ecosystem (macOS of all the capabilities of the GitLab single DevOps platform and not have to manage or operate a build environment. -SaaS runners on macOS are in [Beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta) +SaaS runners on macOS are in [Beta](../../../policy/alpha-beta-support.md#beta-features) and shouldn't be relied upon for mission-critical production jobs. ## Quickstart diff --git a/doc/ci/runners/saas/windows_saas_runner.md b/doc/ci/runners/saas/windows_saas_runner.md index b08be14dbc3..dddb3afee7c 100644 --- a/doc/ci/runners/saas/windows_saas_runner.md +++ b/doc/ci/runners/saas/windows_saas_runner.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # SaaS runners on Windows (beta) **(FREE SAAS)** -SaaS runners on Windows are in [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta) +SaaS runners on Windows are in [beta](../../../policy/alpha-beta-support.md#beta-features) and shouldn't be used for production workloads. During this beta period, the [shared runner quota for CI/CD minutes](../../pipelines/cicd_minutes.md) @@ -15,14 +15,14 @@ change when the beta period ends, as discussed in this [related issue](https://g Windows runners on GitLab.com autoscale by launching virtual machines on the Google Cloud Platform. This solution uses an -[autoscaling driver](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/tree/master/docs/readme.md) +[autoscaling driver](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/-/blob/main/docs/README.md) developed by GitLab for the [custom executor](https://docs.gitlab.com/runner/executors/custom.html). Windows runners execute your CI/CD jobs on `n1-standard-2` instances with 2 vCPUs and 7.5 GB RAM. You can find a full list of available Windows packages in the [package documentation](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/gcp/windows-containers/blob/main/cookbooks/preinstalled-software/README.md). We want to keep iterating to get Windows runners in a stable state and -[generally available](https://about.gitlab.com/handbook/product/gitlab-the-product/#generally-available-ga). +[generally available](../../../policy/alpha-beta-support.md#generally-available-ga). You can follow our work towards this goal in the [related epic](https://gitlab.com/groups/gitlab-org/-/epics/2162). @@ -127,7 +127,7 @@ test: ## Limitations and known issues - All the limitations mentioned in our [beta - definition](https://about.gitlab.com/handbook/product/#beta). + definition](../../../policy/alpha-beta-support.md#beta-features). - The average provisioning time for a new Windows VM is 5 minutes. This means that you may notice slower build start times on the Windows runner fleet during the beta. In a future diff --git a/doc/ci/secrets/index.md b/doc/ci/secrets/index.md index c0a763c80f0..ea0c0d9cc84 100644 --- a/doc/ci/secrets/index.md +++ b/doc/ci/secrets/index.md @@ -1,6 +1,6 @@ --- -stage: Configure -group: Configure +stage: Verify +group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: concepts, howto --- diff --git a/doc/ci/services/index.md b/doc/ci/services/index.md index 51df8c102f7..d14027cb1ef 100644 --- a/doc/ci/services/index.md +++ b/doc/ci/services/index.md @@ -260,10 +260,12 @@ test: | `entrypoint` | no | 9.4 |Command or script to execute as the container's entrypoint. It's translated to Docker's `--entrypoint` option while creating the container. The syntax is similar to [`Dockerfile`'s `ENTRYPOINT`](https://docs.docker.com/engine/reference/builder/#entrypoint) directive, where each shell token is a separate string in the array. | | `command` | no | 9.4 |Command or script that should be used as the container's command. It's translated to arguments passed to Docker after the image's name. The syntax is similar to [`Dockerfile`'s `CMD`](https://docs.docker.com/engine/reference/builder/#cmd) directive, where each shell token is a separate string in the array. | | `alias` (1) | no | 9.4 | Additional alias that can be used to access the service from the job's container. Read [Accessing the services](#accessing-the-services) for more information. | -| `variables` | no | 14.5 | Additional environment variables that are passed exclusively to the service. The syntax is the same as [Job Variables](../variables/index.md). | +| `variables` (2) | no | 14.5 | Additional environment variables that are passed exclusively to the service. The syntax is the same as [Job Variables](../variables/index.md). | (1) Alias support for the Kubernetes executor was [introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2229) in GitLab Runner 12.8, and is only available for Kubernetes version 1.7 or later. +(2) Service variables support for the Docker and the Kubernetes executor was [introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/3158) in GitLab Runner 14.8. + ## Starting multiple services from the same image > Introduced in GitLab and GitLab Runner 9.4. Read more about the [extended configuration options](../docker/using_docker_images.md#extended-docker-configuration-options). @@ -387,7 +389,7 @@ time. ## Debug a job locally The following commands are run without root privileges. You should be -able to run Docker with your regular user account. +able to run Docker with your user account. First start with creating a file named `build_script`: diff --git a/doc/ci/test_cases/index.md b/doc/ci/test_cases/index.md index 384bfc10779..3ef25a4924b 100644 --- a/doc/ci/test_cases/index.md +++ b/doc/ci/test_cases/index.md @@ -20,7 +20,7 @@ use external test planning tools, which require additional overhead, context swi Prerequisite: -- You must have at least the Reporter [role](../../user/permissions.md). +- You must have at least the Reporter role. To create a test case in a GitLab project: @@ -36,7 +36,7 @@ issue list with a search query, including labels or the test case's title. Prerequisite: -- You must have at least the Guest [role](../../user/permissions.md). +- You must have at least the Guest role. To view a test case: @@ -51,7 +51,7 @@ You can edit a test case's title and description. Prerequisite: -- You must have at least the Reporter [role](../../user/permissions.md). +- You must have at least the Reporter role. - Users demoted to the Guest role can continue to edit the test cases they created when they were in the higher role. @@ -68,7 +68,7 @@ When you want to stop using a test case, you can archive it. You can [reopen an Prerequisite: -- You must have at least the Reporter [role](../../user/permissions.md). +- You must have at least the Reporter role. To archive a test case, on the test case's page, select the **Archive test case** button. @@ -81,6 +81,6 @@ To view archived test cases: If you decide to start using an archived test case again, you can reopen it. -You must have at least the Reporter [role](../../user/permissions.md). +You must have at least the Reporter role. To reopen an archived test case, on the test case's page, select **Reopen test case**. diff --git a/doc/ci/triggers/index.md b/doc/ci/triggers/index.md index 1b648a486ef..2c916ba092c 100644 --- a/doc/ci/triggers/index.md +++ b/doc/ci/triggers/index.md @@ -22,7 +22,7 @@ to authenticate an API call. The token impersonates a user's project access and Prerequisite: -- You must have at least the [Maintainer role](../../user/permissions.md) for the project. +- You must have at least the Maintainer role for the project. To create a trigger token: diff --git a/doc/ci/troubleshooting.md b/doc/ci/troubleshooting.md index 4d550f6da13..81cb924532c 100644 --- a/doc/ci/troubleshooting.md +++ b/doc/ci/troubleshooting.md @@ -36,7 +36,7 @@ file is correct. Paste in full `.gitlab-ci.yml` files or individual jobs configu to verify the basic syntax. When a `.gitlab-ci.yml` file is present in a project, you can also use the CI Lint -tool to [simulate the creation of a full pipeline](lint.md#pipeline-simulation). +tool to [simulate the creation of a full pipeline](lint.md#simulate-a-pipeline). It does deeper verification of the configuration syntax. ## Verify variables @@ -69,12 +69,12 @@ if you are using that type: and run separate pipelines in the same project. You can also [dynamically generate the child pipeline's configuration](pipelines/parent_child_pipelines.md#dynamic-child-pipelines) at runtime. -- [Pipelines for Merge Requests](pipelines/merge_request_pipelines.md): Run a pipeline +- [Merge request pipelines](pipelines/merge_request_pipelines.md): Run a pipeline in the context of a merge request. - - [Pipelines for Merge Results](pipelines/pipelines_for_merged_results.md): - Pipelines for merge requests that run on the combined source and target branch - - [Merge Trains](pipelines/merge_trains.md): - Multiple pipelines for merged results that queue and run automatically before + - [Merged results pipelines](pipelines/merged_results_pipelines.md): + Merge request pipelines that run on the combined source and target branch + - [Merge trains](pipelines/merge_trains.md): + Multiple merged results pipelines that queue and run automatically before changes are merged. ### Troubleshooting Guides for CI/CD features @@ -173,8 +173,8 @@ a branch to its remote repository. To illustrate the problem, suppose you've had This occurs because the previous pipeline cannot find a checkout-SHA (which is associated with the pipeline record) from the `example` branch that the commit history has already been overwritten by the force-push. -Similarly, [Pipelines for merged results](pipelines/pipelines_for_merged_results.md) -might have failed intermittently due to [the same reason](pipelines/pipelines_for_merged_results.md#intermittently-pipelines-fail-by-fatal-reference-is-not-a-tree-error). +Similarly, [Merged results pipelines](pipelines/merged_results_pipelines.md) +might have failed intermittently due to [the same reason](pipelines/merged_results_pipelines.md#pipelines-fail-intermittently-with-a-fatal-reference-is-not-a-tree-error). As of GitLab 12.4, we've improved this behavior by persisting pipeline refs exclusively. To illustrate its life cycle: @@ -192,6 +192,21 @@ To illustrate its life cycle: The merge request pipeline widget shows information about the pipeline status in a merge request. It's displayed above the [ability to merge status widget](#merge-request-status-messages). +#### "Checking ability to merge automatically" message + +There is a [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/229352) +where a merge request can be stuck with the `Checking ability to merge automatically` +message. + +If your merge request has this message and it does not disappear after a few minutes, +you can try one of these workarounds: + +- Refresh the merge request page. +- Close & Re-open the merge request. +- Rebase the merge request with the `/rebase` [quick action](../user/project/quick_actions.md). +- If you have already confirmed the merge request is ready to be merged, you can merge + it with the `/merge` quick action. + #### "Checking pipeline status" message This message is shown when the merge request has no pipeline associated with the @@ -226,10 +241,10 @@ This also applies if the pipeline has not been created yet, or if you are waitin for an external CI service. If you don't use pipelines for your project, then you should disable **Pipelines must succeed** so you can accept merge requests. -### "The pipeline for this merge request did not complete. Push a new commit to fix the failure or check the troubleshooting documentation to see other possible actions." message +#### "Merge blocked: pipeline must succeed. Push a new commit that fixes the failure" message This message is shown if the [merge request pipeline](pipelines/merge_request_pipelines.md), -[merged results pipeline](pipelines/pipelines_for_merged_results.md), +[merged results pipeline](pipelines/merged_results_pipelines.md), or [merge train pipeline](pipelines/merge_trains.md) has failed or been canceled. diff --git a/doc/ci/unit_test_reports.md b/doc/ci/unit_test_reports.md index 55fd8c1eb49..e2de47c6c62 100644 --- a/doc/ci/unit_test_reports.md +++ b/doc/ci/unit_test_reports.md @@ -1,8 +1,7 @@ --- stage: Verify -group: Testing +group: Pipeline Insights info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference --- # Unit test reports **(FREE)** @@ -21,7 +20,7 @@ report on the merge request so that it's easier and faster to identify the failure without having to check the entire log. Unit test reports currently only support test reports in the JUnit report format. -If you don't use Merge Requests but still want to see the unit test report +If you don't use merge requests but still want to see the unit test report output without searching through job logs, the full [Unit test reports](#viewing-unit-test-reports-on-gitlab) are available in the pipeline detail view. @@ -67,7 +66,7 @@ execution time and the error output. ### Number of recent failures -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241759) in Merge Requests in GitLab 13.7. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241759) in merge requests in GitLab 13.7. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/268249) in GitLab 13.8. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/235525) in Test Reports in GitLab 13.9. @@ -76,7 +75,7 @@ If a test failed in the project's default branch in the last 14 days, a message ## How to set it up -To enable the Unit test reports in merge requests, you need to add +To enable the Unit test reports in merge requests, you must add [`artifacts:reports:junit`](yaml/artifacts_reports.md#artifactsreportsjunit) in `.gitlab-ci.yml`, and specify the path(s) of the generated test reports. The reports must be `.xml` files, otherwise [GitLab returns an Error 500](https://gitlab.com/gitlab-org/gitlab/-/issues/216575). diff --git a/doc/ci/variables/index.md b/doc/ci/variables/index.md index 7ce58b015ca..ba8451110eb 100644 --- a/doc/ci/variables/index.md +++ b/doc/ci/variables/index.md @@ -154,7 +154,7 @@ job: ### Add a CI/CD variable to a project You can add CI/CD variables to a project's settings. Only project members with the -[Maintainer role](../../user/permissions.md#project-members-permissions) +Maintainer role can add or update project CI/CD variables. To keep a CI/CD variable secret, put it in the project settings, not in the `.gitlab-ci.yml` file. @@ -233,7 +233,7 @@ inherited. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/299879) in GitLab 13.11. To make a CI/CD variable available to all projects and groups in a GitLab instance, -add an instance CI/CD variable. You must have the [Administrator role](../../user/permissions.md). +add an instance CI/CD variable. You must have administrator access. You can define instance variables via the UI or [API](../../api/instance_level_ci_variables.md). @@ -372,7 +372,7 @@ You can protect a project, group or instance CI/CD variable so it is only passed to pipelines running on [protected branches](../../user/project/protected_branches.md) or [protected tags](../../user/project/protected_tags.md). -[Pipelines for merge requests](../pipelines/merge_request_pipelines.md) do not have access to protected variables. +[Merge request pipelines](../pipelines/merge_request_pipelines.md) do not have access to protected variables. An [issue exists](https://gitlab.com/gitlab-org/gitlab/-/issues/28002) regarding this limitation. To protect a variable: @@ -651,9 +651,11 @@ which variables take precedence. The order of precedence for variables is (from highest to lowest): -1. [Trigger variables](../triggers/index.md#pass-cicd-variables-in-the-api-call), - [scheduled pipeline variables](../pipelines/schedules.md#using-variables), - and [manual pipeline run variables](#override-a-variable-when-running-a-pipeline-manually). +1. These all have the same (highest) precedence: + - [Trigger variables](../triggers/index.md#pass-cicd-variables-in-the-api-call). + - [Scheduled pipeline variables](../pipelines/schedules.md#using-variables). + - [Manual pipeline run variables](#override-a-variable-when-running-a-pipeline-manually). + - Variables added when [creating a pipeline with the API](../../api/pipelines.md#create-a-new-pipeline). 1. Project [variables](#custom-cicd-variables). 1. Group [variables](#add-a-cicd-variable-to-a-group). 1. Instance [variables](#add-a-cicd-variable-to-an-instance). @@ -900,7 +902,7 @@ if [[ -d "/builds/gitlab-examples/ci-debug-trace/.git" ]]; then > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/292661) in GitLab 13.8. You can restrict access to debug logging. When restricted, only users with -[developer or higher permissions](../../user/permissions.md#project-members-permissions) +at least the Developer role can view job logs when debug logging is enabled with a variable in: - The [`.gitlab-ci.yml` file](#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file). diff --git a/doc/ci/variables/predefined_variables.md b/doc/ci/variables/predefined_variables.md index 60888cd9b97..0f3461b3674 100644 --- a/doc/ci/variables/predefined_variables.md +++ b/doc/ci/variables/predefined_variables.md @@ -27,7 +27,7 @@ There are also a number of [variables you can use to configure runner behavior]( | `CI_API_V4_URL` | 11.7 | all | The GitLab API v4 root URL. | | `CI_BUILDS_DIR` | all | 11.10 | The top-level directory where builds are executed. | | `CI_COMMIT_AUTHOR` | 13.11 | all | The author of the commit in `Name <email>` format. | -| `CI_COMMIT_BEFORE_SHA` | 11.2 | all | The previous latest commit present on a branch. Is always `0000000000000000000000000000000000000000` in pipelines for merge requests. | +| `CI_COMMIT_BEFORE_SHA` | 11.2 | all | The previous latest commit present on a branch. Is always `0000000000000000000000000000000000000000` in merge request pipelines. | | `CI_COMMIT_BRANCH` | 12.6 | 0.5 | The commit branch name. Available in branch pipelines, including pipelines for the default branch. Not available in merge request pipelines or tag pipelines. | | `CI_COMMIT_DESCRIPTION` | 10.8 | all | The description of the commit. If the title is shorter than 100 characters, the message without the first line. | | `CI_COMMIT_MESSAGE` | 10.8 | all | The full commit message. | @@ -63,7 +63,7 @@ There are also a number of [variables you can use to configure runner behavior]( | `CI_JOB_IMAGE` | 12.9 | 12.9 | The name of the Docker image running the job. | | `CI_JOB_JWT` | 12.10 | all | A RS256 JSON web token to authenticate with third party systems that support JWT authentication, for example [HashiCorp's Vault](../secrets/index.md). | | `CI_JOB_JWT_V1` | 14.6 | all | The same value as `CI_JOB_JWT`. | -| `CI_JOB_JWT_V2` | 14.6 | all | [**alpha:**](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga) A newly formatted RS256 JSON web token to increase compatibility. Similar to `CI_JOB_JWT`, except the issuer (`iss`) claim is changed from `gitlab.com` to `https://gitlab.com`, `sub` has changed from `job_id` to a string that contains the project path, and an `aud` claim is added. Format is subject to change. Be aware, the `aud` field is a constant value. Trusting JWTs in multiple relying parties can lead to [one RP sending a JWT to another one and acting maliciously as a job](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/72555#note_769112331). | +| `CI_JOB_JWT_V2` | 14.6 | all | [**alpha:**](../../policy/alpha-beta-support.md#alpha-features) A newly formatted RS256 JSON web token to increase compatibility. Similar to `CI_JOB_JWT`, except the issuer (`iss`) claim is changed from `gitlab.com` to `https://gitlab.com`, `sub` has changed from `job_id` to a string that contains the project path, and an `aud` claim is added. Format is subject to change. Be aware, the `aud` field is a constant value. Trusting JWTs in multiple relying parties can lead to [one RP sending a JWT to another one and acting maliciously as a job](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/72555#note_769112331). | | `CI_JOB_MANUAL` | 8.12 | all | `true` if a job was started manually. | | `CI_JOB_NAME` | 9.0 | 0.5 | The name of the job. | | `CI_JOB_STAGE` | 9.0 | 0.5 | The name of the job's stage. | @@ -148,12 +148,12 @@ These variables are available when: | `CI_MERGE_REQUEST_PROJECT_URL` | 11.6 | all | The URL of the project of the merge request. For example, `http://192.168.10.15:3000/namespace/awesome-project`. | | `CI_MERGE_REQUEST_REF_PATH` | 11.6 | all | The ref path of the merge request. For example, `refs/merge-requests/1/head`. | | `CI_MERGE_REQUEST_SOURCE_BRANCH_NAME` | 11.6 | all | The source branch name of the merge request. | -| `CI_MERGE_REQUEST_SOURCE_BRANCH_SHA` | 11.9 | all | The HEAD SHA of the source branch of the merge request. The variable is empty in merge request pipelines. The SHA is present only in [merged results pipelines](../pipelines/pipelines_for_merged_results.md). **(PREMIUM)** | +| `CI_MERGE_REQUEST_SOURCE_BRANCH_SHA` | 11.9 | all | The HEAD SHA of the source branch of the merge request. The variable is empty in merge request pipelines. The SHA is present only in [merged results pipelines](../pipelines/merged_results_pipelines.md). | | `CI_MERGE_REQUEST_SOURCE_PROJECT_ID` | 11.6 | all | The ID of the source project of the merge request. | | `CI_MERGE_REQUEST_SOURCE_PROJECT_PATH` | 11.6 | all | The path of the source project of the merge request. | | `CI_MERGE_REQUEST_SOURCE_PROJECT_URL` | 11.6 | all | The URL of the source project of the merge request. | | `CI_MERGE_REQUEST_TARGET_BRANCH_NAME` | 11.6 | all | The target branch name of the merge request. | -| `CI_MERGE_REQUEST_TARGET_BRANCH_SHA` | 11.9 | all | The HEAD SHA of the target branch of the merge request. The variable is empty in merge request pipelines. The SHA is present only in [merged results pipelines](../pipelines/pipelines_for_merged_results.md). **(PREMIUM)** | +| `CI_MERGE_REQUEST_TARGET_BRANCH_SHA` | 11.9 | all | The HEAD SHA of the target branch of the merge request. The variable is empty in merge request pipelines. The SHA is present only in [merged results pipelines](../pipelines/merged_results_pipelines.md). | | `CI_MERGE_REQUEST_TITLE` | 11.9 | all | The title of the merge request. | | `CI_MERGE_REQUEST_EVENT_TYPE` | 12.3 | all | The event type of the merge request. Can be `detached`, `merged_result` or `merge_train`. | | `CI_MERGE_REQUEST_DIFF_ID` | 13.7 | all | The version of the merge request diff. | diff --git a/doc/ci/yaml/artifacts_reports.md b/doc/ci/yaml/artifacts_reports.md index 25fb861cfb7..e010dd21b9e 100644 --- a/doc/ci/yaml/artifacts_reports.md +++ b/doc/ci/yaml/artifacts_reports.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Testing +group: Pipeline Insights info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/ci/yaml/includes.md b/doc/ci/yaml/includes.md index 3c94ddb3c14..34db6c61d0b 100644 --- a/doc/ci/yaml/includes.md +++ b/doc/ci/yaml/includes.md @@ -217,6 +217,69 @@ default: - echo "Job complete." ``` +### Use nested includes with duplicate `includes` entries + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/28987) in GitLab 14.8 + +Nested includes can include the same configuration file. The duplicate configuration +file is included multiple times, but the effect is the same as if it was only +included once. + +For example, with the following nested includes, where `defaults.gitlab-ci.yml` +is included multiple times: + +- Contents of the `.gitlab-ci.yml` file: + + ```yaml + include: + - template: defaults.gitlab-ci.yml + - local: unit-tests.gitlab-ci.yml + - local: smoke-tests.gitlab-ci.yml + ``` + +- Contents of the `defaults.gitlab-ci.yml` file: + + ```yaml + default: + before_script: default-before-script.sh + retry: 2 + ``` + +- Contents of the `unit-tests.gitlab-ci.yml` file: + + ```yaml + include: + - template: defaults.gitlab-ci.yml + + unit-test-job: + script: unit-test.sh + retry: 0 + ``` + +- Contents of the `smoke-tests.gitlab-ci.yml` file: + + ```yaml + include: + - template: defaults.gitlab-ci.yml + + smoke-test-job: + script: smoke-test.sh + ``` + +The final configuration would be: + +```yaml +unit-test-job: + before_script: default-before-script.sh + script: unit-test.sh + retry: 0 + +smoke-test-job: + before_script: default-before-script.sh + script: smoke-test.sh + retry: 2 +``` + ## Use variables with `include` > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/284883) in GitLab 13.8. diff --git a/doc/ci/yaml/index.md b/doc/ci/yaml/index.md index 4530e2675c4..d9ea5498b63 100644 --- a/doc/ci/yaml/index.md +++ b/doc/ci/yaml/index.md @@ -415,7 +415,7 @@ and the pipeline is for either: - You can use the [`workflow:rules` templates](workflow.md#workflowrules-templates) to import a preconfigured `workflow: rules` entry. - [Common `if` clauses for `workflow:rules`](workflow.md#common-if-clauses-for-workflowrules). -- [Use `rules` to run pipelines for merge requests](../pipelines/merge_request_pipelines.md#use-rules-to-add-jobs). +- [Use `rules` to run merge request pipelines](../pipelines/merge_request_pipelines.md#use-rules-to-add-jobs). #### `workflow:rules:variables` @@ -939,7 +939,7 @@ rspec: Use `artifacts:untracked` to add all Git untracked files as artifacts (along with the paths defined in `artifacts:paths`). `artifacts:untracked` ignores configuration -in the repository's `.gitignore` file. +in the repository's `.gitignore`, so matching artifacts in `.gitignore` are included. **Keyword type**: Job keyword. You can use it only as part of a job or in the [`default` section](#default). @@ -974,7 +974,7 @@ failure. - `on_success` (default): Upload artifacts only when the job succeeds. - `on_failure`: Upload artifacts only when the job fails. -- `always`: Always upload artifacts. For example, when +- `always`: Always upload artifacts (except when jobs time out). For example, when [uploading artifacts](../unit_test_reports.md#viewing-junit-screenshots-on-gitlab) required to troubleshoot failing tests. @@ -1322,12 +1322,14 @@ Use `coverage` with a custom regular expression to configure how code coverage is extracted from the job output. The coverage is shown in the UI if at least one line in the job output matches the regular expression. -To extract the code coverage value in the matching line, GitLab uses this -regular expression: `\d+(\.\d+)?`. +To extract the code coverage value from the match, GitLab uses +this smaller regular expression: `\d+(\.\d+)?`. **Possible inputs**: -- A regular expression. Must start and end with `/`. +- A regular expression. Must start and end with `/`. Must match the coverage number. + May match surrounding text as well, so you don't need to use a regular expression character group + to capture the exact number. **Example of `coverage`**: @@ -1339,14 +1341,20 @@ job1: In this example: -1. GitLab checks the job log for a line that matches the regular expression. A line - like `Code coverage: 67.89` would match. -1. GitLab then checks the line to find a match to `\d+(\.\d+)?`. The sample matching - line above gives a code coverage of `67.89`. +1. GitLab checks the job log for a match with the regular expression. A line + like `Code coverage: 67.89% of lines covered` would match. +1. GitLab then checks the matched fragment to find a match to `\d+(\.\d+)?`. + The sample matching line above gives a code coverage of `67.89`. **Additional details**: -- If there is more than one matched line in the job output, the last line is used. +- Coverage regular expressions set in `gitlab-ci.yml` take precedence over coverage regular expression set in the + [GitLab UI](../pipelines/settings.md#add-test-coverage-results-to-a-merge-request-deprecated). +- If there is more than one matched line in the job output, the last line is used + (the first result of reverse search). +- If there are multiple matches in a single line, the last match is searched + for the coverage number. +- If there are multiple coverage numbers found in the matched fragment, the first number is used. - Leading zeros are removed. - Coverage output from [child pipelines](../pipelines/parent_child_pipelines.md) is not recorded or displayed. Check [the related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/280818) @@ -1467,7 +1475,7 @@ Use `environment` to define the [environment](../environments/index.md) that a j formats: - Plain text, including letters, digits, spaces, and these characters: `-`, `_`, `/`, `$`, `{`, `}`. -- CI/CD variables, including predefined, secure, or variables defined in the +- CI/CD variables, including predefined, project, group, instance, or variables defined in the `.gitlab-ci.yml` file. You can't use variables defined in a `script` section. **Example of `environment`**: @@ -1496,7 +1504,7 @@ Common environment names are `qa`, `staging`, and `production`, but you can use formats: - Plain text, including letters, digits, spaces, and these characters: `-`, `_`, `/`, `$`, `{`, `}`. -- CI/CD variables, including predefined, secure, or variables defined in the +- CI/CD variables, including predefined, project, group, instance, or variables defined in the `.gitlab-ci.yml` file. You can't use variables defined in a `script` section. **Example of `environment:name`**: @@ -1518,7 +1526,7 @@ Set a URL for an [environment](../environments/index.md). **Possible inputs**: A single URL, in one of these formats: - Plain text, like `https://prod.example.com`. -- CI/CD variables, including predefined, secure, or variables defined in the +- CI/CD variables, including predefined, project, group, instance, or variables defined in the `.gitlab-ci.yml` file. You can't use variables defined in a `script` section. **Example of `environment:url`**: @@ -2366,7 +2374,7 @@ pipeline based on branch names or pipeline types. | `chat` | For pipelines created by using a [GitLab ChatOps](../chatops/index.md) command. | | `external` | When you use CI services other than GitLab. | | `external_pull_requests` | When an external pull request on GitHub is created or updated (See [Pipelines for external pull requests](../ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests)). | - | `merge_requests` | For pipelines created when a merge request is created or updated. Enables [merge request pipelines](../pipelines/merge_request_pipelines.md), [merged results pipelines](../pipelines/pipelines_for_merged_results.md), and [merge trains](../pipelines/merge_trains.md). | + | `merge_requests` | For pipelines created when a merge request is created or updated. Enables [merge request pipelines](../pipelines/merge_request_pipelines.md), [merged results pipelines](../pipelines/merged_results_pipelines.md), and [merge trains](../pipelines/merge_trains.md). | | `pipelines` | For [multi-project pipelines](../pipelines/multi_project_pipelines.md) created by [using the API with `CI_JOB_TOKEN`](../pipelines/multi_project_pipelines.md#create-multi-project-pipelines-by-using-the-api), or the [`trigger`](#trigger) keyword. | | `pushes` | For pipelines triggered by a `git push` event, including for branches and tags. | | `schedules` | For [scheduled pipelines](../pipelines/schedules.md). | @@ -2465,7 +2473,7 @@ Use `changes` in pipelines with the following refs: - `branches` - `external_pull_requests` -- `merge_requests` (see additional details about [using `only:changes` with pipelines for merge requests](../jobs/job_control.md#use-onlychanges-with-pipelines-for-merge-requests)) +- `merge_requests` (see additional details about [using `only:changes` with merge request pipelines](../jobs/job_control.md#use-onlychanges-with-merge-request-pipelines)) **Keyword type**: Job keyword. You can use it only as part of a job. @@ -2507,7 +2515,7 @@ docker build: - [`only: changes` and `except: changes` examples](../jobs/job_control.md#onlychanges--exceptchanges-examples). - If you use `changes` with [only allow merge requests to be merged if the pipeline succeeds](../../user/project/merge_requests/merge_when_pipeline_succeeds.md#only-allow-merge-requests-to-be-merged-if-the-pipeline-succeeds), - you should [also use `only:merge_requests`](../jobs/job_control.md#use-onlychanges-with-pipelines-for-merge-requests). + you should [also use `only:merge_requests`](../jobs/job_control.md#use-onlychanges-with-merge-request-pipelines). - [Jobs or pipelines can run unexpectedly when using `only: changes`](../jobs/job_control.md#jobs-or-pipelines-run-unexpectedly-when-using-changes). #### `only:kubernetes` / `except:kubernetes` @@ -2654,6 +2662,7 @@ deploystacks: [vultr, processing] - [Run a one-dimensional matrix of parallel jobs](../jobs/job_control.md#run-a-one-dimensional-matrix-of-parallel-jobs). - [Run a matrix of triggered parallel jobs](../jobs/job_control.md#run-a-matrix-of-parallel-trigger-jobs). +- [Select different runner tags for each parallel matrix job](../jobs/job_control.md#select-different-runner-tags-for-each-parallel-matrix-job). ### `release` @@ -3079,7 +3088,7 @@ job: - [Common `if` expressions for `rules`](../jobs/job_control.md#common-if-clauses-for-rules). - [Avoid duplicate pipelines](../jobs/job_control.md#avoid-duplicate-pipelines). -- [Use `rules` to run pipelines for merge requests](../pipelines/merge_request_pipelines.md#use-rules-to-add-jobs). +- [Use `rules` to run merge request pipelines](../pipelines/merge_request_pipelines.md#use-rules-to-add-jobs). #### `rules:changes` @@ -3569,6 +3578,7 @@ In this example, only runners with *both* the `ruby` and `postgres` tags can run **Related topics**: - [Use tags to control which jobs a runner can run](../runners/configure_runners.md#use-tags-to-control-which-jobs-a-runner-can-run). +- [Select different runner tags for each parallel matrix job](../jobs/job_control.md#select-different-runner-tags-for-each-parallel-matrix-job). ### `timeout` @@ -3642,7 +3652,7 @@ trigger_job: - Jobs with `trigger` can only use a [limited set of keywords](../pipelines/multi_project_pipelines.md#define-multi-project-pipelines-in-your-gitlab-ciyml-file). For example, you can't run commands with [`script`](#script), [`before_script`](#before_script), - or [`after_script`](#after_script). + or [`after_script`](#after_script). Also, [`environment`](#environment) is not supported with `trigger`. - You [cannot use the API to start `when:manual` trigger jobs](https://gitlab.com/gitlab-org/gitlab/-/issues/284086). - In [GitLab 13.5 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/201938), you can use [`when:manual`](#when) in the same job as `trigger`. In GitLab 13.4 and diff --git a/doc/ci/yaml/script.md b/doc/ci/yaml/script.md index ca1e79c2395..4bffcbca1cc 100644 --- a/doc/ci/yaml/script.md +++ b/doc/ci/yaml/script.md @@ -4,7 +4,7 @@ group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab CI/CD script syntax **(FREE)** +# Format scripts and job logs **(FREE)** You can use special syntax in [`script`](index.md#script) sections to: diff --git a/doc/ci/yaml/workflow.md b/doc/ci/yaml/workflow.md index 332214638d8..b46d504edfa 100644 --- a/doc/ci/yaml/workflow.md +++ b/doc/ci/yaml/workflow.md @@ -126,7 +126,7 @@ makes your pipelines run for branches and tags. Branch pipeline status is displayed in merge requests that use the branch as a source. However, this pipeline type does not support any features offered by [merge request pipelines](../pipelines/merge_request_pipelines.md), like -[pipelines for merged results](../pipelines/pipelines_for_merged_results.md) +[merged results pipelines](../pipelines/merged_results_pipelines.md) or [merge trains](../pipelines/merge_trains.md). This template intentionally avoids those features. @@ -140,7 +140,7 @@ include: The [`MergeRequest-Pipelines` template](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates/Workflows/MergeRequest-Pipelines.gitlab-ci.yml) makes your pipelines run for the default branch, tags, and all types of merge request pipelines. Use this template if you use any of the -the [pipelines for merge requests features](../pipelines/merge_request_pipelines.md). +the [merge request pipelines features](../pipelines/merge_request_pipelines.md). To [include](index.md#include) it: diff --git a/doc/development/agent/gitops.md b/doc/development/agent/gitops.md index 3d59f5bd845..7c741408ae6 100644 --- a/doc/development/agent/gitops.md +++ b/doc/development/agent/gitops.md @@ -1,9 +1,9 @@ --- redirect_to: 'https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/gitops.md' -remove_date: '2022-06-24' +remove_date: '2022-02-01' --- This file was moved to [another location](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/gitops.md). -<!-- This redirect file can be deleted after <2022-06-24>. --> +<!-- This redirect file can be deleted after <2022-02-01>. --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/agent/identity.md b/doc/development/agent/identity.md index 67084a6d995..6caf108a32a 100644 --- a/doc/development/agent/identity.md +++ b/doc/development/agent/identity.md @@ -1,9 +1,9 @@ --- redirect_to: 'https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/identity_and_auth.md' -remove_date: '2022-06-24' +remove_date: '2022-02-01' --- This file was moved to [another location](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/identity_and_auth.md). -<!-- This redirect file can be deleted after <2022-06-24>. --> +<!-- This redirect file can be deleted after <2022-02-01>. --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/agent/index.md b/doc/development/agent/index.md index 2cb05e2dd8f..474f8a02933 100644 --- a/doc/development/agent/index.md +++ b/doc/development/agent/index.md @@ -1,9 +1,9 @@ --- redirect_to: 'https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/architecture.md' -remove_date: '2022-06-24' +remove_date: '2022-02-01' --- This file was moved to [another location](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/architecture.md). -<!-- This redirect file can be deleted after <2022-06-24>. --> +<!-- This redirect file can be deleted after <2022-02-01>. --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/agent/local.md b/doc/development/agent/local.md index 1fff5607de4..a4b29bea838 100644 --- a/doc/development/agent/local.md +++ b/doc/development/agent/local.md @@ -1,9 +1,9 @@ --- redirect_to: 'https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/local.md' -remove_date: '2022-06-24' +remove_date: '2022-02-01' --- This file was moved to [another location](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/local.md). -<!-- This redirect file can be deleted after <2022-06-24>. --> +<!-- This redirect file can be deleted after <2022-02-01>. --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/agent/repository_overview.md b/doc/development/agent/repository_overview.md index 43ea99889c5..8ea9dceb32a 100644 --- a/doc/development/agent/repository_overview.md +++ b/doc/development/agent/repository_overview.md @@ -1,9 +1,9 @@ --- redirect_to: 'https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/repository_overview.md' -remove_date: '2022-06-24' +remove_date: '2022-02-01' --- This file was moved to [another location](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/repository_overview.md). -<!-- This redirect file can be deleted after <2022-06-24>. --> +<!-- This redirect file can be deleted after <2022-02-01>. --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/agent/routing.md b/doc/development/agent/routing.md index 7792d6d56a4..364267a45fe 100644 --- a/doc/development/agent/routing.md +++ b/doc/development/agent/routing.md @@ -1,9 +1,9 @@ --- redirect_to: 'https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/kas_request_routing.md' -remove_date: '2022-06-24' +remove_date: '2022-02-01' --- This file was moved to [another location](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/kas_request_routing.md). -<!-- This redirect file can be deleted after <2022-06-24>. --> +<!-- This redirect file can be deleted after <2022-02-01>. --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/agent/user_stories.md b/doc/development/agent/user_stories.md index ce38064b31b..2ed4bbdc9f6 100644 --- a/doc/development/agent/user_stories.md +++ b/doc/development/agent/user_stories.md @@ -1,9 +1,9 @@ --- redirect_to: 'https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/user_stories.md' -remove_date: '2022-06-24' +remove_date: '2022-02-01' --- This file was moved to [another location](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/user_stories.md). -<!-- This redirect file can be deleted after <2022-06-24>. --> +<!-- This redirect file can be deleted after <2022-02-01>. --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/application_slis/index.md b/doc/development/application_slis/index.md index 87c0bcfede5..adb656761c5 100644 --- a/doc/development/application_slis/index.md +++ b/doc/development/application_slis/index.md @@ -57,31 +57,10 @@ Gitlab::Metrics::Sli.initialize_sli(:received_email, [ ]) ``` -Metrics must be initialized before they get -scraped for the first time. This could be done at the start time of the -process that will emit them, in which case we need to pay attention -not to increase application's boot time too much. This is preferable -if possible. - -Alternatively, if initializing would take too long, this can be done -during the first scrape. We need to make sure we don't do it for every -scrape. This can be done as follows: - -```ruby -def initialize_request_slis_if_needed! - return if Gitlab::Metrics::Sli.initialized?(:rails_request_apdex) - Gitlab::Metrics::Sli.initialize_sli(:rails_request_apdex, possible_request_labels) -end -``` - -Also pay attention to do it for the different metrics -endpoints we have. Currently the -[`WebExporter`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/metrics/exporter/web_exporter.rb) -and the -[`HealthController`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/controllers/health_controller.rb) -for Rails and -[`SidekiqExporter`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/metrics/exporter/sidekiq_exporter.rb) -for Sidekiq. +Metrics must be initialized before they get scraped for the first time. +This currently happens during the `on_master_start` [life-cycle event](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/cluster/lifecycle_events.rb). +Since this delays application readiness until metrics initialization returns, make sure the overhead +this adds is understood and acceptable. ## Tracking operations for an SLI diff --git a/doc/development/architecture.md b/doc/development/architecture.md index 36d3655ba93..e8d04d68565 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -339,7 +339,7 @@ Component statuses are linked to configuration documentation for each component. ### Component list -| Component | Description | [Omnibus GitLab](https://docs.gitlab.com/omnibus/) | [GitLab Environment Toolkit (GET)](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit) | [GitLab chart](https://docs.gitlab.com/charts/) | [minikube Minimal](https://docs.gitlab.com/charts/development/minikube/#deploying-gitlab-with-minimal-settings) | [GitLab.com](https://gitlab.com) | [Source](../install/installation.md) | [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit) | [CE/EE](https://about.gitlab.com/install/ce-or-ee/) | +| Component | Description | [Omnibus GitLab](https://docs.gitlab.com/omnibus/) | [GitLab Environment Toolkit (GET)](https://gitlab.com/gitlab-org/gitlab-environment-toolkit) | [GitLab chart](https://docs.gitlab.com/charts/) | [minikube Minimal](https://docs.gitlab.com/charts/development/minikube/#deploying-gitlab-with-minimal-settings) | [GitLab.com](https://gitlab.com) | [Source](../install/installation.md) | [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit) | [CE/EE](https://about.gitlab.com/install/ce-or-ee/) | |-------------------------------------------------------|----------------------------------------------------------------------|:--------------:|:--------------:|:------------:|:----------------:|:----------:|:------:|:---:|:-------:| | [Certificate Management](#certificate-management) | TLS Settings, Let's Encrypt | ✅ | ✅ | ✅ | ⚙ | ✅ | ⚙ | ⚙ | CE & EE | | [Consul](#consul) | Database node discovery, failover | ⚙ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | EE Only | @@ -528,7 +528,7 @@ You can use it either for personal or business websites, such as portfolios, doc #### GitLab Runner -- [Project page](https://gitlab.com/gitlab-org/gitlab-runner/blob/master/README.md) +- [Project page](https://gitlab.com/gitlab-org/gitlab-runner/blob/main/README.md) - Configuration: - [Omnibus](https://docs.gitlab.com/runner/) - [Charts](https://docs.gitlab.com/runner/install/kubernetes.html) diff --git a/doc/development/avoiding_downtime_in_migrations.md b/doc/development/avoiding_downtime_in_migrations.md index 1d36bbf1212..1de96df327c 100644 --- a/doc/development/avoiding_downtime_in_migrations.md +++ b/doc/development/avoiding_downtime_in_migrations.md @@ -399,7 +399,7 @@ migrations: 1. Change the index pattern to `pubsub-sidekiq-inf-gprd*`. 1. Add filter for `json.queue: cronjob:database_batched_background_migration`. -#### PostgerSQL slow queries log +#### PostgreSQL slow queries log Slow queries log keeps track of low queries that took above 1 second to execute. To see them for batched background migration: @@ -408,7 +408,7 @@ for batched background migration: 1. Change the index pattern to `pubsub-postgres-inf-gprd*`. 1. Add filter for `json.endpoint_id.keyword: Database::BatchedBackgroundMigrationWorker`. 1. Optional. To see only updates, add a filter for `json.command_tag.keyword: UPDATE`. -1. Optional. To see only failed statements, add a filter for `json.error_severiry.keyword: ERROR`. +1. Optional. To see only failed statements, add a filter for `json.error_severity.keyword: ERROR`. 1. Optional. Add a filter by table name. #### Grafana dashboards diff --git a/doc/development/cached_queries.md b/doc/development/cached_queries.md index 4f82d721164..492c8d13600 100644 --- a/doc/development/cached_queries.md +++ b/doc/development/cached_queries.md @@ -163,5 +163,5 @@ factors help improve the overall execution time: ## For more information - [Metrics that would help us detect the potential N+1 Cached SQL calls](https://gitlab.com/gitlab-org/gitlab/-/issues/259007) -- [Merge Request performance guidelines for cached queries](merge_request_performance_guidelines.md#cached-queries) +- [Merge request performance guidelines for cached queries](merge_request_performance_guidelines.md#cached-queries) - [Improvements for biggest offenders](https://gitlab.com/groups/gitlab-org/-/epics/4508) diff --git a/doc/development/cascading_settings.md b/doc/development/cascading_settings.md index f9c96e414d0..76ab2c6e693 100644 --- a/doc/development/cascading_settings.md +++ b/doc/development/cascading_settings.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/changelog.md b/doc/development/changelog.md index 7c4a600e1fa..b51db69c2f7 100644 --- a/doc/development/changelog.md +++ b/doc/development/changelog.md @@ -101,7 +101,9 @@ EE: true - _Any_ contribution from a community member, no matter how small, **may** have a changelog entry regardless of these guidelines if the contributor wants one. - Any [GLEX experiment](experiment_guide/gitlab_experiment.md) changes **should not** have a changelog entry. -- [Modifying](feature_flags/#changelog) a feature flag (flag removal, default-on setting). + +For more information, see +[how to handle changelog entries with feature flags](feature_flags/index.md#changelog). ## Writing good changelog entries diff --git a/doc/development/cicd/index.md b/doc/development/cicd/index.md index 17a70393c96..2779b457fb9 100644 --- a/doc/development/cicd/index.md +++ b/doc/development/cicd/index.md @@ -27,7 +27,7 @@ On the left side we have the events that can trigger a pipeline based on various - A `git push` is the most common event that triggers a pipeline. - The [Web API](../../api/pipelines.md#create-a-new-pipeline). - A user clicking the "Run pipeline" button in the UI. -- When a [merge request is created or updated](../../ci/pipelines/merge_request_pipelines.md#pipelines-for-merge-requests). +- When a [merge request is created or updated](../../ci/pipelines/merge_request_pipelines.md). - When an MR is added to a [Merge Train](../../ci/pipelines/merge_trains.md#merge-trains). - A [scheduled pipeline](../../ci/pipelines/schedules.md#pipeline-schedules). - When project is [subscribed to an upstream project](../../ci/pipelines/multi_project_pipelines.md#trigger-a-pipeline-when-an-upstream-project-is-rebuilt). diff --git a/doc/development/cicd/templates.md b/doc/development/cicd/templates.md index b1252b86cc0..d7edad842b8 100644 --- a/doc/development/cicd/templates.md +++ b/doc/development/cicd/templates.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: index, concepts, howto --- -# Development guide for GitLab CI/CD templates +# Development guide for GitLab CI/CD templates **(FREE)** This document explains how to develop [GitLab CI/CD templates](../../ci/examples/index.md). @@ -18,6 +18,7 @@ Before submitting a merge request with a new or updated CI/CD template, you must - Name the template following the `*.gitlab-ci.yml` format. - Use valid [`.gitlab-ci.yml` syntax](../../ci/yaml/index.md). Verify it's valid with the [CI/CD lint tool](../../ci/lint.md). +- [Add template metrics](#add-metrics). - Include [a changelog](../changelog.md) if the merge request introduces a user-facing change. - Follow the [template review process](#contribute-cicd-template-merge-requests). - (Optional but highly recommended) Test the template in an example GitLab project @@ -382,6 +383,77 @@ you must: This information is important for users when [a stable template](#stable-version) is updated in a major version GitLab release. +### Add metrics + +Every CI/CD template must also have metrics defined to track their use. + +To add a metric definition for a new template: + +1. Install and start the [GitLab GDK](https://gitlab.com/gitlab-org/gitlab-development-kit#installation). +1. In the `gitlab` directory in your GDK, check out the branch that contains the new template. +1. [Add the template inclusion event](../service_ping/implement.md#add-new-events) + with this Rake task: + + ```shell + bin/rake gitlab:usage_data:generate_ci_template_events + ``` + + The task adds a section like the following to [`lib/gitlab/usage_data_counters/known_events/ci_templates.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events/ci_templates.yml): + + ```yaml + - name: p_ci_templates_my_template_name + category: ci_templates + redis_slot: ci_templates + aggregation: weekly + ``` + +1. Copy the value of `name` from the new YAML section, and add it to the weekly + and monthly CI/CD template total count metrics: + - [`config/metrics/counts_7d/20210216184557_ci_templates_total_unique_counts_weekly.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216184557_ci_templates_total_unique_counts_weekly.yml) + - [`config/metrics/counts_28d/20210216184559_ci_templates_total_unique_counts_monthly.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184559_ci_templates_total_unique_counts_monthly.yml) + +1. Use the same `name` as above as the last argument in the following command to + [add new metric definitions](../service_ping/metrics_dictionary.md#metrics-added-dynamic-to-service-ping-payload): + + ```shell + bundle exec rails generate gitlab:usage_metric_definition:redis_hll ci_templates <template_metric_event_name> + ``` + + The output should look like: + + ```shell + $ bundle exec rails generate gitlab:usage_metric_definition:redis_hll ci_templates p_ci_templates_my_template_name + create config/metrics/counts_7d/20220120073740_p_ci_templates_my_template_name_weekly.yml + create config/metrics/counts_28d/20220120073746_p_ci_templates_my_template_name_monthly.yml + ``` + +1. Edit both newly generated files as follows: + + - `name:` and `performance_indicator_type:`: Delete (not needed). + - `introduced_by_url:`: The URL of the MR adding the template. + - `data_source:`: Set to `redis_hll`. + - All other fields that have no values: Set to empty strings (`''`). + - Add the following to the end of each file: + + ```yaml + options: + events: + - p_ci_templates_my_template_name + ``` + +1. Commit and push the changes. + +For example, these are the metrics configuration files for the +[5 Minute Production App template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/5-Minute-Production-App.gitlab-ci.yml): + +- The template inclusion event: [`lib/gitlab/usage_data_counters/known_events/ci_templates.yml#L438-L441`](https://gitlab.com/gitlab-org/gitlab/-/blob/dcddbf83c29d1ad0839d55362c1b43888304f453/lib/gitlab/usage_data_counters/known_events/ci_templates.yml#L438-L441) +- The weekly and monthly metrics definitions: + - [`config/metrics/counts_7d/20210901223501_p_ci_templates_5_minute_production_app_weekly.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/1a6eceff3914f240864b2ca15ae2dc076ea67bf6/config/metrics/counts_7d/20210216184515_p_ci_templates_5_min_production_app_weekly.yml) + - [`config/metrics/counts_28d/20210901223505_p_ci_templates_5_minute_production_app_monthly.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184517_p_ci_templates_5_min_production_app_monthly.yml) +- The metrics count totals: + - [`config/metrics/counts_7d/20210216184557_ci_templates_total_unique_counts_weekly.yml#L19`](https://gitlab.com/gitlab-org/gitlab/-/blob/4e01ef2b094763943348655ef77008aba7a052ae/config/metrics/counts_7d/20210216184557_ci_templates_total_unique_counts_weekly.yml#L19) + - [`config/metrics/counts_28d/20210216184559_ci_templates_total_unique_counts_monthly.yml#L19`](https://gitlab.com/gitlab-org/gitlab/-/blob/4e01ef2b094763943348655ef77008aba7a052ae/config/metrics/counts_28d/20210216184559_ci_templates_total_unique_counts_monthly.yml#L19) + ## Security A template could contain malicious code. For example, a template that contains the `export` shell command in a job diff --git a/doc/development/code_review.md b/doc/development/code_review.md index 1ed510c6ad7..3664ca7642a 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -285,7 +285,7 @@ Maintainers should check before merging if the merge request is approved by the required approvers. If still awaiting further approvals from others, remove yourself as a reviewer then `@` mention the author and explain why in a comment. Stay as reviewer if you're merging the code. Maintainers must check before merging if the merge request is introducing new -vulnerabilities, by inspecting the list in the Merge Request +vulnerabilities, by inspecting the list in the merge request [Security Widget](../user/application_security/index.md). When in doubt, a [Security Engineer](https://about.gitlab.com/company/team/) can be involved. The list of detected vulnerabilities must be either empty or containing: @@ -296,8 +296,8 @@ vulnerabilities must be either empty or containing: Maintainers should **never** dismiss vulnerabilities to "empty" the list, without duly verifying them. -Note that certain Merge Requests may target a stable branch. These are rare -events. These types of Merge Requests cannot be merged by the Maintainer. +Note that certain merge requests may target a stable branch. These are rare +events. These types of merge requests cannot be merged by the Maintainer. Instead, these should be sent to the [Release Manager](https://about.gitlab.com/community/release-managers/). After merging, a maintainer should stay as the reviewer listed on the merge request. @@ -479,7 +479,7 @@ WARNING: [very specific cases](https://about.gitlab.com/handbook/engineering/workflow/#criteria-for-merging-during-broken-master). For other cases, follow these [handbook instructions](https://about.gitlab.com/handbook/engineering/workflow/#merging-during-broken-master). - If the latest pipeline was created before the merge request was approved, start a new pipeline to ensure that full RSpec suite has been run. You may skip this step only if the merge request does not contain any backend change. - - If the **latest [Pipeline for Merged Results](../ci/pipelines/pipelines_for_merged_results.md)** finished less than 2 hours ago, you + - If the **latest [merged results pipeline](../ci/pipelines/merged_results_pipelines.md)** finished less than 2 hours ago, you may merge without starting a new pipeline as the merge request is close enough to `main`. - When you set the MR to "Merge When Pipeline Succeeds", you should take over @@ -489,7 +489,7 @@ WARNING: the squashed commit's default commit message is taken from the merge request title. You're encouraged to [select a commit with a more informative commit message](../user/project/merge_requests/squash_and_merge.md) before merging. -Thanks to **Pipeline for Merged Results**, authors no longer have to rebase their +Thanks to **merged results pipelines**, authors no longer have to rebase their branch as frequently anymore (only when there are conflicts) because the Merge Results Pipeline already incorporate the latest changes from `main`. This results in faster review/merge cycles because maintainers don't have to ask @@ -501,7 +501,7 @@ Merge Results against the latest `main` at the time of the pipeline creation. WARNING: **Review all changes thoroughly for malicious code before starting a -[Pipeline for Merged Results](../ci/pipelines/merge_request_pipelines.md#run-pipelines-in-the-parent-project).** +[merged results pipeline](../ci/pipelines/merge_request_pipelines.md#run-pipelines-in-the-parent-project).** When reviewing merge requests added by wider community contributors: @@ -510,6 +510,9 @@ When reviewing merge requests added by wider community contributors: fetching of malicious packages. - Review links and images, especially in documentation MRs. - When in doubt, ask someone from `@gitlab-com/gl-security/appsec` to review the merge request **before manually starting any merge request pipeline**. +- Only set the milestone when the merge request is likely to be included in + the current milestone. This is to avoid confusion around when it'll be + merged and avoid moving milestone too often when it's not yet ready. If the MR source branch is more than 1,000 commits behind the target branch: @@ -599,7 +602,7 @@ Enterprise Edition instance. This has some implications: - [Background migrations](background_migrations.md) run in Sidekiq, and should only be done for migrations that would take an extreme amount of time at GitLab.com scale. -1. **Sidekiq workers** [cannot change in a backwards-incompatible way](sidekiq_style_guide.md#sidekiq-compatibility-across-updates): +1. **Sidekiq workers** [cannot change in a backwards-incompatible way](sidekiq/compatibility_across_updates.md): 1. Sidekiq queues are not drained before a deploy happens, so there are workers in the queue from the previous version of GitLab. 1. If you need to change a method signature, try to do so across two releases, @@ -670,7 +673,7 @@ Properties of customer critical merge requests: - It is required that the reviewer(s) and maintainer(s) involved with a customer critical merge request are engaged as soon as this decision is made. - It is required to prioritize work for those involved on a customer critical merge request so that they have the time available necessary to focus on it. - It is required to adhere to GitLab [values](https://about.gitlab.com/handbook/values/) and processes when working on customer critical merge requests, taking particular note of family and friends first/work second, definition of done, iteration, and release when it's ready. -- Customer critical merge requests are required to not reduce security, introduce data-loss risk, reduce availability, nor break existing functionality per the process for [prioritizing technical decisions](https://about.gitlab.com/handbook/engineering/#prioritizing-technical-decisions.md). +- Customer critical merge requests are required to not reduce security, introduce data-loss risk, reduce availability, nor break existing functionality per the process for [prioritizing technical decisions](https://about.gitlab.com/handbook/engineering/principles/#prioritizing-technical-decisions). - On customer critical requests, it is _recommended_ that those involved _consider_ coordinating synchronously (Zoom, Slack) in addition to asynchronously (merge requests comments) if they believe this may reduce the elapsed time to merge even though this _may_ sacrifice [efficiency](https://about.gitlab.com/company/culture/all-remote/asynchronous/#evaluating-efficiency.md). - After a customer critical merge request is merged, a retrospective must be completed with the intention of reducing the frequency of future customer critical merge requests. diff --git a/doc/development/contributing/design.md b/doc/development/contributing/design.md index e85f5dd8349..efa8d4b0c41 100644 --- a/doc/development/contributing/design.md +++ b/doc/development/contributing/design.md @@ -60,7 +60,7 @@ Check visual design properties using your browser's _elements inspector_ ([Chrom guidelines. - _Optionally_ consider [dark mode](../../user/profile/preferences.md#dark-mode). [^1] - [^1]: You're not required to design for [dark mode](../../user/profile/preferences.md#dark-mode) while the feature is in [alpha](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha). The [UX Foundations team](https://about.gitlab.com/direction/ecosystem/foundations/) plans to improve the dark mode in the future. Until we integrate [Pajamas](https://design.gitlab.com/) components into the product and the underlying design strategy is in place to support dark mode, we cannot guarantee that we won't introduce bugs and debt to this mode. At your discretion, evaluate the need to create dark mode patches. + [^1]: You're not required to design for [dark mode](../../user/profile/preferences.md#dark-mode) while the feature is in [alpha](../../policy/alpha-beta-support.md#alpha-features). The [UX Foundations team](https://about.gitlab.com/direction/ecosystem/foundations/) plans to improve the dark mode in the future. Until we integrate [Pajamas](https://design.gitlab.com/) components into the product and the underlying design strategy is in place to support dark mode, we cannot guarantee that we won't introduce bugs and debt to this mode. At your discretion, evaluate the need to create dark mode patches. ### States diff --git a/doc/development/contributing/index.md b/doc/development/contributing/index.md index ec05f75c709..ea54f36a7e5 100644 --- a/doc/development/contributing/index.md +++ b/doc/development/contributing/index.md @@ -91,7 +91,7 @@ Sign up for the mailing list, answer GitLab questions on StackOverflow or respon If you would like to contribute to GitLab: - Issues with the - [`~Accepting merge requests` label](issue_workflow.md#label-for-community-contributors) + [`~Seeking community contributions` label](issue_workflow.md#label-for-community-contributors) are a great place to start. - Optimizing our tests is another great opportunity to contribute. You can use [RSpec profiling statistics](https://gitlab-org.gitlab.io/rspec_profiling_stats/) to identify diff --git a/doc/development/contributing/issue_workflow.md b/doc/development/contributing/issue_workflow.md index 6a7bbb2e302..ad8403d242c 100644 --- a/doc/development/contributing/issue_workflow.md +++ b/doc/development/contributing/issue_workflow.md @@ -57,6 +57,10 @@ Most issues will have labels for at least one of the following: - Priority: `~"priority::1"`, `~"priority::2"`, `~"priority::3"`, `~"priority::4"` - Severity: ~`"severity::1"`, `~"severity::2"`, `~"severity::3"`, `~"severity::4"` +Please add `~"breaking change"` label if the issue can be considered as a [breaking change](index.md#breaking-changes). + +Please add `~security` label if the issue is related to application security. + All labels, their meaning and priority are defined on the [labels page](https://gitlab.com/gitlab-org/gitlab/-/labels). @@ -282,12 +286,12 @@ Please refer to the issue triage [severity label](https://about.gitlab.com/handb ### Label for community contributors -Issues that are beneficial to our users, 'nice to haves', that we currently do -not have the capacity for or want to give the priority to, are labeled as -~"Accepting merge requests", so the community can make a contribution. +There are many issues that have a clear solution with uncontroversial benefit to GitLab users. +However, GitLab might not have the capacity for all these proposals in the current roadmap. +These issues are labeled ~"Seeking community contributions" because we welcome merge requests to resolve them. Community contributors can submit merge requests for any issue they want, but -the ~"Accepting merge requests" label has a special meaning. It points to +the ~"Seeking community contributions" label has a special meaning. It points to changes that: 1. We already agreed on, @@ -295,20 +299,24 @@ changes that: 1. Are likely to get accepted by a maintainer. We want to avoid a situation when a contributor picks an -~"Accepting merge requests" issue and then their merge request gets closed, +~"Seeking community contributions" issue and then their merge request gets closed, because we realize that it does not fit our vision, or we want to solve it in a different way. -We automatically add the ~"Accepting merge requests" label to issues -that match the [triage policy](https://about.gitlab.com/handbook/engineering/quality/triage-operations/#accepting-merge-requests). +We manually add the ~"Seeking community contributions" label to issues +that fit the criteria described above. +We do not automatically add this label, because it requires human evaluation. We recommend people that have never contributed to any open source project to -look for issues labeled `~"Accepting merge requests"` with a [weight of 1](https://gitlab.com/groups/gitlab-org/-/issues?state=opened&label_name[]=Accepting+merge+requests&assignee_id=None&sort=weight&weight=1) or the `~"good for new contributors"` [label](https://gitlab.com/gitlab-org/gitlab/-/issues?scope=all&state=opened&label_name[]=good%20for%20new%20contributors&assignee_id=None) attached to it. +look for issues labeled `~"Seeking community contributions"` with a +[weight of 1](https://gitlab.com/groups/gitlab-org/-/issues?sort=created_date&state=opened&label_name[]=Seeking+community+contributions&assignee_id=None&weight=1) or the `~"good for new contributors"` +[label](https://gitlab.com/gitlab-org/gitlab/-/issues?scope=all&state=opened&label_name[]=good%20for%20new%20contributors&assignee_id=None) +attached to it. More experienced contributors are very welcome to tackle -[any of them](https://gitlab.com/groups/gitlab-org/-/issues?state=opened&label_name[]=Accepting+merge+requests&assignee_id=None). +[any of them](https://gitlab.com/groups/gitlab-org/-/issues?sort=created_date&state=opened&label_name[]=Seeking+community+contributions&assignee_id=None). For more complex features that have a weight of 2 or more and clear scope, we recommend looking at issues -with the [label `~"Community Challenge"`](https://gitlab.com/gitlab-org/gitlab/-/issues?scope=all&state=opened&label_name[]=Accepting%20merge%20requests&label_name[]=Community%20challenge). +with the [label `~"Community Challenge"`](https://gitlab.com/gitlab-org/gitlab/-/issues?sort=created_date&state=opened&label_name[]=Seeking+community+contributions&label_name[]=Community+challenge). If your MR for the `~"Community Challenge"` issue gets merged, you will also have a chance to win a custom GitLab merchandise. @@ -319,7 +327,7 @@ members to further discuss scope, design, and technical considerations. This wil ensure that your contribution is aligned with the GitLab product and minimize any rework and delay in getting it merged into main. -GitLab team members who apply the ~"Accepting merge requests" label to an issue +GitLab team members who apply the ~"Seeking community contributions" label to an issue should update the issue description with a responsible product manager, inviting any potential community contributor to @-mention per above. diff --git a/doc/development/contributing/merge_request_workflow.md b/doc/development/contributing/merge_request_workflow.md index cc6997e1a20..02148f2a717 100644 --- a/doc/development/contributing/merge_request_workflow.md +++ b/doc/development/contributing/merge_request_workflow.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w We welcome merge requests from everyone, with fixes and improvements to GitLab code, tests, and documentation. The issues that are specifically suitable -for community contributions are listed with the [`Accepting merge requests`](issue_workflow.md#label-for-community-contributors) +for community contributions have the [`Seeking community contributions`](issue_workflow.md#label-for-community-contributors) label, but you are free to contribute to any issue you want. If an issue is marked for the current milestone at any time, even @@ -18,7 +18,7 @@ in order to ensure the work is finished before the release date. If you want to add a new feature that is not labeled, it is best to first create an issue (if there isn't one already) and leave a comment asking for it -to be marked as `Accepting merge requests`. See the [feature proposals](issue_workflow.md#feature-proposals) +to be labeled as `Seeking community contributions`. See the [feature proposals](issue_workflow.md#feature-proposals) section. Merge requests should be submitted to the appropriate project at GitLab.com, for example @@ -71,6 +71,9 @@ request is as follows: 1. The MR must include *Before* and *After* screenshots if UI changes are made. 1. Include any steps or setup required to ensure reviewers can view the changes you've made (for example, include any information about feature flags). 1. If you're allowed to, set a relevant milestone and [labels](issue_workflow.md). + MR labels should generally match the corresponding issue (if there is one). + The group label should reflect the group that executed or coached the work, + not necessarily the group that owns the feature. 1. UI changes should use available components from the GitLab Design System, [Pajamas](https://design.gitlab.com/). 1. If the MR changes CSS classes, please include the list of affected pages, which diff --git a/doc/development/contributing/style_guides.md b/doc/development/contributing/style_guides.md index fdb6e99fdcd..da926005466 100644 --- a/doc/development/contributing/style_guides.md +++ b/doc/development/contributing/style_guides.md @@ -51,20 +51,10 @@ This should return a fully qualified path command with no other output. ### Lefthook configuration -The current Lefthook configuration can be found in [`lefthook.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lefthook.yml). +Lefthook is configured with a combination of: -Before you push your changes, Lefthook automatically runs the following checks: - -- Danger: Runs a subset of checks that `danger-review` runs on your merge requests. -- ES lint: Run `yarn run lint:eslint` checks (with the [`.eslintrc.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.eslintrc.yml) configuration) on the modified `*.{js,vue}` files. Tags: `frontend`, `style`. -- HAML lint: Run `bundle exec haml-lint` checks (with the [`.haml-lint.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.haml-lint.yml) configuration) on the modified `*.html.haml` files. Tags: `view`, `haml`, `style`. -- Markdown lint: Run `yarn markdownlint` checks on the modified `*.md` files. Tags: `documentation`, `style`. -- SCSS lint: Run `yarn lint:stylelint` checks (with the [`.stylelintrc`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.stylelintrc) configuration) on the modified `*.scss{,.css}` files. Tags: `stylesheet`, `css`, `style`. -- RuboCop: Run `bundle exec rubocop` checks (with the [`.rubocop.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.rubocop.yml) configuration) on the modified `*.rb` files. Tags: `backend`, `style`. -- Vale: Run `vale` checks (with the [`.vale.ini`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.vale.ini) configuration) on the modified `*.md` files. Tags: `documentation`, `style`. -- Documentation metadata: Run checks for the absence of [documentation metadata](../documentation/index.md#metadata). - -In addition to the default configuration, you can define a [local configuration](https://github.com/Arkweid/lefthook/blob/master/docs/full_guide.md#local-config). +- Project configuration in [`lefthook.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lefthook.yml). +- Any [local configuration](https://github.com/Arkweid/lefthook/blob/master/docs/full_guide.md#local-config). ### Disable Lefthook temporarily diff --git a/doc/development/dangerbot.md b/doc/development/dangerbot.md index 374e4e5de68..8da1f5700e5 100644 --- a/doc/development/dangerbot.md +++ b/doc/development/dangerbot.md @@ -66,7 +66,7 @@ continue to apply. However, there are a few things that deserve special emphasis Danger is a powerful tool and flexible tool, but not always the most appropriate way to solve a given problem or workflow. -First, be aware of the GitLab [commitment to dogfooding](https://about.gitlab.com/handbook/engineering/#dogfooding). +First, be aware of the GitLab [commitment to dogfooding](https://about.gitlab.com/handbook/engineering/principles/#dogfooding). The code we write for Danger is GitLab-specific, and it **may not** be most appropriate place to implement functionality that addresses a need we encounter. Our users, customers, and even our own satellite projects, such as [Gitaly](https://gitlab.com/gitlab-org/gitaly), diff --git a/doc/development/database/dbcheck-migrations-job.md b/doc/development/database/dbcheck-migrations-job.md new file mode 100644 index 00000000000..af72e28a875 --- /dev/null +++ b/doc/development/database/dbcheck-migrations-job.md @@ -0,0 +1,67 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# db:check-migrations job + +This job runs on the test stage of a merge request pipeline. It checks: + +1. A schema dump comparison between the author's working branch and the target branch, + after executing a rollback of your new migrations. This check validates that the + schema properly resets to what it was before executing this new migration. +1. A schema dump comparison between the author's working branch and the `db/structure.sql` + file that the author committed. This check validates that it contains all the expected changes + in the migration. +1. A Git diff between the `db/schema_migrations` that the author committed and the + one that the script generated after running migrations. This check validates that everything + was properly committed. + +## Troubleshooting + +### False positives + +This job is allowed to fail, because it can throw some false positives. + +For example, when we drop a column and then roll back, this column is always +re-added at the end of the list of columns. If the column was previously in +the middle of the list, the rollback can't return the schema back exactly to +its previous state. The job fails, but it's an acceptable situation. + +For a real-life example, refer to +[this failed job](https://gitlab.com/gitlab-org/gitlab/-/jobs/2006544972#L138). +Here, the author dropped the `position` column. + +### Schema dump comparison fails after rollback + +This failure often happens if your working branch is behind the target branch. +A real scenario: + +```mermaid +graph LR + Main((main<br>commit A)) ===> |remove constraint<br>fk_rails_dbebdaa8fe| MainB((main<br>commit B)) + Main((main<br>commit A)) --> |checkout<br>dev| DevA((dev<br>commit A)):::dev + DevA((dev<br>commit A)) --> |add column<br>dependency_proxy_size| DevC((dev<br>commit C)):::dev + DevC -.-> |CI pipeline<br>executes| JOB-FAILED((JOB FAILED!)):::error + + classDef main fill:#f4f0ff,stroke:#7b58cf + classDef dev fill:#e9f3fc,stroke:#1f75cb + classDef error fill:#f15146,stroke:#d4121a +``` + +1. You check out the `dev` working branch from the `main` target branch. At this point, + each branch has their `HEAD` at commit A. +1. Someone works on the `main` branch and drops the `fk_rails_dbebdaa8fe` constraint, + thus creating commit B on `main`. +1. You add column `dependency_proxy_size` to your `dev` branch. +1. The `db:check-migrations` job fails for your `dev` branch's CI/CD pipeline, because + the `structure.sql` file did not rollback to its expected state. + +This happened because branch `dev` contained commits A and C, not B. Its database schema +did not know about the removal of the `fk_rails_dbebdaa8fe` constraint. When comparing the two +schemas, the `dev` branch contained this constraint while the `main` branch didn't. + +This example really happened. Read the [job failure logs](https://gitlab.com/gitlab-org/gitlab/-/jobs/1992050890). + +To fix these kind of issues, rebase the working branch onto the target branch to get the latest changes. diff --git a/doc/development/database/efficient_in_operator_queries.md b/doc/development/database/efficient_in_operator_queries.md index 76518a6f5e2..2503be826ea 100644 --- a/doc/development/database/efficient_in_operator_queries.md +++ b/doc/development/database/efficient_in_operator_queries.md @@ -368,7 +368,7 @@ then the query might perform worse than the non-optimized query. The `milestone_ "index_issues_on_milestone_id" btree (milestone_id) ``` -Adding the `miletone_id = X` filter to the `scope` argument or to the optimized scope causes bad performance. +Adding the `milestone_id = X` filter to the `scope` argument or to the optimized scope causes bad performance. Example (bad): @@ -618,7 +618,7 @@ The following example shows the final `ORDER BY` clause: ORDER BY extract('epoch' FROM epics.closed_at - epics.created_at) DESC, epics.id DESC ``` -Snippet for loading records ordered by the calcualted duration: +Snippet for loading records ordered by the calculated duration: ```ruby arel_table = Epic.arel_table @@ -641,7 +641,7 @@ records = Gitlab::Pagination::Keyset::InOperatorOptimization::QueryBuilder.new( array_mapping_scope: -> (id_expression) { Epic.where(Epic.arel_table[:group_id].eq(id_expression)) } ).execute.limit(20) -puts records.pluck(:duration_in_seconds, :id) # other columnns are not available +puts records.pluck(:duration_in_seconds, :id) # other columns are not available ``` Building the query requires quite a bit of configuration. For the order configuration you diff --git a/doc/development/database/index.md b/doc/development/database/index.md index a7b752e14ef..efc48f72d00 100644 --- a/doc/development/database/index.md +++ b/doc/development/database/index.md @@ -19,6 +19,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w - [Understanding EXPLAIN plans](../understanding_explain_plans.md) - [explain.depesz.com](https://explain.depesz.com/) or [explain.dalibo.com](https://explain.dalibo.com/) for visualizing the output of `EXPLAIN` - [pgFormatter](https://sqlformat.darold.net/) a PostgreSQL SQL syntax beautifier +- [db:check-migrations job](dbcheck-migrations-job.md) ## Migrations diff --git a/doc/development/database/loose_foreign_keys.md b/doc/development/database/loose_foreign_keys.md index 97d150b1a18..d08e90683fe 100644 --- a/doc/development/database/loose_foreign_keys.md +++ b/doc/development/database/loose_foreign_keys.md @@ -61,7 +61,7 @@ following information: - Child table name (`ci_pipelines`) - The data cleanup method (`async_delete` or `async_nullify`) -The YAML file is located at `lib/gitlab/database/gitlab_loose_foreign_keys.yml`. The file groups +The YAML file is located at `config/gitlab_loose_foreign_keys.yml`. The file groups foreign key definitions by the name of the child table. The child table can have multiple loose foreign key definitions, therefore we store them as an array. diff --git a/doc/development/database_review.md b/doc/development/database_review.md index 082c843a12c..8e217725a17 100644 --- a/doc/development/database_review.md +++ b/doc/development/database_review.md @@ -49,7 +49,7 @@ If new migrations are introduced, in the MR **you are required to provide**: Note that we have automated tooling for [GitLab](https://gitlab.com/gitlab-org/gitlab) (provided by the -`db:check-migrations` pipeline job) that provides this output for migrations on +[`db:check-migrations`](database/dbcheck-migrations-job.md) pipeline job) that provides this output for migrations on ~database merge requests. You do not need to provide this information manually if the bot can do it for you. The bot also checks that migrations are correctly reversible. @@ -66,7 +66,7 @@ Refer to [Preparation when adding or modifying queries](#preparation-when-adding ### Roles and process -A Merge Request **author**'s role is to: +A merge request **author**'s role is to: - Decide whether a database review is needed. - If database review is needed, add the ~database label. diff --git a/doc/development/dependencies.md b/doc/development/dependencies.md index c81c6408211..329539f0cc2 100644 --- a/doc/development/dependencies.md +++ b/doc/development/dependencies.md @@ -51,6 +51,6 @@ This has certain benefits as outlined in our <a href="https://docs.gitlab.com/ee You might find that we do not currently update DEPENDENCY automatically, but we are planning to do so in [the near future](https://gitlab.com/gitlab-org/frontend/rfcs/-/issues/21). -Thank you for understanding, I will close this Merge Request. +Thank you for understanding, I will close this merge request. /close ``` diff --git a/doc/development/deprecation_guidelines/index.md b/doc/development/deprecation_guidelines/index.md index 27c29a1ed7c..08e29e373f6 100644 --- a/doc/development/deprecation_guidelines/index.md +++ b/doc/development/deprecation_guidelines/index.md @@ -6,8 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Deprecation guidelines -This page includes information about how and when to remove or make breaking -changes to GitLab features. +This page includes information about how and when to remove or make [breaking +changes](../contributing/index.md#breaking-changes) to GitLab features. ## Terminology diff --git a/doc/development/diffs.md b/doc/development/diffs.md index aaa3340af33..d61de740f15 100644 --- a/doc/development/diffs.md +++ b/doc/development/diffs.md @@ -35,7 +35,7 @@ have changed since then, it should still serve as a good introduction. ### Merge request diffs -When refreshing a Merge Request (pushing to a source branch, force-pushing to target branch, or if the target branch now contains any commits from the MR) +When refreshing a merge request (pushing to a source branch, force-pushing to target branch, or if the target branch now contains any commits from the MR) we fetch the comparison information using `Gitlab::Git::Compare`, which fetches `base` and `head` data using Gitaly and diff between them through `Gitlab::Git::Diff.between`. The diffs fetching process _limits_ single file diff sizes and the overall size of the whole diff through a series of constant values. Raw diff files are @@ -45,7 +45,7 @@ Even though diffs larger than 10% of the value of `ApplicationSettings#diff_max_ we still keep them on PostgreSQL. However, diff files larger than defined _safety limits_ (see the [Diff limits section](#diff-limits)) are _not_ persisted in the database. -In order to present diffs information on the Merge Request diffs page, we: +In order to present diffs information on the merge request diffs page, we: 1. Fetch all diff files from database `merge_request_diff_files` 1. Fetch the _old_ and _new_ file blobs in batch to: diff --git a/doc/development/directory_structure.md b/doc/development/directory_structure.md index 8ba77bade2c..f909fda897f 100644 --- a/doc/development/directory_structure.md +++ b/doc/development/directory_structure.md @@ -38,7 +38,7 @@ end ### About namespace naming A good guideline for naming a top-level namespace (bounded context) is to use the related -feature category. For example, `Continuous Integration` feature category maps to `Ci::` namespace. +[feature category](https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/data/categories.yml). For example, `Continuous Integration` feature category maps to `Ci::` namespace. Alternatively a new class could be added to `Projects::` or `Groups::` if it's either: diff --git a/doc/development/distributed_tracing.md b/doc/development/distributed_tracing.md index 1e85abf585c..680ac71f857 100644 --- a/doc/development/distributed_tracing.md +++ b/doc/development/distributed_tracing.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md index 776e5aefd00..5cf7bb74549 100644 --- a/doc/development/documentation/index.md +++ b/doc/development/documentation/index.md @@ -68,7 +68,7 @@ Adhere to the [Documentation Style Guide](styleguide/index.md). If a style stand ## Folder structure and files -See the [Structure](styleguide/index.md#structure) section of the [Documentation Style Guide](styleguide/index.md). +See the [Folder structure](site_architecture/folder_structure.md) page. ## Metadata @@ -290,9 +290,7 @@ For example, [GitLab.com's `/help`](https://gitlab.com/help). ## Docs site architecture -See the [Docs site architecture](site_architecture/index.md) page to learn -how we build and deploy the site at <https://docs.gitlab.com> and -to review all the assets and libraries in use. +For information on how we build and deploy <https://docs.gitlab.com>, see [Docs site architecture](site_architecture/index.md). ### Global navigation diff --git a/doc/development/documentation/review_apps.md b/doc/development/documentation/review_apps.md index 4b58778a20c..8cb9e6437b8 100644 --- a/doc/development/documentation/review_apps.md +++ b/doc/development/documentation/review_apps.md @@ -41,7 +41,7 @@ the GitLab team to run the job. If you want to know the in-depth details, here's what's really happening: 1. You manually run the `review-docs-deploy` job in a merge request. -1. The job runs the [`scripts/trigger-build`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/scripts/trigger-build) +1. The job runs the [`scripts/trigger-build.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/scripts/trigger-build.rb) script with the `docs deploy` flag, which triggers the "Triggered from `gitlab-org/gitlab` 'review-docs-deploy' job" pipeline trigger in the `gitlab-org/gitlab-docs` project for the `$DOCS_BRANCH` (defaults to `main`). 1. The preview URL is shown both at the job output and in the merge request @@ -61,7 +61,7 @@ The following GitLab features are used among others: - [Review Apps](../../ci/review_apps/index.md) - [Artifacts](../../ci/yaml/index.md#artifacts) - [Specific runner](../../ci/runners/runners_scope.md#prevent-a-specific-runner-from-being-enabled-for-other-projects) -- [Pipelines for merge requests](../../ci/pipelines/merge_request_pipelines.md) +- [Merge request pipelines](../../ci/pipelines/merge_request_pipelines.md) ## Troubleshooting review apps diff --git a/doc/development/documentation/site_architecture/deployment_process.md b/doc/development/documentation/site_architecture/deployment_process.md index 25bc699c9d4..5203ca52922 100644 --- a/doc/development/documentation/site_architecture/deployment_process.md +++ b/doc/development/documentation/site_architecture/deployment_process.md @@ -82,7 +82,7 @@ for the stable branch of the image to rebuild. You might do this: ## Latest documentation -A Docker image (tagged `latest`) is built that contains: +We build a Docker image (tagged `latest`) that contains: - The latest online version of the documentation. - The documentation from the stable branches of upstream projects. diff --git a/doc/development/documentation/site_architecture/folder_structure.md b/doc/development/documentation/site_architecture/folder_structure.md new file mode 100644 index 00000000000..e960a6491c7 --- /dev/null +++ b/doc/development/documentation/site_architecture/folder_structure.md @@ -0,0 +1,98 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Folder structure for documentation + +The documentation is separated by top-level audience folders [`user`](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/doc/user), +[`administration`](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/doc/administration), +and [`development`](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/doc/development) +(contributing) folders. + +Beyond that, we primarily follow the structure of the GitLab user interface or +API. + +Our goal is to have a clear hierarchical structure with meaningful URLs like +`docs.gitlab.com/user/project/merge_requests/`. With this pattern, you can +immediately tell that you are navigating to user-related documentation about +Project features; specifically about Merge Requests. Our site's paths match +those of our repository, so the clear structure also makes documentation easier +to update. + +Put files for a specific product area into the related folder: + +| Directory | Contents | +|:----------------------|:------------------| +| `doc/user/` | Documentation for users. Anything that can be done in the GitLab user interface goes here, including usage of the `/admin` interface. | +| `doc/administration/` | Documentation that requires the user to have access to the server where GitLab is installed. Administrator settings in the GitLab user interface are under `doc/user/admin_area/`. | +| `doc/api/` | Documentation for the API. | +| `doc/development/` | Documentation related to the development of GitLab, whether contributing code or documentation. Related process and style guides should go here. | +| `doc/legal/` | Legal documents about contributing to GitLab. | +| `doc/install/` | Instructions for installing GitLab. | +| `doc/update/` | Instructions for updating GitLab. | +| `doc/topics/` | Indexes per topic (`doc/topics/topic_name/index.md`): all resources for that topic. | + +## Work with directories and files + +When working with directories and files: + +1. When you create a new directory, always start with an `index.md` file. + Don't use another filename and do not create `README.md` files. +1. Do not use special characters and spaces, or capital letters in file + names, directory names, branch names, and anything that generates a path. +1. When creating or renaming a file or directory and it has more than one word + in its name, use underscores (`_`) instead of spaces or dashes. For example, + proper naming would be `import_project/import_from_github.md`. This applies + to both image files and Markdown files. +1. For image files, do not exceed 100KB. +1. Do not upload video files to the product repositories. + [Link or embed videos](../styleguide/index.md#videos) instead. +1. There are four main directories: `user`, `administration`, `api`, and + `development`. +1. The `doc/user/` directory has five main subdirectories: `project/`, `group/`, + `profile/`, `dashboard/` and `admin_area/`. + - `doc/user/project/` should contain all project related documentation. + - `doc/user/group/` should contain all group related documentation. + - `doc/user/profile/` should contain all profile related documentation. + Every page you would navigate under `/profile` should have its own document, + for example, `account.md`, `applications.md`, or `emails.md`. + - `doc/user/dashboard/` should contain all dashboard related documentation. + - `doc/user/admin_area/` should contain all administrator-related + documentation describing what can be achieved by accessing the GitLab + administrator interface (not to be confused with `doc/administration` where + server access is required). + - Every category under `/admin/application_settings/` should have its + own document located at `doc/user/admin_area/settings/`. For example, + the **Visibility and Access Controls** category should have a document + located at `doc/user/admin_area/settings/visibility_and_access_controls.md`. +1. The `doc/topics/` directory holds topic-related technical content. Create + `doc/topics/topic_name/subtopic_name/index.md` when subtopics become necessary. + General user and administrator documentation should be placed accordingly. +1. The `/university/` directory is *deprecated* and the majority of its documentation + has been moved. + +If you're unsure where to place a document or a content addition, this shouldn't +stop you from authoring and contributing. Use your best judgment, and then ask +the reviewer of your MR to confirm your decision. You can also ask a technical writer at +any stage in the process. The technical writing team reviews all +documentation changes, regardless, and can move content if there is a better +place for it. + +## Avoid duplication + +Do not include the same information in multiple places. +[Link to a single source of truth instead.](../styleguide/index.md#link-instead-of-repeating-text) + +## References across documents + +- Give each folder an `index.md` page that introduces the topic, and both introduces + and links to the child pages, including to the index pages of + any next-level sub-paths. +- To ensure discoverability, ensure each new or renamed doc is linked from its + higher-level index page and other related pages. +- When making reference to other GitLab products and features, link to their + respective documentation, at least on first mention. +- When making reference to third-party products or technologies, link out to + their external sites, documentation, and resources. diff --git a/doc/development/documentation/site_architecture/index.md b/doc/development/documentation/site_architecture/index.md index 60894430d15..e7a915eab09 100644 --- a/doc/development/documentation/site_architecture/index.md +++ b/doc/development/documentation/site_architecture/index.md @@ -52,19 +52,20 @@ product, and all together are pulled to generate the docs website: - [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner/-/tree/main/docs) - [GitLab Chart](https://gitlab.com/charts/gitlab/tree/master/doc) +Learn more about [the docs folder structure](folder_structure.md). + ## Assets To provide an optimized site structure, design, and a search-engine friendly website, along with a discoverable documentation, we use a few assets for the GitLab Documentation website. -### Libraries +### External libraries + +GitLab Docs is built with a combination of external: -- [Bootstrap 4.3.1 components](https://getbootstrap.com/docs/4.3/components/) -- [Bootstrap 4.3.1 JS](https://getbootstrap.com/docs/4.3/getting-started/javascript/) -- [jQuery](https://jquery.com/) 3.3.1 -- [Clipboard JS](https://clipboardjs.com/) -- [Font Awesome 4.7.0](https://fontawesome.com/v4.7.0/icons/) +- [JavaScript libraries](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/package.json). +- [Ruby libraries](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/Gemfile). ### SEO @@ -98,7 +99,7 @@ The pipeline in the `gitlab-docs` project: Once a week on Mondays, a scheduled pipeline runs and rebuilds the Docker images used in various pipeline jobs, like `docs-lint`. The Docker image configuration files are -located in the [Dockerfiles directory](https://gitlab.com/gitlab-org/gitlab-docs/-/tree/master/dockerfiles). +located in the [Dockerfiles directory](https://gitlab.com/gitlab-org/gitlab-docs/-/tree/main/dockerfiles). If you need to rebuild the Docker images immediately (must have maintainer level permissions): diff --git a/doc/development/documentation/structure.md b/doc/development/documentation/structure.md index 6ecffce01b4..21368098f39 100644 --- a/doc/development/documentation/structure.md +++ b/doc/development/documentation/structure.md @@ -5,10 +5,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w description: What to include in GitLab documentation pages. --- -# Documentation topic types +# Documentation topic types (CTRT) At GitLab, we have not traditionally used types for our content. However, we are starting to -move in this direction, and we now use four primary topic types: +move in this direction, and we now use four primary topic types (CTRT): - [Concept](#concept) - [Task](#task) @@ -154,7 +154,7 @@ If multiple causes or workarounds exist, consider putting them into a table form ## Other types of content There are other types of content in the GitLab documentation that don't -classify as one of the four primary [topic types](#documentation-topic-types). +classify as one of the four primary [topic types](#documentation-topic-types-ctrt). These include: - [Tutorials](#tutorials) @@ -174,7 +174,7 @@ In general, you might consider using a tutorial when: Tutorials are learning aids that complement our core documentation. They do not introduce new features. -Always use the primary [topic types](#documentation-topic-types) to document new features. +Always use the primary [topic types](#documentation-topic-types-ctrt) to document new features. Tutorials should be in this format: diff --git a/doc/development/documentation/styleguide/index.md b/doc/development/documentation/styleguide/index.md index e57ffb90028..3e9c0177d48 100644 --- a/doc/development/documentation/styleguide/index.md +++ b/doc/development/documentation/styleguide/index.md @@ -181,115 +181,6 @@ included in backticks. For example: - `git clone` is a command, so it must be lowercase, while Git is the product, so it must have a capital G. -## Structure - -We include concept and task topic types in the same larger topic. - -In general, we have one topic that's a landing page. -Below that topic in the left nav are individual topics. Each of these include a concept -and multiple related tasks, reference, and troubleshooting topics. - -### Folder structure overview - -The documentation is separated by top-level audience folders [`user`](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/doc/user), -[`administration`](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/doc/administration), -and [`development`](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/doc/development) -(contributing) folders. - -Beyond that, we primarily follow the structure of the GitLab user interface or -API. - -Our goal is to have a clear hierarchical structure with meaningful URLs like -`docs.gitlab.com/user/project/merge_requests/`. With this pattern, you can -immediately tell that you are navigating to user-related documentation about -Project features; specifically about Merge Requests. Our site's paths match -those of our repository, so the clear structure also makes documentation easier -to update. - -Put files for a specific product area into the related folder: - -| Directory | Contents | -|:----------------------|:------------------| -| `doc/user/` | Documentation for users. Anything that can be done in the GitLab user interface goes here, including usage of the `/admin` interface. | -| `doc/administration/` | Documentation that requires the user to have access to the server where GitLab is installed. Administrator settings in the GitLab user interface are under `doc/user/admin_area/`. | -| `doc/api/` | Documentation for the API. | -| `doc/development/` | Documentation related to the development of GitLab, whether contributing code or documentation. Related process and style guides should go here. | -| `doc/legal/` | Legal documents about contributing to GitLab. | -| `doc/install/` | Instructions for installing GitLab. | -| `doc/update/` | Instructions for updating GitLab. | -| `doc/topics/` | Indexes per topic (`doc/topics/topic_name/index.md`): all resources for that topic. | - -### Work with directories and files - -When working with directories and files: - -1. When you create a new directory, always start with an `index.md` file. - Don't use another filename and do not create `README.md` files. -1. Do not use special characters and spaces, or capital letters in file - names, directory names, branch names, and anything that generates a path. -1. When creating or renaming a file or directory and it has more than one word - in its name, use underscores (`_`) instead of spaces or dashes. For example, - proper naming would be `import_project/import_from_github.md`. This applies - to both image files and Markdown files. -1. For image files, do not exceed 100KB. -1. Do not upload video files to the product repositories. - [Link or embed videos](#videos) instead. -1. There are four main directories: `user`, `administration`, `api`, and - `development`. -1. The `doc/user/` directory has five main subdirectories: `project/`, `group/`, - `profile/`, `dashboard/` and `admin_area/`. - - `doc/user/project/` should contain all project related documentation. - - `doc/user/group/` should contain all group related documentation. - - `doc/user/profile/` should contain all profile related documentation. - Every page you would navigate under `/profile` should have its own document, - for example, `account.md`, `applications.md`, or `emails.md`. - - `doc/user/dashboard/` should contain all dashboard related documentation. - - `doc/user/admin_area/` should contain all administrator-related - documentation describing what can be achieved by accessing the GitLab - administrator interface (not to be confused with `doc/administration` where - server access is required). - - Every category under `/admin/application_settings/` should have its - own document located at `doc/user/admin_area/settings/`. For example, - the **Visibility and Access Controls** category should have a document - located at `doc/user/admin_area/settings/visibility_and_access_controls.md`. -1. The `doc/topics/` directory holds topic-related technical content. Create - `doc/topics/topic_name/subtopic_name/index.md` when subtopics become necessary. - General user and administrator documentation should be placed accordingly. -1. The `/university/` directory is *deprecated* and the majority of its documentation - has been moved. - -If you're unsure where to place a document or a content addition, this shouldn't -stop you from authoring and contributing. Use your best judgment, and then ask -the reviewer of your MR to confirm your decision. You can also ask a technical writer at -any stage in the process. The technical writing team reviews all -documentation changes, regardless, and can move content if there is a better -place for it. - -### Avoid duplication - -Do not include the same information in multiple places. -[Link to a single source of truth instead.](#link-instead-of-repeating-text) - -### References across documents - -- Give each folder an `index.md` page that introduces the topic, and both introduces - and links to the child pages, including to the index pages of - any next-level sub-paths. -- To ensure discoverability, ensure each new or renamed doc is linked from its - higher-level index page and other related pages. -- When making reference to other GitLab products and features, link to their - respective documentation, at least on first mention. -- When making reference to third-party products or technologies, link out to - their external sites, documentation, and resources. - -### Structure in documents - -- Include any and all applicable subsections as described on the - [structure and template](../structure.md) page. -- Structure content in alphabetical order in tables, lists, and so on, unless - there's a logical reason not to (for example, when mirroring the user - interface or an otherwise ordered sequence). - ## Language GitLab documentation should be clear and easy to understand. @@ -320,10 +211,11 @@ create an issue or an MR to propose a change to the user interface text. #### Feature names - Feature names are typically lowercase. -- Some features are capitalized, typically nouns that name GitLab-specific - capabilities or tools. - -See the [word list](word_list.md) for details. +- Some features require title case, typically nouns that name GitLab-specific capabilities or tools. Features requiring + title case should be: + - Added as a proper name to markdownlint [configuration](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.markdownlint.yml), + so that it can be consistently applied across all documentation. + - Added to the [word list](word_list.md). If the term is not in the word list, ask a GitLab Technical Writer for advice. @@ -1151,7 +1043,8 @@ When you take screenshots: - **Review how the image renders on the page.** Preview the image locally or use the review app in the merge request. Make sure the image isn't blurry or overwhelming. - **Be consistent.** Coordinate screenshots with the other screenshots already on - a documentation page for a consistent reading experience. + a documentation page for a consistent reading experience. Ensure your navigation theme + is **Indigo** and the syntax highlighting theme is **Light**. These are the default preferences. ### Add callouts @@ -1745,11 +1638,12 @@ If the content in a topic is not ready, use the disclaimer in the topic. ### Removing versions after each major release -Whenever a major GitLab release occurs, we remove all version references -to now-unsupported versions of GitLab. Note that this includes the removal of -specific instructions for users of non-supported GitLab versions. For example, -if GitLab versions 11.x and later are supported, special -instructions for users of GitLab 10 should be removed. +When a major GitLab release occurs, we remove all references +to now-unsupported versions. This removal includes version-specific instructions. For example, +if GitLab version 12.1 and later are supported, +instructions for users of GitLab 11 should be removed. + +[View the list of supported versions](https://about.gitlab.com/support/statement-of-support.html#version-support). To view historical information about a feature, review GitLab [release posts](https://about.gitlab.com/releases/), or search for the issue or diff --git a/doc/development/documentation/styleguide/word_list.md b/doc/development/documentation/styleguide/word_list.md index dc3dcba0c95..2c435cdc69d 100644 --- a/doc/development/documentation/styleguide/word_list.md +++ b/doc/development/documentation/styleguide/word_list.md @@ -58,11 +58,7 @@ Capitalize these words when you refer to the UI. Otherwise use lowercase. ## administrator -Use **administrator** instead of **admin** when talking about a user's access level. -Use lowercase unless you are referring to the **Admin** access level you select in the UI. - -To view the administrator access level, in the GitLab UI, go to the Admin Area and select -**Users**. Then select **New user**. +Use **administrator access** instead of **admin** when talking about a user's access level.  @@ -71,7 +67,7 @@ An **administrator** is not a [role](#roles) or [permission](#permissions). Use: - To do this thing, you must be an administrator. -- To do this thing, you must have the administrator access level. +- To do this thing, you must have administrator access. Instead of: @@ -82,6 +78,17 @@ Instead of: Use title case **Admin Area** to refer to the area of the UI that you access when you select **Menu > Admin**. This area of the UI says **Admin Area** at the top of the page and on the menu. +## agent + +Use lowercase to refer to the [GitLab agent for Kubernetes](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent). +For example: + +- To connect your cluster to GitLab, use the GitLab agent for Kubernetes. +- Install the agent in your cluster. +- Select an agent from the list. + +Do not use title case **GitLab Agent** or **GitLab Agent for Kubernetes**. + ## allow, enable Try to avoid **allow** and **enable**, unless you are talking about security-related features. @@ -102,7 +109,7 @@ This phrasing is more active and is from the user perspective, rather than the p Use uppercase for **Alpha**. For example: **The XYZ feature is in Alpha.** or **This Alpha release is ready to test.** -You might also want to link to [this section](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga) +You might also want to link to [this section](../../../policy/alpha-beta-support.md#alpha-features) in the handbook when writing about Alpha features. ## and/or @@ -128,7 +135,7 @@ Try to avoid **below** when referring to an example or table in a documentation Use uppercase for **Beta**. For example: **The XYZ feature is in Beta.** or **This Beta release is ready to test.** -You might also want to link to [this section](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga) +You might also want to link to [this section](../../../policy/alpha-beta-support.md#beta-features) in the handbook when writing about Beta features. ## blacklist @@ -194,6 +201,10 @@ CI/CD is always uppercase. No need to spell it out on first use. Use **CI/CD minutes** instead of **CI minutes**, **pipeline minutes**, **pipeline minutes quota**, or **CI pipeline minutes**. This decision was made in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/342813). +## CI/CD tunnel + +Use lowercase for **tunnel** in **CI/CD tunnel**. + ## click Do not use **click**. Instead, use **select** with buttons, links, menu items, and lists. @@ -312,6 +323,11 @@ Instead of: Use **expand** instead of **open** when you are talking about expanding or collapsing a section in the UI. +## FAQ + +We want users to find information quickly, and they rarely search for the term **FAQ**. +Information in FAQs belongs with other similar information, under an easily searchable topic title. + ## field Use **box** instead of **field** or **text box**. @@ -367,6 +383,11 @@ Use title case for **GitLab Runner**. This is the product you install. See also Use **GitLab self-managed** to refer to the product license for GitLab instances managed by customers themselves. +## guide + +We want to speak directly to users. On `docs.gitlab.com`, do not use **guide** as part of a page title. +For example, **Snowplow Guide**. Instead, speak about the feature itself, and how to use it. For example, **Use Snowplow to do xyz**. + ## Guest When writing about the Guest role: @@ -540,6 +561,8 @@ Do not use **navigate**. Use **go** instead. For example: - Go to this webpage. - Open a terminal and go to the `runner` directory. +([Vale](../testing.md#vale) rule: [`SubstitutionSuggestions.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/SubstitutionSuggestions.yml)) + ## need to, should Try to avoid **needs to**, because it's wordy. Avoid **should** when you can be more specific. If something is required, use **must**. @@ -658,10 +681,22 @@ Use **press** when talking about keyboard keys. For example: Do not use profanity. Doing so may negatively affect other users and contributors, which is contrary to the GitLab value of [Diversity, Inclusion, and Belonging](https://about.gitlab.com/handbook/values/#diversity-inclusion). +## provision + +Use the term **provision** when referring to provisioning cloud infrastructure. You provision the infrastructure, and then deploy applications to it. + +For example, you might write something like: + +- Provision an AWS EKS cluster and deploy your application to it. + ## push rules Use lowercase for **push rules**. +## register + +Use **register** instead of **sign up** when talking about creating an account. + ## Reporter When writing about the Reporter role: @@ -690,6 +725,9 @@ Roles are not the same as [**access levels**](#access-level). Use lowercase for **runners**. These are the agents that run CI/CD jobs. See also [GitLab Runner](#gitlab-runner) and [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/233529). +When referring to runners, if you have to specify that the runners are installed on a customer's GitLab instance, +use **self-managed** rather than **self-hosted**. + ## (s) Do not use **(s)** to make a word optionally plural. It can slow down comprehension. For example: @@ -735,6 +773,10 @@ Instead of: Use **select** with buttons, links, menu items, and lists. **Select** applies to more devices, while **click** is more specific to a mouse. +## self-managed + +Use **self-managed** to refer to a customer's installation of GitLab. Do not use **self-hosted**. + ## Service Desk Use title case for **Service Desk**. @@ -752,6 +794,10 @@ Use **sign in** instead of **sign on** or **log on** or **log in**. If the user You can use **single sign-on**. +## sign up + +Use **register** instead of **sign up** when talking about creating an account. + ## simply, simple Do not use **simply** or **simple**. If the user doesn't find the process to be simple, we lose their trust. ([Vale](../testing.md#vale) rule: [`Simplicity.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/Simplicity.yml)) @@ -883,10 +929,6 @@ Instead of: Do not use **utilize**. Use **use** instead. It's more succinct and easier for non-native English speakers to understand. ([Vale](../testing.md#vale) rule: [`SubstitutionSuggestions.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/SubstitutionSuggestions.yml)) -## Value Stream Analytics - -Use title case for **Value Stream Analytics**. - ## via Do not use Latin abbreviations. Use **with**, **through**, or **by using** instead. ([Vale](../testing.md#vale) rule: [`LatinTerms.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/LatinTerms.yml)) diff --git a/doc/development/documentation/testing.md b/doc/development/documentation/testing.md index 134183a0e75..905b44823c3 100644 --- a/doc/development/documentation/testing.md +++ b/doc/development/documentation/testing.md @@ -126,6 +126,24 @@ If you don't want to install all of the dependencies to test the links, you can: If you manually install `haml-lint` with this process, it does not update automatically and you should make sure your version matches the version used by GitLab. +## Update linter configuration + +[Vale configuration](#vale) and [markdownlint configuration](#markdownlint) is under source control in each +project, so updates must be committed to each project individually. + +We consider the configuration in the `gitlab` project as the source of truth and that's where all updates should +first be made. + +On a regular basis, the changes made in `gitlab` project to the Vale and markdownlint configuration should be +synchronized to the other projects. In `omnibus-gitlab`, `gitlab-runner`, and `charts/gitlab`: + +1. Create a new branch. +1. Copy the configuration files from the `gitlab` project into this branch, overwriting + the project's old configuration. Make sure no project-specific changes from the `gitlab` + project are included. For example, [`RelativeLinks.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/RelativeLinks.yml) + is hard coded for specific projects. +1. Create a merge request and submit it to a technical writer for review and merge. + ## Local linters To help adhere to the [documentation style guidelines](styleguide/index.md), and improve the content @@ -181,7 +199,7 @@ You can find Vale configuration in the following projects: - [`gitlab-runner`](https://gitlab.com/gitlab-org/gitlab-runner/-/tree/main/docs/.vale/gitlab) - [`omnibus-gitlab`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/tree/master/doc/.vale/gitlab) - [`charts`](https://gitlab.com/gitlab-org/charts/gitlab/-/tree/master/doc/.vale/gitlab) -- [`gitlab-development-kit`](https://gitlab.com/gitlab-org/gitlab-development-kit/-/tree/master/doc/.vale/gitlab) +- [`gitlab-development-kit`](https://gitlab.com/gitlab-org/gitlab-development-kit/-/tree/main/doc/.vale/gitlab) This configuration is also used in build pipelines, where [error-level rules](#vale-result-types) are enforced. @@ -293,23 +311,7 @@ To configure Vale in your editor, install one of the following as appropriate: - Sublime Text [`SublimeLinter-contrib-vale` package](https://packagecontrol.io/packages/SublimeLinter-contrib-vale). - Visual Studio Code [`errata-ai.vale-server` extension](https://marketplace.visualstudio.com/items?itemName=errata-ai.vale-server). - You can configure the plugin to - [display only a subset of alerts](#show-subset-of-vale-alerts). - - In the extension's settings: - - <!-- vale gitlab.Spelling = NO --> - - - Select the **Use CLI** checkbox. - - In the **Config** setting, enter an absolute - path to [`.vale.ini`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.vale.ini) - in one of the cloned GitLab repositories on your computer. - - <!-- vale gitlab.Spelling = YES --> - - - In the **Path** setting, enter the absolute path to the Vale binary. In most - cases, `vale` should work. To find the location, run `which vale` in a terminal. - + You can configure the plugin to [display only a subset of alerts](#show-subset-of-vale-alerts). - Vim [ALE plugin](https://github.com/dense-analysis/ale). - Emacs [Flycheck extension](https://github.com/flycheck/flycheck). This requires some configuration: @@ -344,7 +346,7 @@ To configure Vale in your editor, install one of the following as appropriate: In this setup the `markdownlint` checker is set as a "next" checker from the defined `vale` checker. Enabling this custom Vale checker provides error linting from both Vale and markdownlint. -We don't use [Vale Server](https://errata-ai.github.io/vale/#using-vale-with-a-text-editor-or-another-third-party-application). +We don't use [Vale Server](https://docs.errata.ai/vale-server/install). ### Configure pre-push hooks diff --git a/doc/development/documentation/workflow.md b/doc/development/documentation/workflow.md index 49ad51874e3..a12af51e436 100644 --- a/doc/development/documentation/workflow.md +++ b/doc/development/documentation/workflow.md @@ -79,7 +79,7 @@ A member of the Technical Writing team adds these labels: ### Reviewing and merging -Anyone with the [Maintainer role](../../user/permissions.md) to the relevant GitLab project can +Anyone with the Maintainer role to the relevant GitLab project can merge documentation changes. Maintainers must make a good-faith effort to ensure that the content: - Is clear and sufficiently easy for the intended audience to navigate and understand. @@ -157,7 +157,7 @@ Ensure the following if skipping an initial Technical Writer review: - Specific [user permissions](../../user/permissions.md) are documented. - New documents are linked from higher-level indexes, for discoverability. - The style guide is followed: - - For [directories and files](styleguide/index.md#work-with-directories-and-files). + - For [directories and files](site_architecture/folder_structure.md). - For [images](styleguide/index.md#images). Merge requests that change the location of documentation must always be reviewed by a Technical diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index 9f705f2c6f1..17e35d34ec7 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -11,8 +11,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w - **Write documentation.**: Add documentation to the `doc/` directory. Describe the feature and include screenshots, if applicable. Indicate [what editions](documentation/styleguide/index.md#product-tier-badges) the feature applies to. -- **Submit a MR to the `www-gitlab-com` project.**: Add the new feature to the +<!-- markdownlint-disable MD044 --> +- **Submit a MR to the [`www-gitlab-com`](https://gitlab.com/gitlab-com/www-gitlab-com) project.**: Add the new feature to the [EE features list](https://about.gitlab.com/features/). +<!-- markdownlint-enable MD044 --> ## Act as CE when unlicensed diff --git a/doc/development/emails.md b/doc/development/emails.md index 811619bb0ff..5361282334f 100644 --- a/doc/development/emails.md +++ b/doc/development/emails.md @@ -10,8 +10,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w A Sidekiq job is enqueued whenever `deliver_later` is called on an `ActionMailer`. If a mailer argument needs to be added or removed, it is important to ensure -both backward and forward compatibility. Adhere to the Sidekiq Style Guide steps for -[changing the arguments for a worker](sidekiq_style_guide.md#changing-the-arguments-for-a-worker). +both backward and forward compatibility. Adhere to the Sidekiq steps for +[changing the arguments for a worker](sidekiq/compatibility_across_updates.md#changing-the-arguments-for-a-worker). In the following example from [`NotificationService`](https://gitlab.com/gitlab-org/gitlab/-/blob/33ccb22e4fc271dbaac94b003a7a1a2915a13441/app/services/notification_service.rb#L74) adding or removing an argument in this mailer's definition may cause problems @@ -89,6 +89,8 @@ See the [Rails guides](https://guides.rubyonrails.org/action_mailer_basics.html# As mentioned, the part after `+` is ignored, and this message is sent to the mailbox for `gitlab-incoming@gmail.com`. +1. Read the [MailRoom Gem updates](#mailroom-gem-updates) section for more details before you proceed to make sure you have the right version of MailRoom installed. In summary, you need to update the `gitlab-mail_room` version in the `Gemfile` to the latest `gitlab-mail_room` temporarily and run `bundle install`. **Do not commit** this change as it's a temporary workaround. + 1. Run this command in the GitLab root directory to launch `mail_room`: ```shell diff --git a/doc/development/event_store.md b/doc/development/event_store.md index 7f2b9c86d27..c6d553f41cd 100644 --- a/doc/development/event_store.md +++ b/doc/development/event_store.md @@ -185,7 +185,7 @@ Changes to the schema require multiple rollouts. While the new version is being - Events get persisted in the Sidekiq queue as job arguments, so we could have 2 versions of the schema during deployments. As changing the schema ultimately impacts the Sidekiq arguments, please refer to our -[Sidekiq style guide](sidekiq_style_guide.md#changing-the-arguments-for-a-worker) with regards to multiple rollouts. +[Sidekiq style guide](sidekiq/compatibility_across_updates.md#changing-the-arguments-for-a-worker) with regards to multiple rollouts. #### Add properties @@ -232,7 +232,6 @@ the event safely via the `handle_event` method. For example: ```ruby module MergeRequests class UpdateHeadPipelineWorker - include ApplicationWorker include Gitlab::EventStore::Subscriber def handle_event(event) @@ -252,19 +251,20 @@ add a line like this to the `Gitlab::EventStore.configure!` method: ```ruby module Gitlab module EventStore - def self.configure! - Store.new.tap do |store| - # ... + def self.configure!(store) + # ... - store.subscribe ::MergeRequests::UpdateHeadPipelineWorker, to: ::Ci::PipelineCreatedEvent + store.subscribe ::MergeRequests::UpdateHeadPipelineWorker, to: ::Ci::PipelineCreatedEvent - # ... - end + # ... end end end ``` +A worker that is only defined in the EE codebase can subscribe to an event in the same way by +declaring the subscription in `ee/lib/ee/gitlab/event_store.rb`. + Subscriptions are stored in memory when the Rails app is loaded and they are immediately frozen. It's not possible to modify subscriptions at runtime. @@ -274,7 +274,7 @@ A subscription can specify a condition when to accept an event: ```ruby store.subscribe ::MergeRequests::UpdateHeadPipelineWorker, - to: ::Ci::PipelineCreatedEvent, + to: ::Ci::PipelineCreatedEvent, if: -> (event) { event.data[:merge_request_id].present? } ``` diff --git a/doc/development/experiment_guide/gitlab_experiment.md b/doc/development/experiment_guide/gitlab_experiment.md index 36d2a4f6fbf..369690ba86c 100644 --- a/doc/development/experiment_guide/gitlab_experiment.md +++ b/doc/development/experiment_guide/gitlab_experiment.md @@ -4,26 +4,24 @@ group: Adoption info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Implementing an A/B/n experiment using GLEX +# Implementing an A/B/n experiment ## Introduction -`Gitlab::Experiment` (GLEX) is tightly coupled with the concepts provided by -[Feature flags in development of GitLab](../feature_flags/index.md). Here, we refer -to this layer as feature flags, and may also use the term Flipper, because we -built our development and experiment feature flags atop it. - -You're strongly encouraged to read and understand the -[Feature flags in development of GitLab](../feature_flags/index.md) portion of the -documentation before considering running experiments. Experiments add additional -concepts which may seem confusing or advanced without understanding the underpinnings -of how GitLab uses feature flags in development. One concept: GLEX supports -experiments with multiple variants, which are sometimes referred to as A/B/n tests. - -The [`gitlab-experiment` project](https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment) -exists in a separate repository, so it can be shared across any GitLab property that uses -Ruby. You should feel comfortable reading the documentation on that project as well -if you want to dig into more advanced topics. +Experiments in GitLab are tightly coupled with the concepts provided by +[Feature flags in development of GitLab](../feature_flags/index.md). You're strongly encouraged +to read and understand the [Feature flags in development of GitLab](../feature_flags/index.md) +portion of the documentation before considering running experiments. Experiments add additional +concepts which may seem confusing or advanced without understanding the underpinnings of how GitLab +uses feature flags in development. One concept: experiments can be run with multiple variants, +which are sometimes referred to as A/B/n tests. + +We use the [`gitlab-experiment` gem](https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment), +sometimes referred to as GLEX, to run our experiments. The gem exists in a separate repository +so it can be shared across any GitLab property that uses Ruby. You should feel comfortable reading +the documentation on that project if you want to dig into more advanced topics or open issues. Be +aware that the documentation there reflects what's in the main branch and may not be the same as +the version being used within GitLab. ## Glossary of terms @@ -35,41 +33,9 @@ when communicating about experiments: - `control`: The default, or "original" code path. - `candidate`: Defines an experiment with only one code path. - `variant(s)`: Defines an experiment with multiple code paths. +- `behaviors`: Used to reference all possible code paths of an experiment, including the control. -### How it works - -Use this decision tree diagram to understand how GLEX works. When an experiment runs, -the following logic is executed to determine what variant should be provided, -given how the experiment has been defined and using the provided context: - -```mermaid -graph TD - GP[General Pool/Population] --> Running? - Running? -->|Yes| Cached?[Cached? / Pre-segmented?] - Running? -->|No| Excluded[Control / No Tracking] - Cached? -->|No| Excluded? - Cached? -->|Yes| Cached[Cached Value] - Excluded? -->|Yes| Excluded - Excluded? -->|No| Segmented? - Segmented? -->|Yes / Cached| VariantA - Segmented? -->|No| Included?[Experiment Group?] - Included? -->|Yes| Rollout - Included? -->|No| Control - Rollout -->|Cached| VariantA - Rollout -->|Cached| VariantB - Rollout -->|Cached| VariantC - -classDef included fill:#380d75,color:#ffffff,stroke:none -classDef excluded fill:#fca121,stroke:none -classDef cached fill:#2e2e2e,color:#ffffff,stroke:none -classDef default fill:#fff,stroke:#6e49cb - -class VariantA,VariantB,VariantC included -class Control,Excluded excluded -class Cached cached -``` - -## Implement an experiment +## Implementing an experiment [Examples](https://gitlab.com/gitlab-org/growth/growth/-/wikis/GLEX-Framework-code-examples) @@ -87,9 +53,9 @@ experiment in code. An experiment implementation can be as simple as: ```ruby experiment(:pill_color, actor: current_user) do |e| - e.use { 'control' } - e.try(:red) { 'red' } - e.try(:blue) { 'blue' } + e.control { 'control' } + e.variant(:red) { 'red' } + e.variant(:blue) { 'blue' } end ``` @@ -146,11 +112,11 @@ We can also implement this experiment in a HAML file with HTML wrappings: ```haml #cta-interface - experiment(:pill_color, actor: current_user) do |e| - - e.use do + - e.control do .pill-button control - - e.try(:red) do + - e.variant(:red) do .pill-button.red red - - e.try(:blue) do + - e.variant(:blue) do .pill-button.blue blue ``` @@ -212,38 +178,30 @@ wouldn't be resolvable. ### Advanced experimentation -GLEX allows for two general implementation styles: +There are two ways to implement an experiment: 1. The simple experiment style described previously. -1. A more advanced style where an experiment class can be provided. +1. A more advanced style where an experiment class is provided. The advanced style is handled by naming convention, and works similar to what you would expect in Rails. To generate a custom experiment class that can override the defaults in -`ApplicationExperiment` (our base GLEX implementation), use the rails generator: +`ApplicationExperiment` use the Rails generator: ```shell rails generate gitlab:experiment pill_color control red blue ``` This generates an experiment class in `app/experiments/pill_color_experiment.rb` -with the variants (or _behaviors_) we've provided to the generator. Here's an example -of how that class would look after migrating the previous example into it: +with the _behaviors_ we've provided to the generator. Here's an example +of how that class would look after migrating our previous example into it: ```ruby class PillColorExperiment < ApplicationExperiment - def control_behavior - 'control' - end - - def red_behavior - 'red' - end - - def blue_behavior - 'blue' - end + control { 'control' } + variant(:red) { 'red' } + variant(:blue) { 'blue' } end ``` @@ -254,13 +212,13 @@ providing the block we were initially providing, by explicitly calling `run`: experiment(:pill_color, actor: current_user).run ``` -The _behavior_ methods we defined in our experiment class represent the default -implementation. You can still use the block syntax to override these _behavior_ -methods however, so the following would also be valid: +The _behaviors_ we defined in our experiment class represent the default +implementation. You can still use the block syntax to override these _behaviors_ +however, so the following would also be valid: ```ruby experiment(:pill_color, actor: current_user) do |e| - e.use { '<strong>control</strong>' } + e.control { '<strong>control</strong>' } end ``` @@ -279,11 +237,11 @@ variant, and any account older than 2 weeks old would be assigned the _blue_ var ```ruby class PillColorExperiment < ApplicationExperiment + # ...registered behaviors + segment(variant: :red) { context.actor.first_name == 'Richard' } segment :old_account?, variant: :blue - # ...behaviors - private def old_account? @@ -315,9 +273,9 @@ be nothing - but no events are tracked in these cases as well. ```ruby class PillColorExperiment < ApplicationExperiment - exclude :old_account?, ->{ context.actor.first_name == 'Richard' } + # ...registered behaviors - # ...behaviors + exclude :old_account?, ->{ context.actor.first_name == 'Richard' } private @@ -327,23 +285,11 @@ class PillColorExperiment < ApplicationExperiment end ``` -We can also do exclusion when we run the experiment. For instance, -if we wanted to prevent the inclusion of non-administrators in an experiment, consider -the following experiment. This type of logic enables us to do complex experiments -while preventing us from passing things into our experiments, because -we want to minimize passing things into our experiments: - -```ruby -experiment(:pill_color, actor: current_user) do |e| - e.exclude! unless can?(current_user, :admin_project, project) -end -``` - You may also need to check exclusion in custom tracking logic by calling `should_track?`: ```ruby class PillColorExperiment < ApplicationExperiment - # ...behaviors + # ...registered behaviors def expensive_tracking_logic return unless should_track? @@ -353,16 +299,11 @@ class PillColorExperiment < ApplicationExperiment end ``` -Exclusion rules aren't the best way to determine if an experiment is active. Override -the `enabled?` method for a high-level way of determining if an experiment should -run and track. Make the `enabled?` check as efficient as possible because it's the -first early opt-out path an experiment can implement. - ### Tracking events One of the most important aspects of experiments is gathering data and reporting on -it. GLEX provides an interface that allows tracking events across an experiment. -You can implement it consistently if you provide the same context between +it. You can use the `track` method to track events across an experimental implementation. +You can track events consistently to an experiment if you provide the same context between calls to your experiment. If you do not yet understand context, you should read about contexts now. @@ -373,13 +314,13 @@ the arguments you would normally use when of tracking an event in Ruby would be: ```ruby -experiment(:pill_color, actor: current_user).track(:created) +experiment(:pill_color, actor: current_user).track(:clicked) ``` -When you run an experiment with any of these examples, an `:assigned` event +When you run an experiment with any of the examples so far, an `:assigned` event is tracked automatically by default. All events that are tracked from an experiment have a special -[experiment context](https://gitlab.com/gitlab-org/iglu/-/blob/master/public/schemas/com.gitlab/gitlab_experiment/jsonschema/1-0-0) +[experiment context](https://gitlab.com/gitlab-org/iglu/-/blob/master/public/schemas/com.gitlab/gitlab_experiment/jsonschema/1-0-3) added to the event. This can be used - typically by the data team - to create a connection between the events on a given experiment. @@ -390,28 +331,20 @@ event would be tracked at that time for them. NOTE: GitLab tries to be sensitive and respectful of our customers regarding tracking, -so GLEX allows us to implement an experiment without ever tracking identifying +so our experimentation library allows us to implement an experiment without ever tracking identifying IDs. It's not always possible, though, based on experiment reporting requirements. You may be asked from time to time to track a specific record ID in experiments. The approach is largely up to the PM and engineer creating the implementation. No recommendations are provided here at this time. -## Test with RSpec +## Testing with RSpec -This gem provides some RSpec helpers and custom matchers. These are in flux as of GitLab 13.10. - -First, require the RSpec support file to mix in some of the basics: - -```ruby -require 'gitlab/experiment/rspec' -``` - -You still need to include matchers and other aspects, which happens -automatically for files in `spec/experiments`, but for other files and specs -you want to include it in, you can specify the `:experiment` type: +In the course of working with experiments, you'll probably want to utilize the RSpec +tooling that's built in. This happens automatically for files in `spec/experiments`, but +for other files and specs you want to include it in, you can specify the `:experiment` type: ```ruby -it "tests", :experiment do +it "tests experiments nicely", :experiment do end ``` @@ -421,28 +354,32 @@ You can stub experiments using `stub_experiments`. Pass it a hash using experime names as the keys, and the variants you want each to resolve to, as the values: ```ruby -# Ensures the experiments named `:example` & `:example2` are both -# "enabled" and that each will resolve to the given variant -# (`:my_variant` & `:control` respectively). +# Ensures the experiments named `:example` & `:example2` are both "enabled" and +# that each will resolve to the given variant (`:my_variant` and `:control` +# respectively). stub_experiments(example: :my_variant, example2: :control) experiment(:example) do |e| e.enabled? # => true - e.variant.name # => 'my_variant' + e.assigned.name # => 'my_variant' end experiment(:example2) do |e| e.enabled? # => true - e.variant.name # => 'control' + e.assigned.name # => 'control' end ``` -### Exclusion and segmentation matchers +### Exclusion, segmentation, and behavior matchers -You can also test the exclusion and segmentation matchers. +You can also test things like the registered behaviors, the exclusions, and +segmentations using the matchers. ```ruby class ExampleExperiment < ApplicationExperiment + control { } + candidate { '_candidate_' } + exclude { context.actor.first_name == 'Richard' } segment(variant: :candidate) { context.actor.username == 'jejacks0n' } end @@ -450,6 +387,10 @@ end excluded = double(username: 'rdiggitty', first_name: 'Richard') segmented = double(username: 'jejacks0n', first_name: 'Jeremy') +# register_behavior matcher +expect(experiment(:example)).to register_behavior(:control) +expect(experiment(:example)).to register_behavior(:candidate).with('_candidate_') + # exclude matcher expect(experiment(:example)).to exclude(actor: excluded) expect(experiment(:example)).not_to exclude(actor: segmented) @@ -495,35 +436,36 @@ expect(experiment(:example)).to track(:my_event, value: 1, property: '_property_ experiment(:example, :variant_name, foo: :bar).track(:my_event, value: 1, property: '_property_') ``` -### Recording and assignment tracking - -To test assignment tracking and the `record!` method, you can use or adopt the following -shared example: [tracks assignment and records the subject](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/support/shared_examples/lib/gitlab/experimentation_shared_examples.rb). - ## Experiments in the client layer -This is in flux as of GitLab 13.10, and can't be documented just yet. +Any experiment that's been run in the request lifecycle surfaces in `window.gl.experiments`, +and matches [this schema](https://gitlab.com/gitlab-org/iglu/-/blob/master/public/schemas/com.gitlab/gitlab_experiment/jsonschema/1-0-3) +so it can be used when resolving experimentation in the client layer. -Any experiment that's been run in the request lifecycle surfaces in and `window.gl.experiments`, -and matches [this schema](https://gitlab.com/gitlab-org/iglu/-/blob/master/public/schemas/com.gitlab/gitlab_experiment/jsonschema/1-0-0) -so you can use it when resolving some concepts around experimentation in the client layer. +Given that we've defined a class for our experiment, and have defined the variants for it, we can publish that experiment in a couple ways. -### Use experiments in Vue +The first way is simply by running the experiment. Assuming the experiment has been run, it will surface in the client layer without having to do anything special. -With the `gitlab-experiment` component, you can define slots that match the name of the -variants pushed to `window.gl.experiments`. For example, if we alter the `pill_color` -experiment to just use the default variants of `control` and `candidate` like so: +The second way doesn't run the experiment and is intended to be used if the experiment only needs to surface in the client layer. To accomplish this we can simply `.publish` the experiment. This won't run any logic, but does surface the experiment details in the client layer so they can be utilized there. + +An example might be to publish an experiment in a `before_action` in a controller. Assuming we've defined the `PillColorExperiment` class, like we have above, we can surface it to the client by publishing it instead of running it: ```ruby -def show - experiment(:pill_color) do |e| - e.use { } # control - e.try { } # candidate - end.run -end +before_action -> { experiment(:pill_color).publish }, only: [:show] ``` -We can make use of the named slots `control` and `candidate` in the Vue component: +You can then see this surface in the JavaScript console: + +```javascript +window.gl.experiments // => { pill_color: { excluded: false, experiment: "pill_color", key: "ca63ac02", variant: "candidate" } } +``` + +### Using experiments in Vue + +With the `gitlab-experiment` component, you can define slots that match the name of the +variants pushed to `window.gl.experiments`. + +We can make use of the named slots in the Vue component, that match the behaviors defined in : ```vue <script> @@ -540,23 +482,6 @@ export default { <button class="bg-default">Click default button</button> </template> - <template #candidate> - <button class="bg-red">Click red button</button> - </template> - </gitlab-experiment> -</template> -``` - -When you're coding for an experiment with multiple variants, you can use the variant names. -For example, the Vue component for the previously-defined `pill_color` experiment with `red` and `blue` variants would look like this: - -```vue -<template> - <gitlab-experiment name="pill_color"> - <template #control> - <button class="bg-default">Click default button</button> - </template> - <template #red> <button class="bg-red">Click red button</button> </template> @@ -596,7 +521,7 @@ NOTE: This method of stubbing in Jest specs will not automatically un-stub itself at the end of the test. We merge our stubbed experiment in with all the other global data in `window.gl`. If you need to remove the stubbed experiment(s) after your test or ensure a clean global object before your test, you'll need to manage the global object directly yourself: ```javascript -desribe('tests that care about global state', () => { +describe('tests that care about global state', () => { const originalObjects = []; beforeEach(() => { @@ -628,7 +553,7 @@ is viewed as being either `on` or `off`, this isn't accurate for experiments. Generally, `off` means that when we ask if a feature flag is enabled, it will always return `false`, and `on` means that it will always return `true`. An interim state, -considered `conditional`, also exists. GLEX takes advantage of this trinary state of +considered `conditional`, also exists. We take advantage of this trinary state of feature flags. To understand this `conditional` aspect: consider that either of these settings puts a feature flag into this state: @@ -638,7 +563,7 @@ settings puts a feature flag into this state: Conditional means that it returns `true` in some situations, but not all situations. When a feature flag is disabled (meaning the state is `off`), the experiment is -considered _inactive_. You can visualize this in the [decision tree diagram](#how-it-works) +considered _inactive_. You can visualize this in the [decision tree diagram](https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment#how-it-works) as reaching the first `Running?` node, and traversing the negative path. When a feature flag is rolled out to a `percentage_of_actors` or similar (meaning the diff --git a/doc/development/experiment_guide/index.md b/doc/development/experiment_guide/index.md index 8d62f92e0b9..254c136ef79 100644 --- a/doc/development/experiment_guide/index.md +++ b/doc/development/experiment_guide/index.md @@ -54,7 +54,7 @@ This uses [experiment](../feature_flags/index.md#experiment-type) feature flags. Some experiments may require you to add custom icons or illustrations to our codebase. This process is lengthy and at this stage, the outcome of the experiment uncertain. -Therefore, you should postpone this effort until the [experiment cleanup process](https://about.gitlab.com/handbook/engineering/development/growth/#experiment-cleanup-issue). +Therefore, you should postpone this effort until the [experiment cleanup process](https://about.gitlab.com/handbook/engineering/development/growth/experimentation/#experiment-cleanup-issue). We recommend the following workflow: diff --git a/doc/development/export_csv.md b/doc/development/export_csv.md index e2bbd0491d6..ff827023a50 100644 --- a/doc/development/export_csv.md +++ b/doc/development/export_csv.md @@ -11,8 +11,8 @@ This document lists the different implementations of CSV export in GitLab codeba | Export type | How it works | Advantages | Disadvantages | Existing examples | |---|---|---|---|---| | Streaming | - Query and yield data in batches to a response stream.<br>- Download starts immediately. | - Report available immediately. | - No progress indicator.<br>- Requires a reliable connection. | [Export Audit Event Log](../administration/audit_events.md#export-to-csv) | -| Downloading | - Query and write data in batches to a temporary file.<br>- Loads the file into memory.<br>- Sends the file to the client. | - Report available immediately. | - Large amount of data might cause request timeout.<br>- Memory intensive.<br>- Request expires when user navigates to a different page. | [Export Chain of Custody Report](../user/compliance/compliance_report/#chain-of-custody-report) | -| As email attachment | - Asynchronously process the query with background job.<br>- Email uses the export as an attachment. | - Asynchronous processing. | - Requires users use a different app (email) to download the CSV.<br>- Email providers may limit attachment size. | - [Export Issues](../user/project/issues/csv_export.md)<br>- [Export Merge Requests](../user/project/merge_requests/csv_export.md) | +| Downloading | - Query and write data in batches to a temporary file.<br>- Loads the file into memory.<br>- Sends the file to the client. | - Report available immediately. | - Large amount of data might cause request timeout.<br>- Memory intensive.<br>- Request expires when user navigates to a different page. | - [Export Chain of Custody Report](../user/compliance/compliance_report/#chain-of-custody-report)<br>- [Export License Usage File](../subscriptions/self_managed/index.md#export-your-license-usage) | +| As email attachment | - Asynchronously process the query with background job.<br>- Email uses the export as an attachment. | - Asynchronous processing. | - Requires users use a different app (email) to download the CSV.<br>- Email providers may limit attachment size. | - [Export issues](../user/project/issues/csv_export.md)<br>- [Export merge requests](../user/project/merge_requests/csv_export.md) | | As downloadable link in email (*) | - Asynchronously process the query with background job.<br>- Email uses an export link. | - Asynchronous processing.<br>- Bypasses email provider attachment size limit. | - Requires users use a different app (email).<br>- Requires additional storage and cleanup. | [Export User Permissions](https://gitlab.com/gitlab-org/gitlab/-/issues/1772) | | Polling (non-persistent state) | - Asynchronously processes the query with the background job.<br>- Frontend(FE) polls every few seconds to check if CSV file is ready. | - Asynchronous processing.<br>- Automatically downloads to local machine on completion.<br>- In-app solution. | - Non-persistable request - request expires when user navigates to a different page.<br>- API is processed for each polling request. | [Export Vulnerabilities](../user/application_security/vulnerability_report/#export-vulnerability-details) | | Polling (persistent state) (*) | - Asynchronously processes the query with background job.<br>- Backend (BE) maintains the export state<br>- FE polls every few seconds to check status.<br>- FE shows 'Download link' when export is ready.<br>- User can download or regenerate a new report. | - Asynchronous processing.<br>- No database calls made during the polling requests (HTTP 304 status is returned until export status changes).<br>- Does not require user to stay on page until export is complete.<br>- In-app solution.<br>- Can be expanded into a generic CSV feature (such as dashboard / CSV API). | - Requires to maintain export states in DB.<br>- Does not automatically download the CSV export to local machine, requires users to click 'Download' button. | [Export Merge Commits Report](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/43055) | diff --git a/doc/development/fe_guide/design_anti_patterns.md b/doc/development/fe_guide/design_anti_patterns.md index 9e602b1ea04..76825d6ff18 100644 --- a/doc/development/fe_guide/design_anti_patterns.md +++ b/doc/development/fe_guide/design_anti_patterns.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w Anti-patterns may seem like good approaches at first, but it has been shown that they bring more ills than benefits. These should generally be avoided. -Throughout the GitLab codebase, there may be historic uses of these anti-patterns. Please [use discretion](https://about.gitlab.com/handbook/engineering/#balance-refactoring-and-velocity) +Throughout the GitLab codebase, there may be historic uses of these anti-patterns. Please [use discretion](https://about.gitlab.com/handbook/engineering/principles/#balance-refactoring-and-velocity) when figuring out whether or not to refactor, when touching code that uses one of these legacy patterns. NOTE: diff --git a/doc/development/fe_guide/development_process.md b/doc/development/fe_guide/development_process.md index cd82a7dadc3..9921b851344 100644 --- a/doc/development/fe_guide/development_process.md +++ b/doc/development/fe_guide/development_process.md @@ -64,6 +64,30 @@ Please use your best judgment when to use it and please contribute new points th - [ ] Follow up on issues that came out of the review. Create issues for discovered edge cases that should be covered in future iterations. ``` +### Code deletion checklist + +When your merge request deletes code, it's important to also delete all +related code that is no longer used. +When deleting Haml and Vue code, check whether it contains the following types of +code that is unused: + +- CSS. + + For example, we've deleted a Vue component that contained the `.mr-card` class, which is now unused. + The `.mr-card` CSS rule set should then be deleted from `merge_requests.scss`. + +- Ruby variables. + + Deleting unused Ruby variables is important so we don't continue instantiating them with + potentially expensive code. + + For example, we've deleted a Haml template that used the `@total_count` Ruby variable. + The `@total_count` variable was no longer used in the remaining templates for the page. + The instantiation of `@total_count` in `issues_controller.rb` should then be deleted so that we + don't make unnecessary database calls to calculate the count of issues. + +- Ruby methods. + ### Merge Request Review With the purpose of being [respectful of others' time](https://about.gitlab.com/handbook/values/#be-respectful-of-others-time) please follow these guidelines when asking for a review: diff --git a/doc/development/fe_guide/graphql.md b/doc/development/fe_guide/graphql.md index ed71f612061..e79a473df9e 100644 --- a/doc/development/fe_guide/graphql.md +++ b/doc/development/fe_guide/graphql.md @@ -107,9 +107,9 @@ Default client accepts two parameters: `resolvers` and `config`. ### Multiple client queries for the same object -If you are make multiple queries to the same Apollo client object you might encounter the following error: "Store error: the application attempted to write an object with no provided ID but the store already contains an ID of SomeEntity". [This error only should occur when you have made a query with an ID field for a portion, then made another that returns what would be the same object, but is missing the ID field.](https://github.com/apollographql/apollo-client/issues/2510#issue-271829009) +If you are making multiple queries to the same Apollo client object you might encounter the following error: `Cache data may be lost when replacing the someProperty field of a Query object. To address this problem, either ensure all objects of SomeEntityhave an id or a custom merge function`. We are already checking `ID` presence for every GraphQL type that has an `ID`, so this shouldn't be the case. Most likely, the `SomeEntity` type doesn't have an `ID` property, and to fix this warning we need to define a custom merge function. -This is being tracked in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/326101) and the documentation will be updated when this issue is resolved. +We have some client-wide types with `merge: true` defined in the default client as [typePolicies](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/lib/graphql.js) (this means that Apollo will merge existing and incoming responses in the case of subsequent queries). Please consider adding `SomeEntity` there or defining a custom merge function for it. ## GraphQL Queries @@ -667,9 +667,7 @@ apollo: { ``` When we want to move to the next page, we use an Apollo `fetchMore` method, passing a -new cursor (and, optionally, new variables) there. In the `updateQuery` hook, we have -to return a result we want to see in the Apollo cache after fetching the next page. -[`Immer`s `produce`](#immutability-and-cache-updates)-function can help us with the immutability here: +new cursor (and, optionally, new variables) there. ```javascript fetchNextPage(endCursor) { @@ -679,24 +677,114 @@ fetchNextPage(endCursor) { first: 10, after: endCursor, }, - updateQuery(previousResult, { fetchMoreResult }) { - // Here we can implement the logic of adding new designs to existing ones - // (for example, if we use infinite scroll) or replacing old result - // with the new one if we use numbered pages - - const { designs: previousDesigns } = previousResult.project.issue.designCollection; - const { designs: newDesigns } = fetchMoreResult.project.issue.designCollection - - return produce(previousResult, draftData => { - // `produce` gives us a working copy, `draftData`, that we can modify - // as we please and from it will produce the next immutable result for us - draftData.project.issue.designCollection.designs = [...previousDesigns, ...newDesigns]; - }); - }, }); } ``` +##### Defining field merge policy + +We would also need to define a field policy to specify how do we want to merge the existing results with the incoming results. For example, if we have `Previous/Next` buttons, it makes sense to replace the existing result with the incoming one: + +```javascript +const apolloProvider = new VueApollo({ + defaultClient: createDefaultClient( + {}, + { + cacheConfig: { + typePolicies: { + DesignCollection: { + fields: { + designs: { + merge(existing, incoming) { + if (!incoming) return existing; + if (!existing) return incoming; + + // We want to save only incoming nodes and replace existing ones + return incoming + } + } + } + } + } + }, + }, + ), +}); +``` + +When we have an infinite scroll, it would make sense to add the incoming `designs` nodes to existing ones instead of replacing. In this case, merge function would be slightly different: + +```javascript +const apolloProvider = new VueApollo({ + defaultClient: createDefaultClient( + {}, + { + cacheConfig: { + typePolicies: { + DesignCollection: { + fields: { + designs: { + merge(existing, incoming) { + if (!incoming) return existing; + if (!existing) return incoming; + + const { nodes, ...rest } = incoming; + // We only need to merge the nodes array. + // The rest of the fields (pagination) should always be overwritten by incoming + let result = rest; + result.nodes = [...existing.nodes, ...nodes]; + return result; + } + } + } + } + } + }, + }, + ), +}); +``` + +`apollo-client` [provides](https://github.com/apollographql/apollo-client/blob/212b1e686359a3489b48d7e5d38a256312f81fde/src/utilities/policies/pagination.ts) +a few field policies to be used with paginated queries. Here's another way to achieve infinite +scroll pagination with the `concatPagination` policy: + +```javascript +import { concatPagination } from '@apollo/client/utilities'; +import Vue from 'vue'; +import VueApollo from 'vue-apollo'; +import createDefaultClient from '~/lib/graphql'; + +Vue.use(VueApollo); + +export default new VueApollo({ + defaultClient: createDefaultClient( + {}, + { + cacheConfig: { + typePolicies: { + Project: { + fields: { + dastSiteProfiles: { + keyArgs: ['fullPath'], // You might need to set the keyArgs option to enforce the cache's integrity + }, + }, + }, + DastSiteProfileConnection: { + fields: { + nodes: concatPagination(), + }, + }, + }, + }, + }, + ), +}); +``` + +This is similar to the `DesignCollection` example above as new page results are appended to the +previous ones. + #### Using a recursive query in components When it is necessary to fetch all paginated data initially an Apollo query can do the trick for us. @@ -816,7 +904,7 @@ const data = store.readQuery({ }); ``` -Read more about the `@connection` directive in [Apollo's documentation](https://www.apollographql.com/docs/react/v2/caching/cache-interaction/#the-connection-directive). +Read more about the `@connection` directive in [Apollo's documentation](https://www.apollographql.com/docs/react/caching/advanced-topics/#the-connection-directive). ### Managing performance @@ -1017,22 +1105,13 @@ apollo: { issuableId: convertToGraphQLId(this.issuableClass, this.issuableId), }; }, - // Describe how subscription should update the query - updateQuery(prev, { subscriptionData }) { - if (prev && subscriptionData?.data?.issuableAssigneesUpdated) { - const data = produce(prev, (draftData) => { - draftData.workspace.issuable.assignees.nodes = - subscriptionData.data.issuableAssigneesUpdated.assignees.nodes; - }); - return data; - } - return prev; - }, }, }, }, ``` +We would need also to define a field policy similarly like we do it for the [paginated queries](#defining-field-merge-policy) + ### Best Practices #### When to use (and not use) `update` hook in mutations @@ -1081,55 +1160,6 @@ If you use the RubyMine IDE, and have marked the `tmp` directory as `gitlab/tmp/tests/graphql`. This will allow the **JS GraphQL** plugin to automatically find and index the schema. -#### Mocking response as component data - -<!-- vale gitlab.Spelling = NO --> - -With [Vue Test Utils](https://vue-test-utils.vuejs.org/) one can quickly test components that -fetch GraphQL queries. The simplest way is to use `shallowMount` and then set -the data on the component: - -<!-- vale gitlab.Spelling = YES --> - -```javascript -it('tests apollo component', () => { - const vm = shallowMount(App); - - vm.setData({ - ...mockData - }); -}); -``` - -#### Testing loading state - -To test how a component renders when results from the GraphQL API are still loading, mock a loading state into respective Apollo queries/mutations: - -```javascript - function createComponent({ - loading = false, - } = {}) { - const $apollo = { - queries: { - designs: { - loading, - }, - }, - }; - - wrapper = shallowMount(Index, { - sync: false, - mocks: { $apollo } - }); - } - - it('renders loading icon', () => { - createComponent({ loading: true }); - - expect(wrapper.element).toMatchSnapshot(); -}) -``` - #### Testing Apollo components If we use `ApolloQuery` or `ApolloMutation` in our components, in order to test their functionality we need to add a stub first: @@ -1197,11 +1227,9 @@ it('calls mutation on submitting form ', () => { }); ``` -### Testing with mocked Apollo Client - -To test the logic of Apollo cache updates, we might want to mock an Apollo Client in our unit tests. We use [`mock-apollo-client`](https://www.npmjs.com/package/mock-apollo-client) library to mock Apollo client and [`createMockApollo` helper](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/frontend/__helpers__/mock_apollo_helper.js) we created on top of it. +#### Mocking Apollo Client -To separate tests with mocked client from 'usual' unit tests, create an additional factory and pass the created `mockApollo` as an option to the `createComponent`-factory. This way we only create Apollo Client instance when it's necessary. +To test the components with Apollo operations, we need to mock an Apollo Client in our unit tests. We use [`mock-apollo-client`](https://www.npmjs.com/package/mock-apollo-client) library to mock Apollo client and [`createMockApollo` helper](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/frontend/__helpers__/mock_apollo_helper.js) we created on top of it. We need to inject `VueApollo` into the Vue instance by calling `Vue.use(VueApollo)`. This will install `VueApollo` globally for all the tests in the file. It is recommended to call `Vue.use(VueApollo)` just after the imports. @@ -1320,8 +1348,7 @@ it('renders designs list', async () => { const mockApollo = createMockApolloProvider(); const wrapper = createComponent({ mockApollo }); - jest.runOnlyPendingTimers(); - await wrapper.vm.$nextTick(); + await waitForPromises() expect(findDesigns()).toHaveLength(3); }); @@ -1342,8 +1369,7 @@ function createMockApolloProvider() { it('renders error if query fails', async () => { const wrapper = createComponent(); - jest.runOnlyPendingTimers(); - await wrapper.vm.$nextTick(); + await waitForPromises() expect(wrapper.find('.test-error').exists()).toBe(true) }) @@ -1351,7 +1377,7 @@ it('renders error if query fails', async () => { Request handlers can also be passed to component factory as a parameter. -Mutations could be tested the same way with a few additional `nextTick`s to get the updated result: +Mutations could be tested the same way: ```javascript function createMockApolloProvider({ @@ -1391,7 +1417,7 @@ it('calls a mutation with correct parameters and reorders designs', async () => expect(moveDesignHandler).toHaveBeenCalled(); - await wrapper.vm.$nextTick(); + await waitForPromises(); expect( findDesigns() @@ -1407,8 +1433,7 @@ To mock multiple query response states, success and failure, Apollo Client's nat describe('when query times out', () => { const advanceApolloTimers = async () => { jest.runOnlyPendingTimers(); - await wrapper.vm.$nextTick(); - await wrapper.vm.$nextTick(); + await waitForPromises() }; beforeEach(async () => { @@ -1419,7 +1444,7 @@ describe('when query times out', () => { .mockResolvedValueOnce({ errors: [{ message: 'timeout' }] }); createComponentWithApollo(failSucceedFail); - await wrapper.vm.$nextTick(); + await waitForPromises(); }); it('shows correct errors and does not overwrite populated data when data is empty', async () => { @@ -1862,6 +1887,16 @@ relative to `app/graphql/queries` folder: for example, if we need a `app/graphql/queries/repository/files.query.graphql` query, the path is `repository/files`. +## Troubleshooting + +### Mocked client returns empty objects instead of mock response + +If your unit test is failing because response contains empty objects instead of mock data, you would need to add `__typename` field to the mocked response. This happens because mocked client (unlike the real one) does not populate the response with typenames and in some cases we need to do it manually so the client is able to recognize a GraphQL type. + +### Warning about losing cache data + +Sometimes you can see a warning in the console: `Cache data may be lost when replacing the someProperty field of a Query object. To address this problem, either ensure all objects of SomeEntityhave an id or a custom merge function`. Please check section about [multiple queries](#multiple-client-queries-for-the-same-object) to resolve an issue. + ```yaml - current_route_path = request.fullpath.match(/-\/tree\/[^\/]+\/(.+$)/).to_a[1] - add_page_startup_graphql_call('repository/path_last_commit', { projectPath: @project.full_path, ref: current_ref, path: current_route_path || "" }) diff --git a/doc/development/fe_guide/style/javascript.md b/doc/development/fe_guide/style/javascript.md index f22e3ea6395..4a0923ebe19 100644 --- a/doc/development/fe_guide/style/javascript.md +++ b/doc/development/fe_guide/style/javascript.md @@ -136,24 +136,40 @@ the class name with `js-`. ## ES Module Syntax -Use ES module syntax to import modules: +For most JavaScript files, use ES module syntax to import or export from modules. +Prefer named exports, as they improve name consistency. ```javascript -// bad -const SomeClass = require('some_class'); +// bad (with exceptions, see below) +export default SomeClass; +import SomeClass from 'file'; // good -import SomeClass from 'some_class'; +export { SomeClass }; +import { SomeClass } from 'file'; +``` + +Using default exports is acceptable in a few particular circumstances: + +- Vue Single File Components (SFCs) +- Vuex mutation files + +For more information, see [RFC 20](https://gitlab.com/gitlab-org/frontend/rfcs/-/issues/20). +## CommonJS Module Syntax + +Our Node configuration requires CommonJS module syntax. Prefer named exports. + +```javascript // bad module.exports = SomeClass; +const SomeClass = require('./some_class'); // good -export default SomeClass; +module.exports = { SomeClass }; +const { SomeClass } = require('./some_class'); ``` -We still use `require` in `scripts/` and `config/` files. - ## Absolute vs relative paths for modules Use relative paths if the module you are importing is less than two levels up. diff --git a/doc/development/feature_flags/index.md b/doc/development/feature_flags/index.md index 820b4bab802..af402713f6e 100644 --- a/doc/development/feature_flags/index.md +++ b/doc/development/feature_flags/index.md @@ -169,7 +169,7 @@ Each feature flag is defined in a separate YAML file consisting of a number of f | `name` | yes | Name of the feature flag. | | `type` | yes | Type of feature flag. | | `default_enabled` | yes | The default state of the feature flag that is strictly validated, with `default_enabled:` passed as an argument. | -| `introduced_by_url` | no | The URL to the Merge Request that introduced the feature flag. | +| `introduced_by_url` | no | The URL to the merge request that introduced the feature flag. | | `rollout_issue_url` | no | The URL to the Issue covering the feature flag rollout. | | `milestone` | no | Milestone in which the feature was added. | | `group` | no | The [group](https://about.gitlab.com/handbook/product/categories/#devops-stages) that owns the feature flag. | @@ -513,7 +513,7 @@ Feature.remove(:feature_flag_name) - Any change behind a feature flag **disabled** by default **should not** have a changelog entry. - **Exception:** database migrations **should** have a changelog entry. -- Any change related to a feature flag itself (flag removal, default-on setting) **should** have a changelog entry. +- Any change related to a feature flag itself (flag removal, default-on setting) **should** have [a changelog entry](../changelog.md). Use the flowchart to determine the changelog entry type. ```mermaid diff --git a/doc/development/features_inside_dot_gitlab.md b/doc/development/features_inside_dot_gitlab.md index 73ba9cbd674..283a0d5d5fb 100644 --- a/doc/development/features_inside_dot_gitlab.md +++ b/doc/development/features_inside_dot_gitlab.md @@ -11,7 +11,7 @@ When implementing new features, please refer to these existing features to avoid - [Custom Dashboards](../operations/metrics/dashboards/index.md#add-a-new-dashboard-to-your-project): `.gitlab/dashboards/`. - [Issue Templates](../user/project/description_templates.md#create-an-issue-template): `.gitlab/issue_templates/`. -- [Merge Request Templates](../user/project/description_templates.md#create-a-merge-request-template): `.gitlab/merge_request_templates/`. +- [Merge request Templates](../user/project/description_templates.md#create-a-merge-request-template): `.gitlab/merge_request_templates/`. - [GitLab Agent](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/configuration_repository.md#layout): `.gitlab/agents/`. - [CODEOWNERS](../user/project/code_owners.md#set-up-code-owners): `.gitlab/CODEOWNERS`. - [Route Maps](../ci/review_apps/#route-maps): `.gitlab/route-map.yml`. diff --git a/doc/development/img/architecture_simplified.png b/doc/development/img/architecture_simplified.png Binary files differindex bd731758ddd..bab673feb4a 100644 --- a/doc/development/img/architecture_simplified.png +++ b/doc/development/img/architecture_simplified.png diff --git a/doc/development/import_project.md b/doc/development/import_project.md index 9e236b4cfce..86e6e04347c 100644 --- a/doc/development/import_project.md +++ b/doc/development/import_project.md @@ -125,6 +125,14 @@ it fails with this error as `/` is not a valid character in a project name. A project with that name already exists. +##### `Exception: Error importing repository into (namespace) - No space left on device` + +The disk has insufficient space to complete the import. + +During import, the tarball is cached in your configured `shared_path` directory. Verify the +disk has enough free space to accommodate both the cached tarball and the unpacked +project files on disk. + ### Importing via the Rails console The last option is to import a project using a Rails console: diff --git a/doc/development/index.md b/doc/development/index.md index 197c7f48398..0501f70b818 100644 --- a/doc/development/index.md +++ b/doc/development/index.md @@ -221,10 +221,10 @@ the [reviewer values](https://about.gitlab.com/handbook/engineering/workflow/rev - [Geo development](geo.md) - [Redis guidelines](redis.md) - [Adding a new Redis instance](redis/new_redis_instance.md) -- [Sidekiq guidelines](sidekiq_style_guide.md) for working with Sidekiq workers +- [Sidekiq guidelines](sidekiq/index.md) for working with Sidekiq workers - [Working with Gitaly](gitaly.md) - [Elasticsearch integration docs](elasticsearch.md) -- [Working with Merge Request diffs](diffs.md) +- [Working with merge request diffs](diffs.md) - [Approval Rules](approval_rules.md) - [Repository mirroring](repository_mirroring.md) - [File uploads](uploads.md) diff --git a/doc/development/integrations/img/copy_cookies.png b/doc/development/integrations/img/copy_cookies.png Binary files differdeleted file mode 100644 index 21575987173..00000000000 --- a/doc/development/integrations/img/copy_cookies.png +++ /dev/null diff --git a/doc/development/integrations/img/copy_curl.png b/doc/development/integrations/img/copy_curl.png Binary files differdeleted file mode 100644 index 9fa871efbd5..00000000000 --- a/doc/development/integrations/img/copy_curl.png +++ /dev/null diff --git a/doc/development/integrations/jira_connect.md b/doc/development/integrations/jira_connect.md index fc7204fdd5a..cfa1fdba699 100644 --- a/doc/development/integrations/jira_connect.md +++ b/doc/development/integrations/jira_connect.md @@ -65,27 +65,3 @@ If the app install failed, you might need to delete `jira_connect_installations` 1. Open the [database console](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/postgresql.md#access-postgresql). 1. Run `TRUNCATE TABLE jira_connect_installations CASCADE;`. - -## Add a namespace - -To add a [namespace](../../user/group/index.md#namespaces) to Jira: - -1. Make sure you are logged in on your GitLab development instance. -1. On the GitLab app page in Jira, click **Get started**. -1. Open your browser's developer tools and navigate to the **Network** tab. -1. Try to add the namespace in Jira. -1. If the request fails with 401 "not authorized", copy the request as a cURL command - and paste it in your terminal. - -  - -1. Go to your development instance (usually at: <http://localhost:3000>), open developer - tools, navigate to the Network tab and reload the page. -1. Copy all cookies from the first request. - -  - -1. Append the cookies to the cURL command in your terminal: - `--cookies "<cookies from the request>"`. -1. Submit the cURL request. -1. If the response is `{"success":true}`, the namespace was added. diff --git a/doc/development/integrations/secure.md b/doc/development/integrations/secure.md index 147d379cec7..27a166aebf9 100644 --- a/doc/development/integrations/secure.md +++ b/doc/development/integrations/secure.md @@ -10,7 +10,7 @@ Integrating a security scanner into GitLab consists of providing end users with a [CI job definition](../../ci/yaml/index.md) they can add to their CI configuration files to scan their GitLab projects. This CI job should then output its results in a GitLab-specified format. These results are then -automatically presented in various places in GitLab, such as the Pipeline view, Merge Request +automatically presented in various places in GitLab, such as the Pipeline view, merge request widget, and Security Dashboard. The scanning job is usually based on a [Docker image](https://docs.docker.com/) diff --git a/doc/development/internal_api/index.md b/doc/development/internal_api/index.md index 96910892022..db978253747 100644 --- a/doc/development/internal_api/index.md +++ b/doc/development/internal_api/index.md @@ -507,7 +507,7 @@ curl --request POST --header "Gitlab-Kas-Api-Request: <JWT token>" \ Called from the GitLab Agent Server (`kas`) to create a security vulnerability from a Starboard vulnerability report. This request is idempotent. Multiple requests with the same data -create a single vulnerability. +create a single vulnerability. The response contains the UUID of the created vulnerability finding. | Attribute | Type | Required | Description | |:----------------|:-------|:---------|:------------| @@ -553,6 +553,37 @@ curl --request PUT --header "Gitlab-Kas-Api-Request: <JWT token>" \ }' ``` +Example response: + +```json +{ + "uuid": "4773b2ee-5ba5-5e9f-b48c-5f7a17f0faac" +} +``` + +### Resolve Starboard vulnerabilities + +Called from the GitLab Agent Server (`kas`) to resolve Starboard security vulnerabilities. +Accepts a list of finding UUIDs and marks all Starboard vulnerabilities not identified by +the list as resolved. + +| Attribute | Type | Required | Description | +|:----------|:-------------|:---------|:----------------------------------------------------------------------------------------------------------------------------------| +| `uuids` | string array | yes | UUIDs of detected vulnerabilities, as collected from [Create Starboard vulnerability](#create-starboard-vulnerability) responses. | + +```plaintext +POST internal/kubernetes/modules/starboard_vulnerability/scan_result +``` + +Example Request: + +```shell +curl --request POST --header "Gitlab-Kas-Api-Request: <JWT token>" \ + --header "Authorization: Bearer <agent token>" --header "Content-Type: application/json" \ + --url "http://localhost:3000/api/v4/internal/kubernetes/modules/starboard_vulnerability/scan_result" \ + --data '{ "uuids": ["102e8a0a-fe29-59bd-b46c-57c3e9bc6411", "5eb12985-0ed5-51f4-b545-fd8871dc2870"] }' +``` + ## Subscriptions The subscriptions endpoint is used by [CustomersDot](https://gitlab.com/gitlab-org/customers-gitlab-com) (`customers.gitlab.com`) diff --git a/doc/development/issuable-like-models.md b/doc/development/issuable-like-models.md index b3be2b1b2e6..42dfe2e0f2f 100644 --- a/doc/development/issuable-like-models.md +++ b/doc/development/issuable-like-models.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w GitLab Rails codebase contains several models that hold common functionality and behave similarly to [Issues](../user/project/issues/index.md). Other examples of "issuables" -are [Merge Requests](../user/project/merge_requests/index.md) and +are [merge requests](../user/project/merge_requests/index.md) and [Epics](../user/group/epics/index.md). This guide accumulates guidelines on working with such Rails models. diff --git a/doc/development/issue_types.md b/doc/development/issue_types.md index 31fa50e1d97..e6047c62827 100644 --- a/doc/development/issue_types.md +++ b/doc/development/issue_types.md @@ -11,7 +11,7 @@ We are deprecating Issue Types as of GitLab 14.2 in favor of [Work Items and Wor Sometimes when a new resource type is added it's not clear if it should be only an "extension" of Issue (Issue Type) or if it should be a new first-class resource type -(similar to Issue, Epic, Merge Request, Snippet). +(similar to issue, epic, merge request, snippet). The idea of Issue Types was first proposed in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/8767) and its usage was diff --git a/doc/development/iterating_tables_in_batches.md b/doc/development/iterating_tables_in_batches.md index 9ea7fd3dc29..38cdbdf5b79 100644 --- a/doc/development/iterating_tables_in_batches.md +++ b/doc/development/iterating_tables_in_batches.md @@ -142,7 +142,7 @@ database query: SELECT "users"."id" FROM "users" ORDER BY "users"."id" ASC LIMIT 1 ``` - + Notice that the query only reads data from the index (`INDEX ONLY SCAN`), the table is not accessed. Database indexes are sorted so taking out the first item is a very cheap operation. @@ -155,7 +155,7 @@ to get a "shifted" `id` value. SELECT "users"."id" FROM "users" WHERE "users"."id" >= 1 ORDER BY "users"."id" ASC LIMIT 1 OFFSET 5 ``` - + Again, the query only looks into the index. The `OFFSET 5` takes out the sixth `id` value: this query reads a maximum of six items from the index regardless of the table size or the iteration @@ -181,7 +181,7 @@ previous iteration in order to find out the next end `id` value. SELECT "users"."id" FROM "users" WHERE "users"."id" >= 302 ORDER BY "users"."id" ASC LIMIT 1 OFFSET 5 ``` - + Now we can easily construct the `users` query for the second iteration. diff --git a/doc/development/jh_features_review.md b/doc/development/jh_features_review.md index 858048c1e7c..88830a80bf1 100644 --- a/doc/development/jh_features_review.md +++ b/doc/development/jh_features_review.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: https://gitlab.com/gitlab-jh/gitlab +info: https://jihulab.com/gitlab-cn/gitlab --- # Guidelines for reviewing JiHu (JH) Edition related merge requests @@ -21,13 +21,13 @@ We have two kinds of changes related to JH: - We will generalize this so both EE and JH can share the same mechanism, then we wouldn't have to treat them differently. -If needed, review the corresponding JH merge request located at [JH repository](https://gitlab.com/gitlab-jh/gitlab) +If needed, review the corresponding JH merge request located at [JH repository](https://jihulab.com/gitlab-cn/gitlab). ## When to merge files to the GitLab Inc. repository -Files that are added to the `gitlab-jh` repository outside of `jh/` must be mirrored in the GitLab Inc. repository. +Files that are added to the GitLab JH repository outside of `jh/` must be mirrored in the GitLab Inc. repository. -If code that is added to the GitLab Inc. repository references (for example, `render_if_exists`) any `gitlab-jh` file that does not +If code that is added to the GitLab Inc. repository references (for example, `render_if_exists`) any GitLab JH file that does not exist in the GitLab Inc. codebase, add a comment with a link to the JiHu merge request or file. This is to prevent the reference from being misidentified as a missing partial and subsequently deleted in the `gitlab` codebase. @@ -49,7 +49,7 @@ This page is the single source of truth for JiHu-related processes. ## CI pipelines in a JH context EE repository does not have `jh/` directory therefore there is no way to run -JH pipelines in the EE repository. All JH tests should go to [JH repository](https://gitlab.com/gitlab-jh/gitlab). +JH pipelines in the EE repository. All JH tests should go to [JH repository](https://jihulab.com/gitlab-cn/gitlab). The top-level JH CI configuration is located at `jh/.gitlab-ci.yml` (which does not exist in EE repository) and it'll include EE CI configurations @@ -89,7 +89,7 @@ Do not use methods such as `prepend`, `extend`, and `include`. Instead, use the relevant EE and JH modules by the name of the receiver module. If reviewing the corresponding JH file is needed, it should be found at -[JH repository](https://gitlab.com/gitlab-jh/gitlab). +[JH repository](https://jihulab.com/gitlab-cn/gitlab). ### General guidance for writing JH extensions diff --git a/doc/development/logging.md b/doc/development/logging.md index a4eda6ad02e..6a0b50d6970 100644 --- a/doc/development/logging.md +++ b/doc/development/logging.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -398,3 +398,14 @@ end 1. Be sure to update the [GitLab CE/EE documentation](../administration/logs.md) and the [GitLab.com runbooks](https://gitlab.com/gitlab-com/runbooks/blob/master/docs/logging/README.md). + +## Control logging visibility + +An increase in the logs can cause a growing backlog of unacknowledged messages. When adding new log messages, make sure they don't increase the overall volume of logging by more than 10%. + +### Deprecation notices + +If the expected volume of deprecation notices is large: + +- Only log them in the development environment. +- If needed, log them in the testing environment. diff --git a/doc/development/merge_request_performance_guidelines.md b/doc/development/merge_request_performance_guidelines.md index 106db862122..c8e99e8547f 100644 --- a/doc/development/merge_request_performance_guidelines.md +++ b/doc/development/merge_request_performance_guidelines.md @@ -255,15 +255,15 @@ It re-instantiates project object for each build, instead of using the same in-m In this particular case the workaround is fairly easy: ```ruby +ActiveRecord::Associations::Preloader.new.preload(pipeline, [builds: :project]) + pipeline.builds.each do |build| - build.project = pipeline.project build.to_json(only: [:name], include: [project: { only: [:name]}]) end ``` -We can assign `pipeline.project` to each `build.project`, since we know it should point to the same project. -This allows us that each build point to the same in-memory project, -avoiding the cached SQL query and re-instantiation of the project object for each build. +`ActiveRecord::Associations::Preloader` uses the same in-memory object for the same project. +This avoids the cached SQL query and also avoids re-instantiation of the project object for each build. ## Executing Queries in Loops diff --git a/doc/development/multi_version_compatibility.md b/doc/development/multi_version_compatibility.md index 27c4edf15f4..bdab92f5185 100644 --- a/doc/development/multi_version_compatibility.md +++ b/doc/development/multi_version_compatibility.md @@ -14,14 +14,14 @@ In a sense, these scenarios are all transient states. But they can often persist ### When modifying a Sidekiq worker -For example when [changing arguments](sidekiq_style_guide.md#changing-the-arguments-for-a-worker): +For example when [changing arguments](sidekiq/compatibility_across_updates.md#changing-the-arguments-for-a-worker): - Is it ok if jobs are being enqueued with the old signature but executed by the new monthly release? - Is it ok if jobs are being enqueued with the new signature but executed by the previous monthly release? ### When adding a new Sidekiq worker -Is it ok if these jobs don't get executed for several hours because [Sidekiq nodes are not yet updated](sidekiq_style_guide.md#adding-new-workers)? +Is it ok if these jobs don't get executed for several hours because [Sidekiq nodes are not yet updated](sidekiq/compatibility_across_updates.md#adding-new-workers)? ### When modifying JavaScript @@ -89,7 +89,7 @@ Many users [skip some monthly releases](../update/index.md#upgrading-to-a-new-ma - 13.0 => 13.12 -These users accept some downtime during the update. Unfortunately we can't ignore this case completely. For example, 13.12 may execute Sidekiq jobs from 13.0, which illustrates why [we avoid removing arguments from jobs until a major release](sidekiq_style_guide.md#deprecate-and-remove-an-argument). The main question is: Will the deployment get to a good state after the update is complete? +These users accept some downtime during the update. Unfortunately we can't ignore this case completely. For example, 13.12 may execute Sidekiq jobs from 13.0, which illustrates why [we avoid removing arguments from jobs until a major release](sidekiq/compatibility_across_updates.md#deprecate-and-remove-an-argument). The main question is: Will the deployment get to a good state after the update is complete? ## What kind of components can GitLab be broken down into? @@ -180,7 +180,7 @@ coexists in production. ### Changing Sidekiq worker's parameters -This topic is explained in detail in [Sidekiq Compatibility across Updates](sidekiq_style_guide.md#sidekiq-compatibility-across-updates). +This topic is explained in detail in [Sidekiq Compatibility across Updates](sidekiq/compatibility_across_updates.md). When we need to add a new parameter to a Sidekiq worker class, we can split this into the following steps: diff --git a/doc/development/new_fe_guide/modules/widget_extensions.md b/doc/development/new_fe_guide/modules/widget_extensions.md index b833ba7c630..37712cb2cec 100644 --- a/doc/development/new_fe_guide/modules/widget_extensions.md +++ b/doc/development/new_fe_guide/modules/widget_extensions.md @@ -1,6 +1,6 @@ --- stage: Create -group: Source Code +group: Code Review info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -11,27 +11,39 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Summary Extensions in the merge request widget enable you to add new features -into the widget that match the existing design and interaction as other extensions. +into the merge request widget that match the design framework. +With extensions we get a lot of benefits out of the box without much effort required, like: + +- A consistent look and feel. +- Tracking when the extension is opened. +- Virtual scrolling for performance. ## Usage -To use extensions you need to first create a new extension object to fetch the -data to render in the extension. See the example file in -`app/assets/javascripts/vue_merge_request_widget/extensions/issues.js` for a working example. +To use extensions you must first create a new extension object to fetch the +data to render in the extension. For a working example, refer to the example file in +`app/assets/javascripts/vue_merge_request_widget/extensions/issues.js`. -The basic object structure is as below: +The basic object structure: ```javascript export default { - name: '', - props: [], + name: '', // Required: This helps identify the widget + props: [], // Required: Props passed from the widget state + i18n: { // Required: Object to hold i18n text + label: '', // Required: Used for tooltips and aria-labels + loading: '', // Required: Loading text for when data is loading + }, + expandEvent: '', // Optional: RedisHLL event name to track expanding content + enablePolling: false, // Optional: Tells extension to poll for data computed: { - summary() {}, - statusIcon() {}, + summary(data) {}, // Required: Level 1 summary text + statusIcon(data) {}, // Required: Level 1 status icon + tertiaryButtons() {}, // Optional: Level 1 action buttons }, methods: { - fetchCollapsedData() {}, - fetchFullData() {}, + fetchCollapsedData(props) {}, // Required: Fetches data required for collapsed state + fetchFullData(props) {}, // Required: Fetches data for the full expanded content }, }; ``` @@ -39,10 +51,8 @@ export default { By following the same data structure, each extension can follow the same registering structure, but each extension can manage its data sources. -After creating this structure you need to register it. Registering the extension can happen at any -point _after_ the widget has been created. - -To register a extension the following can be done: +After creating this structure, you must register it. You can register the extension at any +point _after_ the widget has been created. To register a extension: ```javascript // Import the register method @@ -55,18 +65,174 @@ import issueExtension from '~/vue_merge_request_widget/extensions/issues'; registerExtension(issueExtension); ``` -## Fetching errors +## Data fetching + +Each extension must fetch data. Fetching is handled when registering the extension, +not by the core component itself. This approach allows for various different +data fetching methods to be used, such as GraphQL or REST API calls. + +### API calls + +For performance reasons, it is best if the collapsed state fetches only the data required to +render the collapsed state. This fetching happens within the `fetchCollapsedData` method. +This method is called with the props as an argument, so you can easily access +any paths set in the state. + +To allow the extension to set the data, this method **must** return the data. No +special formatting is required. When the extension receives this data, +it is set to `collapsedData`. You can access `collapsedData` in any computed property or +method. + +When the user clicks **Expand**, the `fetchFullData` method is called. This method +also gets called with the props as an argument. This method **must** also return +the full data. However, this data needs to be correctly formatted to match the format +mentioned in the data structure section. + +#### Technical debt + +For some of the current extensions, there is no split in data fetching. All the data +is fetched through the `fetchCollapsedData` method. While less performant, +it allows for faster iteration. + +To handle this the `fetchFullData` returns the data set through +the `fetchCollapsedData` method call. In these cases, the `fetchFullData` must +return a promise: + +```javascript +fetchCollapsedData() { + return ['Some data']; +}, +fetchFullData() { + return Promise.resolve(this.collapsedData) +}, +``` + +### Data structure + +The data returned from `fetchFullData` must match the format below. This format +allows the core component to render the data in a way that matches +the design framework. Any text properties can use the styling placeholders +mentioned below: + +```javascript +{ + id: data.id, // Required: ID used as a key for each row + header: 'Header' || ['Header', 'sub-header'], // Required: String or array can be used for the header text + text: '', // Required: Main text for the row + subtext: '', // Optional: Smaller sub-text to be displayed below the main text + icon: { // Optional: Icon object + name: EXTENSION_ICONS.success, // Required: The icon name for the row + }, + badge: { // Optional: Badge displayed after text + text: '', // Required: Text to be displayed inside badge + variant: '', // Optional: GitLab UI badge variant, defaults to info + }, + actions: [], // Optional: Action button for row +} +``` + +### Polling + +To enable polling for an extension, an options flag must be present in the extension: + +```javascript +export default { + //... + enablePolling: true +}; +``` + +This flag tells the base component we should poll the `fetchCollapsedData()` +defined in the extension. Polling stops if the response has data, or if an error is present. + +When writing the logic for `fetchCollapsedData()`, a complete Axios response must be returned +from the method. The polling utility needs data like polling headers to work correctly: + +```javascript +export default { + //... + enablePolling: true + methods: { + fetchCollapsedData() { + return axios.get(this.reportPath) + }, + }, +}; +``` + +Most of the time the data returned from the extension's endpoint is not in the format +the UI needs. We must format the data before setting the collapsed data in the base component. + +If the computed property `summary` can rely on `collapsedData`, you can format the data +when `fetchFullData` is invoked: + +```javascript +export default { + //... + enablePolling: true + methods: { + fetchCollapsedData() { + return axios.get(this.reportPath) + }, + fetchFullData() { + return Promise.resolve(this.prepareReports()); + }, + // custom method + prepareReports() { + // unpack values from collapsedData + const { new_errors, existing_errors, resolved_errors } = this.collapsedData; + + // perform data formatting + + return [...newErrors, ...existingErrors, ...resolvedErrors] + } + }, +}; +``` + +If the extension relies on `collapsedData` being formatted before invoking `fetchFullData()`, +then `fetchCollapsedData()` must return the Axios response as well as the formatted data: + +```javascript +export default { + //... + enablePolling: true + methods: { + fetchCollapsedData() { + return axios.get(this.reportPath).then(res => { + const formattedData = this.prepareReports(res.data) + + return { + ...res, + data: formattedData, + } + }) + }, + // Custom method + prepareReports() { + // Unpack values from collapsedData + const { new_errors, existing_errors, resolved_errors } = this.collapsedData; + + // Perform data formatting + + return [...newErrors, ...existingErrors, ...resolvedErrors] + } + }, +}; +``` + +### Errors If `fetchCollapsedData()` or `fetchFullData()` methods throw an error: -- The loading state of the extension is updated to `LOADING_STATES.collapsedError` and `LOADING_STATES.expandedError` - respectively. +- The loading state of the extension is updated to `LOADING_STATES.collapsedError` + and `LOADING_STATES.expandedError` respectively. - The extensions header displays an error icon and updates the text to be either: - The text defined in `$options.i18n.error`. - "Failed to load" if `$options.i18n.error` is not defined. - The error is sent to Sentry to log that it occurred. -To customise the error text, you need to add it to the `i18n` object in your extension: +To customise the error text, add it to the `i18n` object in your extension: ```javascript export default { @@ -77,3 +243,77 @@ export default { }, }; ``` + +## Icons + +Level 1 and all subsequent levels can have their own status icons. To keep with +the design framework, import the `EXTENSION_ICONS` constant +from the `constants.js` file: + +```javascript +import { EXTENSION_ICONS } from '~/vue_merge_request_widget/constants.js'; +``` + +This constant has the below icons available for use. Per the design framework, +only some of these icons should be used on level 1: + +- `failed` +- `warning` +- `success` +- `neutral` +- `error` +- `notice` +- `severityCritical` +- `severityHigh` +- `severityMedium` +- `severityLow` +- `severityInfo` +- `severityUnknown` + +## Text styling + +Any area that has text can be styled with the placeholders below. This +technique follows the same technique as `sprintf`. However, instead of specifying +these through `sprintf`, the extension does this automatically. + +Every placeholder contains starting and ending tags. For example, `success` uses +`Hello %{success_start}world%{success_end}`. The extension then +adds the start and end tags with the correct styling classes. + +| Placeholder | Style | +|---|---| +| success | `gl-font-weight-bold gl-text-green-500` | +| danger | `gl-font-weight-bold gl-text-red-500` | +| critical | `gl-font-weight-bold gl-text-red-800` | +| same | `gl-font-weight-bold gl-text-gray-700` | +| strong | `gl-font-weight-bold` | +| small | `gl-font-sm` | + +## Action buttons + +You can add action buttons to all level 1 and 2 in each extension. These buttons +are meant as a way to provide links or actions for each row: + +- Action buttons for level 1 can be set through the `tertiaryButtons` computed property. + This property should return an array of objects for each action button. +- Action buttons for level 2 can be set by adding the `actions` key to the level 2 rows object. + The value for this key must also be an array of objects for each action button. + +Links must follow this structure: + +```javascript +{ + text: 'Click me', + href: this.someLinkHref, + target: '_blank', // Optional +} +``` + +For internal action buttons, follow this structure: + +```javascript +{ + text: 'Click me', + onClick() {} +} +``` diff --git a/doc/development/permissions.md b/doc/development/permissions.md index 72e20d31cf8..a5d211a5d2e 100644 --- a/doc/development/permissions.md +++ b/doc/development/permissions.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md index a9c68905095..f3a4f47eb22 100644 --- a/doc/development/pipelines.md +++ b/doc/development/pipelines.md @@ -12,7 +12,7 @@ which itself includes files under [`.gitlab/ci/`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/.gitlab/ci) for easier maintenance. -We're striving to [dogfood](https://about.gitlab.com/handbook/engineering/#dogfooding) +We're striving to [dogfood](https://about.gitlab.com/handbook/engineering/principles/#dogfooding) GitLab [CI/CD features and best-practices](../ci/yaml/index.md) as much as possible. @@ -90,6 +90,13 @@ In addition, there are a few circumstances where we would always run the full Je - when any vendored JavaScript file is changed (i.e. `vendor/assets/javascripts/**/*`) - when any backend file is changed ([see the patterns list for details](https://gitlab.com/gitlab-org/gitlab/-/blob/3616946936c1adbd9e754c1bd06f86ba670796d8/.gitlab/ci/rules.gitlab-ci.yml#L205-216)) +### Fork pipelines + +We only run the minimal RSpec & Jest jobs for fork pipelines unless the `pipeline:run-all-rspec` +label is set on the MR. The goal is to reduce the CI minutes consumed by fork pipelines. + +See the [experiment issue](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/1170). + ## Fail-fast job in merge request pipelines To provide faster feedback when a merge request breaks existing tests, we are experimenting with a @@ -170,10 +177,23 @@ After that, the next pipeline uses the up-to-date `knapsack/report-master.json` ### Flaky tests +#### Automatic skipping of flaky tests + Tests that are [known to be flaky](testing_guide/flaky_tests.md#automatic-retries-and-flaky-tests-detection) are skipped unless the `$SKIP_FLAKY_TESTS_AUTOMATICALLY` variable is set to `false` or if the `~"pipeline:run-flaky-tests"` label is set on the MR. +See the [experiment issue](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/1069). + +#### Automatic retry of failing tests in a separate process + +When the `$RETRY_FAILED_TESTS_IN_NEW_PROCESS` variable is set to `true`, RSpec tests that failed are automatically retried once in a separate +RSpec process. The goal is to get rid of most side-effects from previous tests that may lead to a subsequent test failure. + +We keep track of retried tests in the `$RETRIED_TESTS_REPORT_FILE` file saved as artifact by the `rspec:flaky-tests-report` job. + +See the [experiment issue](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/1148). + ### Monitoring The GitLab test suite is [monitored](performance.md#rspec-profiling) for the `main` branch, and any branch @@ -194,7 +214,7 @@ If you want to force a Review App to be deployed regardless of your changes, you ## As-if-FOSS jobs The `* as-if-foss` jobs run the GitLab test suite "as if FOSS", meaning as if the jobs would run in the context -of the `gitlab-org/gitlab-foss` project. These jobs are only created in the following cases: +of `gitlab-org/gitlab-foss`. These jobs are only created in the following cases: - when the `pipeline:run-as-if-foss` label is set on the merge request - when the merge request is created in the `gitlab-org/security/gitlab` project @@ -203,13 +223,12 @@ of the `gitlab-org/gitlab-foss` project. These jobs are only created in the foll The `* as-if-foss` jobs are run in addition to the regular EE-context jobs. They have the `FOSS_ONLY='1'` variable set and get the `ee/` folder removed before the tests start running. -The intent is to ensure that a change doesn't introduce a failure after the `gitlab-org/gitlab` project is synced to -the `gitlab-org/gitlab-foss` project. +The intent is to ensure that a change doesn't introduce a failure after `gitlab-org/gitlab` is synced to `gitlab-org/gitlab-foss`. ## As-if-JH jobs The `* as-if-jh` jobs run the GitLab test suite "as if JiHu", meaning as if the jobs would run in the context -of [the `gitlab-jh/gitlab` project](jh_features_review.md). These jobs are only created in the following cases: +of [GitLab JH](jh_features_review.md). These jobs are only created in the following cases: - when the `pipeline:run-as-if-jh` label is set on the merge request - when the `pipeline:run-all-rspec` label is set on the merge request @@ -218,16 +237,18 @@ of [the `gitlab-jh/gitlab` project](jh_features_review.md). These jobs are only The `* as-if-jh` jobs are run in addition to the regular EE-context jobs. The `jh/` folder is added before the tests start running. -The intent is to ensure that a change doesn't introduce a failure after the `gitlab-org/gitlab` project is synced to -the `gitlab-jh/gitlab` project. +The intent is to ensure that a change doesn't introduce a failure after `gitlab-org/gitlab` is synced to [GitLab JH](https://jihulab.com/gitlab-cn/gitlab). ### Corresponding JH branch -You can create a corresponding JH branch on the `gitlab-jh/gitlab` project by +You can create a corresponding JH branch on [GitLab JH](https://jihulab.com/gitlab-cn/gitlab) by appending `-jh` to the branch name. If a corresponding JH branch is found, `* as-if-jh` jobs grab the `jh` folder from the respective branch, rather than from the default branch. +NOTE: +For now, CI will try to fetch the branch on the [GitLab JH mirror](https://gitlab.com/gitlab-org/gitlab-jh/gitlab), so it might take some time for the new JH branch to propagate to the mirror. + ## `undercover` RSpec test > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/74859) in GitLab 14.6. @@ -235,7 +256,7 @@ rather than from the default branch. The `rspec:undercoverage` job runs [`undercover`](https://rubygems.org/gems/undercover) to detect, and fail if any changes introduced in the merge request has zero coverage. -The `rsepc:undercoverage` job obtains coverage data from the `rspec:coverage` +The `rspec:undercoverage` job obtains coverage data from the `rspec:coverage` job. In the event of an emergency, or false positive from this job, add the diff --git a/doc/development/policies.md b/doc/development/policies.md index 61cc6e0edf5..c9e4fdb4350 100644 --- a/doc/development/policies.md +++ b/doc/development/policies.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -111,7 +111,7 @@ Each line represents a rule that was evaluated. There are a few things to note: Here you can see that the first four rules were evaluated `false` for which user and subject. For example, you can see in the last line that -the rule was activated because the user `john` had the Reporter [role](../user/permissions.md) on +the rule was activated because the user `john` had the Reporter role on `Project/4`. When a policy is asked whether a particular ability is allowed diff --git a/doc/development/product_qualified_lead_guide/index.md b/doc/development/product_qualified_lead_guide/index.md new file mode 100644 index 00000000000..f9d18bacecd --- /dev/null +++ b/doc/development/product_qualified_lead_guide/index.md @@ -0,0 +1,94 @@ +--- +stage: Growth +group: Conversion +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Product Qualified Lead (PQL) development guide + +The Product Qualified Lead (PQL) funnel connects our users with our team members. Read more about [PQL product principles](https://about.gitlab.com/handbook/product/product-principles/#product-qualified-leads-pqls). + +A hand-raise PQL is a user who requests to speak to sales from within the product. + +## Embed a hand-raise lead form + +[HandRaiseLeadButton](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/assets/javascripts/hand_raise_leads/hand_raise_lead/components/hand_raise_lead_button.vue) is a reusable component that adds a button and a hand-raise modal to any screen. + +You can import a hand-raise lead button the following way. + +```javascript +import HandRaiseLeadButton from 'ee/hand_raise_leads/hand_raise_lead/components/hand_raise_lead_button.vue'; + +export default { + components: { + HandRaiseLeadButton, +... +</script> + +<template> + +<hand-raise-lead-button /> + +``` + +The hand-raise lead form accepts the following parameters via provide or inject. + +```javascript + provide: { + user: { + namespaceId, + userName, + firstName, + lastName, + companyName, + glmContent, + }, + }, +``` + +### Monitor the lead location + +When embedding a new hand raise form, use a unique `glmContent` or `glm_content` field that is different to any existing values. + +We currently use the following `glm content` values: + +| glm_content value | Notes | +| ------ | ------ | +| discover-group-security | This value is used in the group security feature discovery page. | +| discover-group-security-pqltest | This value is used in the group security feature discovery page [experiment with 3 CTAs](https://gitlab.com/gitlab-org/gitlab/-/issues/349799). | +| discover-project-security | This value is used in the project security feature discovery page. | +| discover-project-security-pqltest | This value is used in the project security feature discovery page [experiment with 3 CTAs](https://gitlab.com/gitlab-org/gitlab/-/issues/349799). | +| group-billing | This value is used in the group billing page. | +| trial-status-show-group | This value is used in the top left nav when a namespace has an active trial. | + +### Test the component + +In a jest test, you may test the presence of the component. + +```javascript +expect(wrapper.findComponent(HandRaiseLeadButton).exists()).toBe(true); +``` + +## PQL lead flow + +The flow of a PQL lead is as follows: + +1. A user triggers a [`HandRaiseLeadButton` component](#embed-a-hand-raise-lead-form) on `gitlab.com`. +1. The `HandRaiseLeadButton` submits any information to the following API endpoint: `/-/trials/create_hand_raise_lead`. +1. That endpoint reposts the form to the CustomersDot `trials/create_hand_raise_lead` endpoint. +1. CustomersDot records the form data to the `leads` table and posts the form to [Platypus](https://gitlab.com/gitlab-com/business-technology/enterprise-apps/integrations/platypus). +1. Platypus posts the form to Workato (which is under the responsibility of the Business Operations team). +1. Workato sends the form to Marketo. +1. Marketo does scoring and sends the form to Salesforce. +1. Our Sales team uses Salesforce to connect to the leads. + +## Monitor and manually test leads + +- Check the application and Sidekiq logs on `gitlab.com` and CustomersDot to monitor leads. +- Check the `leads` table in CustomersDot. +- Set up staging credentials for Platypus, and track the leads on the [Platypus Dashboard](https://staging.ci.nexus.gitlabenvironment.cloud/admin/queues/queue/new-lead-queue). +- Ask for access to the Marketo Sandbox and validate the leads there. + +## Trials + +Trials follow the same flow as the PQL leads. diff --git a/doc/development/profiling.md b/doc/development/profiling.md index 789e0640933..deb743569c5 100644 --- a/doc/development/profiling.md +++ b/doc/development/profiling.md @@ -91,11 +91,6 @@ printer = RubyProf::CallStackPrinter.new(result) printer.print(File.open('/tmp/profile.html', 'w')) ``` -[GitLab-Profiler](https://gitlab.com/gitlab-com/gitlab-profiler) is a project -that builds on this to add some additional niceties, such as allowing -configuration with a single YAML file for multiple URLs, and uploading of the -profile and log output to S3. - ## Speedscope flamegraphs You can generate a flamegraph for a particular URL by selecting a flamegraph sampling mode button in the performance bar or by adding the `performance_bar=flamegraph` parameter to the request. diff --git a/doc/development/prometheus_metrics.md b/doc/development/prometheus_metrics.md index da6ba14cdd8..b3f259efc3d 100644 --- a/doc/development/prometheus_metrics.md +++ b/doc/development/prometheus_metrics.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/ruby_upgrade.md b/doc/development/ruby_upgrade.md index 2102a256645..a208a93e300 100644 --- a/doc/development/ruby_upgrade.md +++ b/doc/development/ruby_upgrade.md @@ -272,4 +272,4 @@ and merged back independently. - **Give yourself enough time to fix problems ahead of a milestone release.** GitLab moves fast. As a Ruby upgrade requires many MRs to be sent and reviewed, make sure all changes are merged at least a week before the 22nd. This gives us extra time to act if something breaks. If in doubt, it is better to -postpone the upgrade to the following month, as we [prioritize availability over velocity](https://about.gitlab.com/handbook/engineering/#prioritizing-technical-decisions). +postpone the upgrade to the following month, as we [prioritize availability over velocity](https://about.gitlab.com/handbook/engineering/principles/#prioritizing-technical-decisions). diff --git a/doc/development/scalability.md b/doc/development/scalability.md index 7a3f3c7097d..fe7063be0e8 100644 --- a/doc/development/scalability.md +++ b/doc/development/scalability.md @@ -45,7 +45,7 @@ many groups or projects, and the access level (including guest, developer, or maintainer) to groups and projects determines what users can see and what they can access. -Users with the administrator role can access all projects and even impersonate +Users with administrator access can access all projects and even impersonate users. #### Sharding and partitioning diff --git a/doc/development/secure_coding_guidelines.md b/doc/development/secure_coding_guidelines.md index 51d3338e5ed..d34b12c6361 100644 --- a/doc/development/secure_coding_guidelines.md +++ b/doc/development/secure_coding_guidelines.md @@ -184,13 +184,15 @@ and [possessive quantifiers](https://www.regular-expressions.info/possessive.htm - Avoid nested quantifiers if possible (for example `(a+)+`) - Try to be as precise as possible in your regex and avoid the `.` if there's an alternative - For example, Use `_[^_]+_` instead of `_.*_` to match `_text here_` +- Use reasonable ranges (for example, `{1,10}`) for repeating patterns instead of unbounded `*` and `+` matchers +- When possible, perform simple input validation such as maximum string length checks before using regular expressions - If in doubt, don't hesitate to ping `@gitlab-com/gl-security/appsec` #### Go Go's [`regexp`](https://pkg.go.dev/regexp) package uses `re2` and isn't vulnerable to backtracking issues. -## Further Links +### Further Links - [Rubular](https://rubular.com/) is a nice online tool to fiddle with Ruby Regexps. - [Runaway Regular Expressions](https://www.regular-expressions.info/catastrophic.html) diff --git a/doc/development/service_ping/implement.md b/doc/development/service_ping/implement.md index c32789740c3..3a1e4c6d87b 100644 --- a/doc/development/service_ping/implement.md +++ b/doc/development/service_ping/implement.md @@ -793,6 +793,8 @@ The Service Ping JSON payload for GitLab.com is shared in the You may also use the [Service Ping QA dashboard](https://app.periscopedata.com/app/gitlab/632033/Usage-Ping-QA) to check how well your metric performs. The dashboard allows filtering by GitLab version, by "Self-managed" and "SaaS", and shows you how many failures have occurred for each metric. Whenever you notice a high failure rate, you can re-optimize your metric. +Use [Metrics Dictionary](https://metrics.gitlab.com/) [copy query to clipboard feature](https://www.youtube.com/watch?v=n4o65ivta48&list=PL05JrBw4t0Krg3mbR6chU7pXtMt_es6Pb) to get a query ready to run in Sisense for a specific metric. + ## Set up and test Service Ping locally To set up Service Ping locally, you must: diff --git a/doc/development/service_ping/index.md b/doc/development/service_ping/index.md index 315ff2b090c..86e70cc8bbc 100644 --- a/doc/development/service_ping/index.md +++ b/doc/development/service_ping/index.md @@ -84,7 +84,7 @@ Registration is not yet required for participation, but will be added in a futur #### Enable Registration Features -1. Sign in as a user with the [Administrator](../../user/permissions.md) role. +1. Sign in as a user with administrator access. 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Metrics and profiling**. 1. Expand the **Usage statistics** section. @@ -96,7 +96,7 @@ Registration is not yet required for participation, but will be added in a futur You can view the exact JSON payload sent to GitLab Inc. in the Admin Area. To view the payload: -1. Sign in as a user with the [Administrator](../../user/permissions.md) role. +1. Sign in as a user with administrator access. 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Metrics and profiling**. 1. Expand the **Usage statistics** section. @@ -118,7 +118,7 @@ configuration file. To disable Service Ping in the GitLab UI: -1. Sign in as a user with the [Administrator](../../user/permissions.md) role. +1. Sign in as a user with administrator access. 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Metrics and profiling**. 1. Expand the **Usage statistics** section. @@ -198,13 +198,32 @@ sequenceDiagram ## How Service Ping works 1. The Service Ping [cron job](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/gitlab_service_ping_worker.rb#L24) is set in Sidekiq to run weekly. -1. When the cron job runs, it calls [`Gitlab::UsageData.to_json`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/service_ping/submit_service.rb#L49). -1. `Gitlab::UsageData.to_json` [cascades down](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb) to ~400+ other counter method calls. -1. The response of all methods calls are [merged together](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb#L68) into a single JSON payload in `Gitlab::UsageData.to_json`. +1. When the cron job runs, it calls [`Gitlab::Usage::ServicePingReport.for(output: :all_metrics_values)`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/service_ping/submit_service.rb). +1. `Gitlab::Usage::ServicePingReport.for(output: :all_metrics_values)` [cascades down](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb) to ~400+ other counter method calls. +1. The response of all methods calls are [merged together](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb#L68) into a single JSON payload. 1. The JSON payload is then [posted to the Versions application](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/service_ping/submit_service.rb#L20) If a firewall exception is needed, the required URL depends on several things. If the hostname is `version.gitlab.com`, the protocol is `TCP`, and the port number is `443`, the required URL is <https://version.gitlab.com/>. +1. In case of an error, it will be reported to the Version application along with following pieces of information: + +- `uuid` - GitLab instance unique identifier +- `hostname` - GitLab instance hostname +- `version` - GitLab instance current versions +- `elapsed` - Amount of time which passed since Service Ping report process started and moment of error occurrence +- `message` - Error message + +<pre> +<code> +{ + "uuid"=>"02333324-1cd7-4c3b-a45b-a4993f05fb1d", + "hostname"=>"127.0.0.1", + "version"=>"14.7.0-pre", + "elapsed"=>0.006946, + "message"=>'PG::UndefinedColumn: ERROR: column \"non_existent_attribute\" does not exist\nLINE 1: SELECT COUNT(non_existent_attribute) FROM \"issues\" /*applica...' +} +</code> +</pre> ### On a Geo secondary site @@ -510,7 +529,7 @@ To generate Service Ping, use [Teleport](https://goteleport.com/docs/) or a deta ### Verification (After approx 30 hours) -#### Verify with a detached screen session +#### Verify with Teleport 1. Follow [the steps](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/Teleport/Connect_to_Rails_Console_via_Teleport.md#how-to-use-teleport-to-connect-to-rails-console) to request a new access to the required environment and connect to the Rails console 1. Check the last payload in `raw_usage_data` table: `RawUsageData.last.payload` @@ -557,6 +576,28 @@ skip_db_write: ServicePing::SubmitService.new(skip_db_write: true).execute ``` +## Manually upload Service Ping payload + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/7388) in GitLab 14.8 with a flag named `admin_application_settings_service_usage_data_center`. Disabled by default. + +Service Ping payload can be uploaded to GitLab even if your application instance doesn't have access to the internet, +or you don't have Service Ping [cron job](#how-service-ping-works) enabled. + +To upload payload manually: + +1. Sign in as a user with administrator access. +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Settings > Service** usage data. +1. Select **Download payload**. +1. Save the JSON file. +1. Visit [Service usage data center](https://version.gitlab.com/usage_data/new). +1. Select **Choose file** and choose the file from p5. +1. Select **Upload**. + +## Monitoring + +Service Ping reporting process state is monitored with [internal SiSense dashboard](https://app.periscopedata.com/app/gitlab/968489/Product-Intelligence---Service-Ping-Health). + ## Troubleshooting ### Cannot disable Service Ping using the configuration file diff --git a/doc/development/service_ping/metrics_dictionary.md b/doc/development/service_ping/metrics_dictionary.md index 808c5064cf3..93eec4efabd 100644 --- a/doc/development/service_ping/metrics_dictionary.md +++ b/doc/development/service_ping/metrics_dictionary.md @@ -6,8 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Metrics Dictionary Guide -[Service Ping](index.md) metrics are defined in the -[Metrics Dictionary](https://metrics.gitlab.com/index.html). +[Service Ping](index.md) metrics are defined in individual YAML files definitions from which the +[Metrics Dictionary](https://metrics.gitlab.com/) is built. This guide describes the dictionary and how it's implemented. ## Metrics Definition and validation @@ -39,7 +39,7 @@ Each metric is defined in a separate YAML file consisting of a number of fields: | `product_group` | yes | The [group](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml) that owns the metric. | | `product_category` | no | The [product category](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/categories.yml) for the metric. | | `value_type` | yes | `string`; one of [`string`, `number`, `boolean`, `object`](https://json-schema.org/understanding-json-schema/reference/type.html). | -| `status` | yes | `string`; [status](#metric-statuses) of the metric, may be set to `active`, `deprecated`, `removed`, `broken`. | +| `status` | yes | `string`; [status](#metric-statuses) of the metric, may be set to `active`, `removed`, `broken`. | | `time_frame` | yes | `string`; may be set to a value like `7d`, `28d`, `all`, `none`. | | `data_source` | yes | `string`; may be set to a value like `database`, `redis`, `redis_hll`, `prometheus`, `system`. | | `data_category` | yes | `string`; [categories](#data-category) of the metric, may be set to `operational`, `optional`, `subscription`, `standard`. The default value is `optional`.| @@ -49,7 +49,7 @@ Each metric is defined in a separate YAML file consisting of a number of fields: | `tier` | yes | `array`; may contain one or a combination of `free`, `premium` or `ultimate`. The [tier]( https://about.gitlab.com/handbook/marketing/strategic-marketing/tiers/) where the tracked feature is available. This should be verbose and contain all tiers where a metric is available. | | `milestone` | no | The milestone when the metric is introduced. | | `milestone_removed` | no | The milestone when the metric is removed. | -| `introduced_by_url` | no | The URL to the Merge Request that introduced the metric. | +| `introduced_by_url` | no | The URL to the merge request that introduced the metric. | | `repair_issue_url` | no | The URL of the issue that was created to repair a metric with a `broken` status. | | `options` | no | `object`: options information needed to calculate the metric value. | | `skip_validation` | no | This should **not** be set. [Used for imported metrics until we review, update and make them valid](https://gitlab.com/groups/gitlab-org/-/epics/5425). | @@ -60,7 +60,6 @@ Metric definitions can have one of the following statuses: - `active`: Metric is used and reports data. - `broken`: Metric reports broken data (for example, -1 fallback), or does not report data at all. A metric marked as `broken` must also have the `repair_issue_url` attribute. -- `deprecated`: Metric is deprecated and possibly planned to be removed. - `removed`: Metric was removed, but it may appear in Service Ping payloads sent from instances running on older versions of GitLab. ### Metric value_type @@ -194,7 +193,7 @@ tier: - ultimate ``` -## Create a new metric definition +### Create a new metric definition The GitLab codebase provides a dedicated [generator](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/generators/gitlab/usage_metric_definition_generator.rb) to create new metric definitions. @@ -229,7 +228,7 @@ bundle exec rails generate gitlab:usage_metric_definition counts.issues --ee --d create ee/config/metrics/counts_7d/issues.yml ``` -## Metrics added dynamic to Service Ping payload +### Metrics added dynamic to Service Ping payload The [Redis HLL metrics](implement.md#known-events-are-added-automatically-in-service-data-payload) are added automatically to Service Ping payload. @@ -250,3 +249,13 @@ bundle exec rails generate gitlab:usage_metric_definition:redis_hll issues users create config/metrics/counts_7d/i_closed_weekly.yml create config/metrics/counts_28d/i_closed_monthly.yml ``` + +## Metrics Dictionary + +[Metrics Dictionary is a separate application](https://gitlab.com/gitlab-org/growth/product-intelligence/metric-dictionary). + +All metrics available in Service Ping are in the [Metrics Dictionary](https://metrics.gitlab.com/). + +### Copy query to clipboard + +To check if a metric has data in Sisense, use the copy query to clipboard feature. This copies a query that's ready to use in Sisense. The query gets the last five service ping data for GitLab.com for a given metric. For information about how to check if a Service Ping metric has data in Sisense, see this [demo](https://www.youtube.com/watch?v=n4o65ivta48). diff --git a/doc/development/service_ping/metrics_lifecycle.md b/doc/development/service_ping/metrics_lifecycle.md index ebfab6341e9..a7ecf15a493 100644 --- a/doc/development/service_ping/metrics_lifecycle.md +++ b/doc/development/service_ping/metrics_lifecycle.md @@ -14,6 +14,11 @@ Follow the [Implement Service Ping](implement.md) guide. ## Change an existing metric +See [this video tutorial](https://youtu.be/bYf3c01KCls) for help with the update of metric attributes. + +NOTE: +The `key_path` attribute represents the location of the metric in Service Ping payload and must not be changed. + Because we do not control when customers update their self-managed instances of GitLab, we **STRONGLY DISCOURAGE** changes to the logic used to calculate any metric. Any such changes lead to inconsistent reports from multiple GitLab instances. @@ -85,10 +90,9 @@ To remove a metric: 1. Verify that removing the metric from the Service Ping payload does not cause errors in [Version App](https://gitlab.com/gitlab-services/version-gitlab-com) when the updated payload is collected and processed. Version App collects - and persists all Service Ping reports. To do that you can modify - [fixtures](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/master/spec/support/usage_data_helpers.rb#L540) - used to test - [`UsageDataController#create`](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/3760ef28/spec/controllers/usage_data_controller_spec.rb#L75) + and persists all Service Ping reports. To verify Service Ping processing in your local development environment, follow this [guide](https://www.youtube.com/watch?v=FS5emplabRU). + Alternatively, you can modify [fixtures](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/master/spec/support/usage_data_helpers.rb#L540) + used to test the [`UsageDataController#create`](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/3760ef28/spec/controllers/usage_data_controller_spec.rb#L75) endpoint, and assure that test suite does not fail when metric that you wish to remove is not included into test payload. 1. Create an issue in the diff --git a/doc/development/service_ping/review_guidelines.md b/doc/development/service_ping/review_guidelines.md index eb64d460b5a..137e11608cf 100644 --- a/doc/development/service_ping/review_guidelines.md +++ b/doc/development/service_ping/review_guidelines.md @@ -14,7 +14,7 @@ general best practices for code reviews, refer to our [code review guide](../cod ## Resources for reviewers - [Service Ping Guide](index.md) -- [Metrics Dictionary](https://metrics.gitlab.com/index.html) +- [Metrics Dictionary](https://metrics.gitlab.com/) ## Review process diff --git a/doc/development/service_ping/troubleshooting.md b/doc/development/service_ping/troubleshooting.md new file mode 100644 index 00000000000..770b6650764 --- /dev/null +++ b/doc/development/service_ping/troubleshooting.md @@ -0,0 +1,31 @@ +--- +stage: Growth +group: Product Intelligence +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Troubleshooting + +## Service Ping Payload drop + +### Symptoms + +You will be alerted by a [Sisense alert](https://app.periscopedata.com/app/gitlab/alert/Service-ping-payload-drop-Alert/5a5b8766b8af43b4a75d6e9c684a365f/edit) that is sent to `#g_product_intelligence` Slack channel + +### Locating the problem + +First you need to identify at which stage in Service Ping data pipeline the drop is occurring. + +Start at [Service Ping Health Dashboard](https://app.periscopedata.com/app/gitlab/968489/Product-Intelligence---Service-Ping-Health) on Sisense. + +The alert compares the current daily value with the daily value from previous week, excluding the last 48 hours as this is the interval where we export the data in GCP and get it into DWH. + +You can use [this query](https://gitlab.com/gitlab-org/gitlab/-/issues/347298#note_836685350) as an example, to start detecting when the drop started. + +### Troubleshooting GitLab application layer + +For results about an investigation conducted into an unexpected drop in Service ping Payload events volume, see [this issue](https://gitlab.com/gitlab-data/analytics/-/issues/11071). + +### Troubleshooting data warehouse layer + +Reach out to the [Data team](https://about.gitlab.com/handbook/business-technology/data-team) to ask about current state of data warehouse. On their handbook page there is a [section with contact details](https://about.gitlab.com/handbook/business-technology/data-team/#how-to-connect-with-us). diff --git a/doc/development/sidekiq/compatibility_across_updates.md b/doc/development/sidekiq/compatibility_across_updates.md new file mode 100644 index 00000000000..919f6935139 --- /dev/null +++ b/doc/development/sidekiq/compatibility_across_updates.md @@ -0,0 +1,159 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Sidekiq Compatibility across Updates + +The arguments for a Sidekiq job are stored in a queue while it is +scheduled for execution. During a online update, this could lead to +several possible situations: + +1. An older version of the application publishes a job, which is executed by an + upgraded Sidekiq node. +1. A job is queued before an upgrade, but executed after an upgrade. +1. A job is queued by a node running the newer version of the application, but + executed on a node running an older version of the application. + +## Adding new workers + +On GitLab.com, we [do not currently have a Sidekiq deployment in the +canary stage](https://gitlab.com/gitlab-org/gitlab/-/issues/19239). This +means that a new worker than can be scheduled from an HTTP endpoint may +be scheduled from canary but not run on Sidekiq until the full +production deployment is complete. This can be several hours later than +scheduling the job. For some workers, this will not be a problem. For +others - particularly [latency-sensitive +jobs](worker_attributes.md#latency-sensitive-jobs) - this will result in a poor user +experience. + +This only applies to new worker classes when they are first introduced. +As we recommend [using feature flags](../feature_flags/) as a general +development process, it's best to control the entire change (including +scheduling of the new Sidekiq worker) with a feature flag. + +## Changing the arguments for a worker + +Jobs need to be backward and forward compatible between consecutive versions +of the application. Adding or removing an argument may cause problems +during deployment before all Rails and Sidekiq nodes have the updated code. + +### Deprecate and remove an argument + +**Before you remove arguments from the `perform_async` and `perform` methods.**, deprecate them. The +following example deprecates and then removes `arg2` from the `perform_async` method: + +1. Provide a default value (usually `nil`) and use a comment to mark the + argument as deprecated in the coming minor release. (Release M) + + ```ruby + class ExampleWorker + # Keep arg2 parameter for backwards compatibility. + def perform(object_id, arg1, arg2 = nil) + # ... + end + end + ``` + +1. One minor release later, stop using the argument in `perform_async`. (Release M+1) + + ```ruby + ExampleWorker.perform_async(object_id, arg1) + ``` + +1. At the next major release, remove the value from the worker class. (Next major release) + + ```ruby + class ExampleWorker + def perform(object_id, arg1) + # ... + end + end + ``` + +### Add an argument + +There are two options for safely adding new arguments to Sidekiq workers: + +1. Set up a [multi-step deployment](#multi-step-deployment) in which the new argument is first added to the worker. +1. Use a [parameter hash](#parameter-hash) for additional arguments. This is perhaps the most flexible option. + +#### Multi-step deployment + +This approach requires multiple releases. + +1. Add the argument to the worker with a default value (Release M). + + ```ruby + class ExampleWorker + def perform(object_id, new_arg = nil) + # ... + end + end + ``` + +1. Add the new argument to all the invocations of the worker (Release M+1). + + ```ruby + ExampleWorker.perform_async(object_id, new_arg) + ``` + +1. Remove the default value (Release M+2). + + ```ruby + class ExampleWorker + def perform(object_id, new_arg) + # ... + end + end + ``` + +#### Parameter hash + +This approach doesn't require multiple releases if an existing worker already +uses a parameter hash. + +1. Use a parameter hash in the worker to allow future flexibility. + + ```ruby + class ExampleWorker + def perform(object_id, params = {}) + # ... + end + end + ``` + +## Removing workers + +Try to avoid removing workers and their queues in minor and patch +releases. + +During online update instance can have pending jobs and removing the queue can +lead to those jobs being stuck forever. If you can't write migration for those +Sidekiq jobs, please consider removing the worker in a major release only. + +## Renaming queues + +For the same reasons that removing workers is dangerous, care should be taken +when renaming queues. + +When renaming queues, use the `sidekiq_queue_migrate` helper migration method +in a **post-deployment migration**: + +```ruby +class MigrateTheRenamedSidekiqQueue < Gitlab::Database::Migration[1.0] + def up + sidekiq_queue_migrate 'old_queue_name', to: 'new_queue_name' + end + + def down + sidekiq_queue_migrate 'new_queue_name', to: 'old_queue_name' + end +end + +``` + +You must rename the queue in a post-deployment migration not in a normal +migration. Otherwise, it runs too early, before all the workers that +schedule these jobs have stopped running. See also [other examples](../post_deployment_migrations.md#use-cases). diff --git a/doc/development/sidekiq/idempotent_jobs.md b/doc/development/sidekiq/idempotent_jobs.md new file mode 100644 index 00000000000..4b201e22ca9 --- /dev/null +++ b/doc/development/sidekiq/idempotent_jobs.md @@ -0,0 +1,208 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Sidekiq idempotent jobs + +It's known that a job can fail for multiple reasons. For example, network outages or bugs. +In order to address this, Sidekiq has a built-in retry mechanism that is +used by default by most workers within GitLab. + +It's expected that a job can run again after a failure without major side-effects for the +application or users, which is why Sidekiq encourages +jobs to be [idempotent and transactional](https://github.com/mperham/sidekiq/wiki/Best-Practices#2-make-your-job-idempotent-and-transactional). + +As a general rule, a worker can be considered idempotent if: + +- It can safely run multiple times with the same arguments. +- Application side-effects are expected to happen only once + (or side-effects of a second run do not have an effect). + +A good example of that would be a cache expiration worker. + +A job scheduled for an idempotent worker is [deduplicated](#deduplication) when +an unstarted job with the same arguments is already in the queue. + +## Ensuring a worker is idempotent + +Make sure the worker tests pass using the following shared example: + +```ruby +include_examples 'an idempotent worker' do + it 'marks the MR as merged' do + # Using subject inside this block will process the job multiple times + subject + + expect(merge_request.state).to eq('merged') + end +end +``` + +Use the `perform_multiple` method directly instead of `job.perform` (this +helper method is automatically included for workers). + +## Declaring a worker as idempotent + +```ruby +class IdempotentWorker + include ApplicationWorker + + # Declares a worker is idempotent and can + # safely run multiple times. + idempotent! + + # ... +end +``` + +It's encouraged to only have the `idempotent!` call in the top-most worker class, even if +the `perform` method is defined in another class or module. + +If the worker class isn't marked as idempotent, a cop fails. Consider skipping +the cop if you're not confident your job can safely run multiple times. + +## Deduplication + +When a job for an idempotent worker is enqueued while another +unstarted job is already in the queue, GitLab drops the second +job. The work is skipped because the same work would be +done by the job that was scheduled first; by the time the second +job executed, the first job would do nothing. + +### Strategies + +GitLab supports two deduplication strategies: + +- `until_executing` +- `until_executed` + +More [deduplication strategies have been +suggested](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/195). If +you are implementing a worker that could benefit from a different +strategy, please comment in the issue. + +#### Until Executing + +This strategy takes a lock when a job is added to the queue, and removes that lock before the job starts. + +For example, `AuthorizedProjectsWorker` takes a user ID. When the +worker runs, it recalculates a user's authorizations. GitLab schedules +this job each time an action potentially changes a user's +authorizations. If the same user is added to two projects at the +same time, the second job can be skipped if the first job hasn't +begun, because when the first job runs, it creates the +authorizations for both projects. + +```ruby +module AuthorizedProjectUpdate + class UserRefreshOverUserRangeWorker + include ApplicationWorker + + deduplicate :until_executing + idempotent! + + # ... + end +end +``` + +#### Until Executed + +This strategy takes a lock when a job is added to the queue, and removes that lock after the job finishes. +It can be used to prevent jobs from running simultaneously multiple times. + +```ruby +module Ci + class BuildTraceChunkFlushWorker + include ApplicationWorker + + deduplicate :until_executed + idempotent! + + # ... + end +end +``` + +Also, you can pass `if_deduplicated: :reschedule_once` option to re-run a job once after +the currently running job finished and deduplication happened at least once. +This ensures that the latest result is always produced even if a race condition +happened. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/342123) for more information. + +### Scheduling jobs in the future + +GitLab doesn't skip jobs scheduled in the future, as we assume that +the state has changed by the time the job is scheduled to +execute. Deduplication of jobs scheduled in the feature is possible +for both `until_executed` and `until_executing` strategies. + +If you do want to deduplicate jobs scheduled in the future, +this can be specified on the worker by passing `including_scheduled: true` argument +when defining deduplication strategy: + +```ruby +module AuthorizedProjectUpdate + class UserRefreshOverUserRangeWorker + include ApplicationWorker + + deduplicate :until_executing, including_scheduled: true + idempotent! + + # ... + end +end +``` + +## Setting the deduplication time-to-live (TTL) + +Deduplication depends on an idempotency key that is stored in Redis. This is normally +cleared by the configured deduplication strategy. + +However, the key can remain until its TTL in certain cases like: + +1. `until_executing` is used but the job was never enqueued or executed after the Sidekiq + client middleware was run. + +1. `until_executed` is used but the job fails to finish due to retry exhaustion, gets + interrupted the maximum number of times, or gets lost. + +The default value is 6 hours. During this time, jobs won't be enqueued even if the first +job never executed or finished. + +The TTL can be configured with: + +```ruby +class ProjectImportScheduleWorker + include ApplicationWorker + + idempotent! + deduplicate :until_executing, ttl: 5.minutes +end +``` + +Duplicate jobs can happen when the TTL is reached, so make sure you lower this only for jobs +that can tolerate some duplication. + +### Preserve the latest WAL location for idempotent jobs + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69372) in GitLab 14.3. +> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/338350) in GitLab 14.4. +> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/338350) in GitLab 14.6. + +The deduplication always take into account the latest binary replication pointer, not the first one. +This happens because we drop the same job scheduled for the second time and the Write-Ahead Log (WAL) is lost. +This could lead to comparing the old WAL location and reading from a stale replica. + +To support both deduplication and maintaining data consistency with load balancing, +we are preserving the latest WAL location for idempotent jobs in Redis. +This way we are always comparing the latest binary replication pointer, +making sure that we read from the replica that is fully caught up. + +FLAG: +On self-managed GitLab, by default this feature is available. To hide the feature, ask an administrator to +[disable the feature flag](../../administration/feature_flags.md) named `preserve_latest_wal_locations_for_idempotent_jobs`. + +This feature flag is related to GitLab development and is not intended to be used by GitLab administrators, though. +On GitLab.com, this feature is available. diff --git a/doc/development/sidekiq/index.md b/doc/development/sidekiq/index.md new file mode 100644 index 00000000000..c9906c4c768 --- /dev/null +++ b/doc/development/sidekiq/index.md @@ -0,0 +1,190 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Sidekiq guides + +We use [Sidekiq](https://github.com/mperham/sidekiq) as our background +job processor. These guides are for writing jobs that will work well on +GitLab.com and be consistent with our existing worker classes. For +information on administering GitLab, see [configuring Sidekiq](../../administration/sidekiq.md). + +There are pages with additional detail on the following topics: + +1. [Compatibility across updates](compatibility_across_updates.md) +1. [Job idempotency and job deduplication](idempotent_jobs.md) +1. [Limited capacity worker: continuously performing work with a specified concurrency](limited_capacity_worker.md) +1. [Logging](logging.md) +1. [Worker attributes](worker_attributes.md) + 1. **Job urgency** specifies queuing and execution SLOs + 1. **Resource boundaries** and **external dependencies** for describing the workload + 1. **Feature categorization** + 1. **Database load balancing** + +## ApplicationWorker + +All workers should include `ApplicationWorker` instead of `Sidekiq::Worker`, +which adds some convenience methods and automatically sets the queue based on +the [routing rules](../../administration/operations/extra_sidekiq_routing.md#queue-routing-rules). + +## Retries + +Sidekiq defaults to using [25 +retries](https://github.com/mperham/sidekiq/wiki/Error-Handling#automatic-job-retry), +with back-off between each retry. 25 retries means that the last retry +would happen around three weeks after the first attempt (assuming all 24 +prior retries failed). + +For most workers - especially [idempotent workers](idempotent_jobs.md) - +the default of 25 retries is more than sufficient. Many of our older +workers declare 3 retries, which used to be the default within the +GitLab application. 3 retries happen over the course of a couple of +minutes, so the jobs are prone to failing completely. + +A lower retry count may be applicable if any of the below apply: + +1. The worker contacts an external service and we do not provide + guarantees on delivery. For example, webhooks. +1. The worker is not idempotent and running it multiple times could + leave the system in an inconsistent state. For example, a worker that + posts a system note and then performs an action: if the second step + fails and the worker retries, the system note will be posted again. +1. The worker is a cronjob that runs frequently. For example, if a cron + job runs every hour, then we don't need to retry beyond an hour + because we don't need two of the same job running at once. + +Each retry for a worker is counted as a failure in our metrics. A worker +which always fails 9 times and succeeds on the 10th would have a 90% +error rate. + +## Sidekiq Queues + +Previously, each worker had its own queue, which was automatically set based on the +worker class name. For a worker named `ProcessSomethingWorker`, the queue name +would be `process_something`. You can now route workers to a specific queue using +[queue routing rules](../../administration/operations/extra_sidekiq_routing.md#queue-routing-rules). +In GDK, new workers are routed to a queue named `default`. + +If you're not sure what queue a worker uses, +you can find it using `SomeWorker.queue`. There is almost never a reason to +manually override the queue name using `sidekiq_options queue: :some_queue`. + +After adding a new worker, run `bin/rake +gitlab:sidekiq:all_queues_yml:generate` to regenerate +`app/workers/all_queues.yml` or `ee/app/workers/all_queues.yml` so that +it can be picked up by +[`sidekiq-cluster`](../../administration/operations/extra_sidekiq_processes.md) +in installations that don't use routing rules. To learn more about potential changes, +read [Use routing rules by default and deprecate queue selectors for self-managed](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/596). + +Additionally, run +`bin/rake gitlab:sidekiq:sidekiq_queues_yml:generate` to regenerate +`config/sidekiq_queues.yml`. + +## Queue Namespaces + +While different workers cannot share a queue, they can share a queue namespace. + +Defining a queue namespace for a worker makes it possible to start a Sidekiq +process that automatically handles jobs for all workers in that namespace, +without needing to explicitly list all their queue names. If, for example, all +workers that are managed by `sidekiq-cron` use the `cronjob` queue namespace, we +can spin up a Sidekiq process specifically for these kinds of scheduled jobs. +If a new worker using the `cronjob` namespace is added later on, the Sidekiq +process also picks up jobs for that worker (after having been restarted), +without the need to change any configuration. + +A queue namespace can be set using the `queue_namespace` DSL class method: + +```ruby +class SomeScheduledTaskWorker + include ApplicationWorker + + queue_namespace :cronjob + + # ... +end +``` + +Behind the scenes, this sets `SomeScheduledTaskWorker.queue` to +`cronjob:some_scheduled_task`. Commonly used namespaces have their own +concern module that can easily be included into the worker class, and that may +set other Sidekiq options besides the queue namespace. `CronjobQueue`, for +example, sets the namespace, but also disables retries. + +`bundle exec sidekiq` is namespace-aware, and listens on all +queues in a namespace (technically: all queues prefixed with the namespace name) +when a namespace is provided instead of a simple queue name in the `--queue` +(`-q`) option, or in the `:queues:` section in `config/sidekiq_queues.yml`. + +Note that adding a worker to an existing namespace should be done with care, as +the extra jobs take resources away from jobs from workers that were already +there, if the resources available to the Sidekiq process handling the namespace +are not adjusted appropriately. + +## Versioning + +Version can be specified on each Sidekiq worker class. +This is then sent along when the job is created. + +```ruby +class FooWorker + include ApplicationWorker + + version 2 + + def perform(*args) + if job_version == 2 + foo = args.first['foo'] + else + foo = args.first + end + end +end +``` + +Under this schema, any worker is expected to be able to handle any job that was +enqueued by an older version of that worker. This means that when changing the +arguments a worker takes, you must increment the `version` (or set `version 1` +if this is the first time a worker's arguments are changing), but also make sure +that the worker is still able to handle jobs that were queued with any earlier +version of the arguments. From the worker's `perform` method, you can read +`self.job_version` if you want to specifically branch on job version, or you +can read the number or type of provided arguments. + +## Job size + +GitLab stores Sidekiq jobs and their arguments in Redis. To avoid +excessive memory usage, we compress the arguments of Sidekiq jobs +if their original size is bigger than 100KB. + +After compression, if their size still exceeds 5MB, it raises an +[`ExceedLimitError`](https://gitlab.com/gitlab-org/gitlab/-/blob/f3dd89e5e510ea04b43ffdcb58587d8f78a8d77c/lib/gitlab/sidekiq_middleware/size_limiter/exceed_limit_error.rb#L8) +error when scheduling the job. + +If this happens, rely on other means of making the data +available in Sidekiq. There are possible workarounds such as: + +- Rebuild the data in Sidekiq with data loaded from the database or + elsewhere. +- Store the data in [object storage](../file_storage.md#object-storage) + before scheduling the job, and retrieve it inside the job. + +## Job weights + +Some jobs have a weight declared. This is only used when running Sidekiq +in the default execution mode - using +[`sidekiq-cluster`](../../administration/operations/extra_sidekiq_processes.md) +does not account for weights. + +As we are [moving towards using `sidekiq-cluster` in +Free](https://gitlab.com/gitlab-org/gitlab/-/issues/34396), newly-added +workers do not need to have weights specified. They can use the +default weight, which is 1. + +## Tests + +Each Sidekiq worker must be tested using RSpec, just like any other class. These +tests should be placed in `spec/workers`. diff --git a/doc/development/sidekiq/limited_capacity_worker.md b/doc/development/sidekiq/limited_capacity_worker.md new file mode 100644 index 00000000000..5b86e01781c --- /dev/null +++ b/doc/development/sidekiq/limited_capacity_worker.md @@ -0,0 +1,84 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Sidekiq limited capacity worker + +It is possible to limit the number of concurrent running jobs for a worker class +by using the `LimitedCapacity::Worker` concern. + +The worker must implement three methods: + +- `perform_work`: The concern implements the usual `perform` method and calls + `perform_work` if there's any available capacity. +- `remaining_work_count`: Number of jobs that have work to perform. +- `max_running_jobs`: Maximum number of jobs allowed to run concurrently. + +```ruby +class MyDummyWorker + include ApplicationWorker + include LimitedCapacity::Worker + + def perform_work(*args) + end + + def remaining_work_count(*args) + 5 + end + + def max_running_jobs + 25 + end +end +``` + +Additional to the regular worker, a cron worker must be defined as well to +backfill the queue with jobs. the arguments passed to `perform_with_capacity` +are passed to the `perform_work` method. + +```ruby +class ScheduleMyDummyCronWorker + include ApplicationWorker + include CronjobQueue + + def perform(*args) + MyDummyWorker.perform_with_capacity(*args) + end +end +``` + +## How many jobs are running? + +It runs `max_running_jobs` at almost all times. + +The cron worker checks the remaining capacity on each execution and it +schedules at most `max_running_jobs` jobs. Those jobs on completion +re-enqueue themselves immediately, but not on failure. The cron worker is in +charge of replacing those failed jobs. + +## Handling errors and idempotence + +This concern disables Sidekiq retries, logs the errors, and sends the job to the +dead queue. This is done to have only one source that produces jobs and because +the retry would occupy a slot with a job to perform in the distant future. + +We let the cron worker enqueue new jobs, this could be seen as our retry and +back off mechanism because the job might fail again if executed immediately. +This means that for every failed job, we run at a lower capacity +until the cron worker fills the capacity again. If it is important for the +worker not to get a backlog, exceptions must be handled in `#perform_work` and +the job should not raise. + +The jobs are deduplicated using the `:none` strategy, but the worker is not +marked as `idempotent!`. + +## Metrics + +This concern exposes three Prometheus metrics of gauge type with the worker class +name as label: + +- `limited_capacity_worker_running_jobs` +- `limited_capacity_worker_max_running_jobs` +- `limited_capacity_worker_remaining_work_count` diff --git a/doc/development/sidekiq/logging.md b/doc/development/sidekiq/logging.md new file mode 100644 index 00000000000..015376b0fc6 --- /dev/null +++ b/doc/development/sidekiq/logging.md @@ -0,0 +1,155 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Sidekiq logging + +## Worker context + +> [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/9) in GitLab 12.8. + +To have some more information about workers in the logs, we add +[metadata to the jobs in the form of an +`ApplicationContext`](../logging.md#logging-context-metadata-through-rails-or-grape-requests). +In most cases, when scheduling a job from a request, this context is already +deducted from the request and added to the scheduled job. + +When a job runs, the context that was active when it was scheduled +is restored. This causes the context to be propagated to any job +scheduled from within the running job. + +All this means that in most cases, to add context to jobs, we don't +need to do anything. + +There are however some instances when there would be no context +present when the job is scheduled, or the context that is present is +likely to be incorrect. For these instances, we've added Rubocop rules +to draw attention and avoid incorrect metadata in our logs. + +As with most our cops, there are perfectly valid reasons for disabling +them. In this case it could be that the context from the request is +correct. Or maybe you've specified a context already in a way that +isn't picked up by the cops. In any case, leave a code comment +pointing to which context to use when disabling the cops. + +When you do provide objects to the context, make sure that the +route for namespaces and projects is pre-loaded. This can be done by using +the `.with_route` scope defined on all `Routable`s. + +### Cron workers + +The context is automatically cleared for workers in the cronjob queue +(`include CronjobQueue`), even when scheduling them from +requests. We do this to avoid incorrect metadata when other jobs are +scheduled from the cron worker. + +Cron workers themselves run instance wide, so they aren't scoped to +users, namespaces, projects, or other resources that should be added to +the context. + +However, they often schedule other jobs that _do_ require context. + +That is why there needs to be an indication of context somewhere in +the worker. This can be done by using one of the following methods +somewhere within the worker: + +1. Wrap the code that schedules jobs in the `with_context` helper: + + ```ruby + def perform + deletion_cutoff = Gitlab::CurrentSettings + .deletion_adjourned_period.days.ago.to_date + projects = Project.with_route.with_namespace + .aimed_for_deletion(deletion_cutoff) + + projects.find_each(batch_size: 100).with_index do |project, index| + delay = index * INTERVAL + + with_context(project: project) do + AdjournedProjectDeletionWorker.perform_in(delay, project.id) + end + end + end + ``` + +1. Use the a batch scheduling method that provides context: + + ```ruby + def schedule_projects_in_batch(projects) + ProjectImportScheduleWorker.bulk_perform_async_with_contexts( + projects, + arguments_proc: -> (project) { project.id }, + context_proc: -> (project) { { project: project } } + ) + end + ``` + + Or, when scheduling with delays: + + ```ruby + diffs.each_batch(of: BATCH_SIZE) do |diffs, index| + DeleteDiffFilesWorker + .bulk_perform_in_with_contexts(index * 5.minutes, + diffs, + arguments_proc: -> (diff) { diff.id }, + context_proc: -> (diff) { { project: diff.merge_request.target_project } }) + end + ``` + +### Jobs scheduled in bulk + +Often, when scheduling jobs in bulk, these jobs should have a separate +context rather than the overarching context. + +If that is the case, `bulk_perform_async` can be replaced by the +`bulk_perform_async_with_context` helper, and instead of +`bulk_perform_in` use `bulk_perform_in_with_context`. + +For example: + +```ruby + ProjectImportScheduleWorker.bulk_perform_async_with_contexts( + projects, + arguments_proc: -> (project) { project.id }, + context_proc: -> (project) { { project: project } } + ) +``` + +Each object from the enumerable in the first argument is yielded into 2 +blocks: + +- The `arguments_proc` which needs to return the list of arguments the + job needs to be scheduled with. + +- The `context_proc` which needs to return a hash with the context + information for the job. + +## Arguments logging + +As of GitLab 13.6, Sidekiq job arguments are logged by default, unless [`SIDEKIQ_LOG_ARGUMENTS`](../../administration/troubleshooting/sidekiq.md#log-arguments-to-sidekiq-jobs) +is disabled. + +By default, the only arguments logged are numeric arguments, because +arguments of other types could contain sensitive information. To +override this, use `loggable_arguments` inside a worker with the indexes +of the arguments to be logged. (Numeric arguments do not need to be +specified here.) + +For example: + +```ruby +class MyWorker + include ApplicationWorker + + loggable_arguments 1, 3 + + # object_id will be logged as it's numeric + # string_a will be logged due to the loggable_arguments call + # string_b will be filtered from logs + # string_c will be logged due to the loggable_arguments call + def perform(object_id, string_a, string_b, string_c) + end +end +``` diff --git a/doc/development/sidekiq/worker_attributes.md b/doc/development/sidekiq/worker_attributes.md new file mode 100644 index 00000000000..d681e17a053 --- /dev/null +++ b/doc/development/sidekiq/worker_attributes.md @@ -0,0 +1,325 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Sidekiq worker attributes + +## Job urgency + +Jobs can have an `urgency` attribute set, which can be `:high`, +`:low`, or `:throttled`. These have the below targets: + +| **Urgency** | **Queue Scheduling Target** | **Execution Latency Requirement** | +|--------------|-----------------------------|------------------------------------| +| `:high` | 10 seconds | p50 of 1 second, p99 of 10 seconds | +| `:low` | 1 minute | Maximum run time of 5 minutes | +| `:throttled` | None | Maximum run time of 5 minutes | + +To set a job's urgency, use the `urgency` class method: + +```ruby +class HighUrgencyWorker + include ApplicationWorker + + urgency :high + + # ... +end +``` + +### Latency sensitive jobs + +If a large number of background jobs get scheduled at once, queueing of jobs may +occur while jobs wait for a worker node to be become available. This is normal +and gives the system resilience by allowing it to gracefully handle spikes in +traffic. Some jobs, however, are more sensitive to latency than others. + +In general, latency-sensitive jobs perform operations that a user could +reasonably expect to happen synchronously, rather than asynchronously in a +background worker. A common example is a write following an action. Examples of +these jobs include: + +1. A job which updates a merge request following a push to a branch. +1. A job which invalidates a cache of known branches for a project after a push + to the branch. +1. A job which recalculates the groups and projects a user can see after a + change in permissions. +1. A job which updates the status of a CI pipeline after a state change to a job + in the pipeline. + +When these jobs are delayed, the user may perceive the delay as a bug: for +example, they may push a branch and then attempt to create a merge request for +that branch, but be told in the UI that the branch does not exist. We deem these +jobs to be `urgency :high`. + +Extra effort is made to ensure that these jobs are started within a very short +period of time after being scheduled. However, in order to ensure throughput, +these jobs also have very strict execution duration requirements: + +1. The median job execution time should be less than 1 second. +1. 99% of jobs should complete within 10 seconds. + +If a worker cannot meet these expectations, then it cannot be treated as a +`urgency :high` worker: consider redesigning the worker, or splitting the +work between two different workers, one with `urgency :high` code that +executes quickly, and the other with `urgency :low`, which has no +execution latency requirements (but also has lower scheduling targets). + +### Changing a queue's urgency + +On GitLab.com, we run Sidekiq in several +[shards](https://dashboards.gitlab.net/d/sidekiq-shard-detail/sidekiq-shard-detail), +each of which represents a particular type of workload. + +When changing a queue's urgency, or adding a new queue, we need to take +into account the expected workload on the new shard. Note that, if we're +changing an existing queue, there is also an effect on the old shard, +but that always reduces work. + +To do this, we want to calculate the expected increase in total execution time +and RPS (throughput) for the new shard. We can get these values from: + +- The [Queue Detail + dashboard](https://dashboards.gitlab.net/d/sidekiq-queue-detail/sidekiq-queue-detail) + has values for the queue itself. For a new queue, we can look for + queues that have similar patterns or are scheduled in similar + circumstances. +- The [Shard Detail + dashboard](https://dashboards.gitlab.net/d/sidekiq-shard-detail/sidekiq-shard-detail) + has Total Execution Time and Throughput (RPS). The Shard Utilization + panel displays if there is currently any excess capacity for this + shard. + +We can then calculate the RPS * average runtime (estimated for new jobs) +for the queue we're changing to see what the relative increase in RPS and +execution time we expect for the new shard: + +```ruby +new_queue_consumption = queue_rps * queue_duration_avg +shard_consumption = shard_rps * shard_duration_avg + +(new_queue_consumption / shard_consumption) * 100 +``` + +If we expect an increase of **less than 5%**, then no further action is needed. + +Otherwise, please ping `@gitlab-org/scalability` on the merge request and ask +for a review. + +## Jobs with External Dependencies + +Most background jobs in the GitLab application communicate with other GitLab +services. For example, PostgreSQL, Redis, Gitaly, and Object Storage. These are considered +to be "internal" dependencies for a job. + +However, some jobs are dependent on external services in order to complete +successfully. Some examples include: + +1. Jobs which call web-hooks configured by a user. +1. Jobs which deploy an application to a k8s cluster configured by a user. + +These jobs have "external dependencies". This is important for the operation of +the background processing cluster in several ways: + +1. Most external dependencies (such as web-hooks) do not provide SLOs, and + therefore we cannot guarantee the execution latencies on these jobs. Since we + cannot guarantee execution latency, we cannot ensure throughput and + therefore, in high-traffic environments, we need to ensure that jobs with + external dependencies are separated from high urgency jobs, to ensure + throughput on those queues. +1. Errors in jobs with external dependencies have higher alerting thresholds as + there is a likelihood that the cause of the error is external. + +```ruby +class ExternalDependencyWorker + include ApplicationWorker + + # Declares that this worker depends on + # third-party, external services in order + # to complete successfully + worker_has_external_dependencies! + + # ... +end +``` + +A job cannot be both high urgency and have external dependencies. + +## CPU-bound and Memory-bound Workers + +Workers that are constrained by CPU or memory resource limitations should be +annotated with the `worker_resource_boundary` method. + +Most workers tend to spend most of their time blocked, waiting on network responses +from other services such as Redis, PostgreSQL, and Gitaly. Since Sidekiq is a +multi-threaded environment, these jobs can be scheduled with high concurrency. + +Some workers, however, spend large amounts of time _on-CPU_ running logic in +Ruby. Ruby MRI does not support true multi-threading - it relies on the +[GIL](https://thoughtbot.com/blog/untangling-ruby-threads#the-global-interpreter-lock) +to greatly simplify application development by only allowing one section of Ruby +code in a process to run at a time, no matter how many cores the machine +hosting the process has. For IO bound workers, this is not a problem, since most +of the threads are blocked in underlying libraries (which are outside of the +GIL). + +If many threads are attempting to run Ruby code simultaneously, this leads +to contention on the GIL which has the effect of slowing down all +processes. + +In high-traffic environments, knowing that a worker is CPU-bound allows us to +run it on a different fleet with lower concurrency. This ensures optimal +performance. + +Likewise, if a worker uses large amounts of memory, we can run these on a +bespoke low concurrency, high memory fleet. + +Note that memory-bound workers create heavy GC workloads, with pauses of +10-50ms. This has an impact on the latency requirements for the +worker. For this reason, `memory` bound, `urgency :high` jobs are not +permitted and fail CI. In general, `memory` bound workers are +discouraged, and alternative approaches to processing the work should be +considered. + +If a worker needs large amounts of both memory and CPU time, it should +be marked as memory-bound, due to the above restriction on high urgency +memory-bound workers. + +## Declaring a Job as CPU-bound + +This example shows how to declare a job as being CPU-bound. + +```ruby +class CPUIntensiveWorker + include ApplicationWorker + + # Declares that this worker will perform a lot of + # calculations on-CPU. + worker_resource_boundary :cpu + + # ... +end +``` + +## Determining whether a worker is CPU-bound + +We use the following approach to determine whether a worker is CPU-bound: + +- In the Sidekiq structured JSON logs, aggregate the worker `duration` and + `cpu_s` fields. +- `duration` refers to the total job execution duration, in seconds +- `cpu_s` is derived from the + [`Process::CLOCK_THREAD_CPUTIME_ID`](https://www.rubydoc.info/stdlib/core/Process:clock_gettime) + counter, and is a measure of time spent by the job on-CPU. +- Divide `cpu_s` by `duration` to get the percentage time spend on-CPU. +- If this ratio exceeds 33%, the worker is considered CPU-bound and should be + annotated as such. +- Note that these values should not be used over small sample sizes, but + rather over fairly large aggregates. + +## Feature category + +All Sidekiq workers must define a known [feature category](../feature_categorization/index.md#sidekiq-workers). + +## Job data consistency strategies + +In GitLab 13.11 and earlier, Sidekiq workers would always send database queries to the primary +database node, +both for reads and writes. This ensured that data integrity +is both guaranteed and immediate, since in a single-node scenario it is impossible to encounter +stale reads even for workers that read their own writes. +If a worker writes to the primary, but reads from a replica, however, the possibility +of reading a stale record is non-zero due to replicas potentially lagging behind the primary. + +When the number of jobs that rely on the database increases, ensuring immediate data consistency +can put unsustainable load on the primary database server. We therefore added the ability to use +[Database Load Balancing for Sidekiq workers](../../administration/postgresql/database_load_balancing.md). +By configuring a worker's `data_consistency` field, we can then allow the scheduler to target read replicas +under several strategies outlined below. + +## Trading immediacy for reduced primary load + +We require Sidekiq workers to make an explicit decision around whether they need to use the +primary database node for all reads and writes, or whether reads can be served from replicas. This is +enforced by a RuboCop rule, which ensures that the `data_consistency` field is set. + +When setting this field, consider the following trade-off: + +- Ensure immediately consistent reads, but increase load on the primary database. +- Prefer read replicas to add relief to the primary, but increase the likelihood of stale reads that have to be retried. + +To maintain the same behavior compared to before this field was introduced, set it to `:always`, so +database operations will only target the primary. Reasons for having to do so include workers +that mostly or exclusively perform writes, or workers that read their own writes and who might run +into data consistency issues should a stale record be read back from a replica. **Try to avoid +these scenarios, since `:always` should be considered the exception, not the rule.** + +To allow for reads to be served from replicas, we added two additional consistency modes: `:sticky` and `:delayed`. + +When you declare either `:sticky` or `:delayed` consistency, workers become eligible for database +load-balancing. + +In both cases, if the replica is not up-to-date and the time from scheduling the job was less than the minimum delay interval, + the jobs sleep up to the minimum delay interval (0.8 seconds). This gives the replication process time to finish. +The difference is in what happens when there is still replication lag after the delay: `sticky` workers +switch over to the primary right away, whereas `delayed` workers fail fast and are retried once. +If they still encounter replication lag, they also switch to the primary instead. +**If your worker never performs any writes, it is strongly advised to apply one of these consistency settings, +since it will never need to rely on the primary database node.** + +The table below shows the `data_consistency` attribute and its values, ordered by the degree to which +they prefer read replicas and will wait for replicas to catch up: + +| **Data Consistency** | **Description** | +|--------------|-----------------------------| +| `:always` | The job is required to use the primary database (default). It should be used for workers that primarily perform writes or that have strict requirements around data consistency when reading their own writes. | +| `:sticky` | The job prefers replicas, but switches to the primary for writes or when encountering replication lag. It should be used for jobs that require to be executed as fast as possible but can sustain a small initial queuing delay. | +| `:delayed` | The job prefers replicas, but switches to the primary for writes. When encountering replication lag before the job starts, the job is retried once. If the replica is still not up to date on the next retry, it switches to the primary. It should be used for jobs where delaying execution further typically does not matter, such as cache expiration or web hooks execution. | + +In all cases workers read either from a replica that is fully caught up, +or from the primary node, so data consistency is always ensured. + +To set a data consistency for a worker, use the `data_consistency` class method: + +```ruby +class DelayedWorker + include ApplicationWorker + + data_consistency :delayed + + # ... +end +``` + +### `feature_flag` property + +The `feature_flag` property allows you to toggle a job's `data_consistency`, +which permits you to safely toggle load balancing capabilities for a specific job. +When `feature_flag` is disabled, the job defaults to `:always`, which means that the job will always use the primary database. + +The `feature_flag` property does not allow the use of +[feature gates based on actors](../feature_flags/index.md). +This means that the feature flag cannot be toggled only for particular +projects, groups, or users, but instead, you can safely use [percentage of time rollout](../feature_flags/index.md). +Note that since we check the feature flag on both Sidekiq client and server, rolling out a 10% of the time, +will likely results in 1% (`0.1` `[from client]*0.1` `[from server]`) of effective jobs using replicas. + +Example: + +```ruby +class DelayedWorker + include ApplicationWorker + + data_consistency :delayed, feature_flag: :load_balancing_for_delayed_worker + + # ... +end +``` + +### Data consistency with idempotent jobs + +For [idempotent jobs](idempotent_jobs.md) that declare either `:sticky` or `:delayed` data consistency, we are +[preserving the latest WAL location](idempotent_jobs.md#preserve-the-latest-wal-location-for-idempotent-jobs) while deduplicating, +ensuring that we read from the replica that is fully caught up. diff --git a/doc/development/sidekiq_style_guide.md b/doc/development/sidekiq_style_guide.md index af994e7138d..3756dd6b598 100644 --- a/doc/development/sidekiq_style_guide.md +++ b/doc/development/sidekiq_style_guide.md @@ -1,1101 +1,9 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'sidekiq/index.md' +remove_date: '2022-04-13' --- -# Sidekiq Style Guide +This document was moved to [another location](sidekiq/index.md). -This document outlines various guidelines that should be followed when adding or -modifying Sidekiq workers. - -## ApplicationWorker - -All workers should include `ApplicationWorker` instead of `Sidekiq::Worker`, -which adds some convenience methods and automatically sets the queue based on -the [routing rules](../administration/operations/extra_sidekiq_routing.md#queue-routing-rules). - -## Retries - -Sidekiq defaults to using [25 -retries](https://github.com/mperham/sidekiq/wiki/Error-Handling#automatic-job-retry), -with back-off between each retry. 25 retries means that the last retry -would happen around three weeks after the first attempt (assuming all 24 -prior retries failed). - -For most workers - especially [idempotent workers](#idempotent-jobs) - -the default of 25 retries is more than sufficient. Many of our older -workers declare 3 retries, which used to be the default within the -GitLab application. 3 retries happen over the course of a couple of -minutes, so the jobs are prone to failing completely. - -A lower retry count may be applicable if any of the below apply: - -1. The worker contacts an external service and we do not provide - guarantees on delivery. For example, webhooks. -1. The worker is not idempotent and running it multiple times could - leave the system in an inconsistent state. For example, a worker that - posts a system note and then performs an action: if the second step - fails and the worker retries, the system note will be posted again. -1. The worker is a cronjob that runs frequently. For example, if a cron - job runs every hour, then we don't need to retry beyond an hour - because we don't need two of the same job running at once. - -Each retry for a worker is counted as a failure in our metrics. A worker -which always fails 9 times and succeeds on the 10th would have a 90% -error rate. - -## Sidekiq Queues - -Previously, each worker had its own queue, which was automatically set based on the -worker class name. For a worker named `ProcessSomethingWorker`, the queue name -would be `process_something`. You can now route workers to a specific queue using -[queue routing rules](../administration/operations/extra_sidekiq_routing.md#queue-routing-rules). -In GDK, new workers are routed to a queue named `default`. - -If you're not sure what queue a worker uses, -you can find it using `SomeWorker.queue`. There is almost never a reason to -manually override the queue name using `sidekiq_options queue: :some_queue`. - -After adding a new worker, run `bin/rake -gitlab:sidekiq:all_queues_yml:generate` to regenerate -`app/workers/all_queues.yml` or `ee/app/workers/all_queues.yml` so that -it can be picked up by -[`sidekiq-cluster`](../administration/operations/extra_sidekiq_processes.md) -in installations that don't use routing rules. To learn more about potential changes, -read [Use routing rules by default and deprecate queue selectors for self-managed](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/596). - -Additionally, run -`bin/rake gitlab:sidekiq:sidekiq_queues_yml:generate` to regenerate -`config/sidekiq_queues.yml`. - -## Queue Namespaces - -While different workers cannot share a queue, they can share a queue namespace. - -Defining a queue namespace for a worker makes it possible to start a Sidekiq -process that automatically handles jobs for all workers in that namespace, -without needing to explicitly list all their queue names. If, for example, all -workers that are managed by `sidekiq-cron` use the `cronjob` queue namespace, we -can spin up a Sidekiq process specifically for these kinds of scheduled jobs. -If a new worker using the `cronjob` namespace is added later on, the Sidekiq -process also picks up jobs for that worker (after having been restarted), -without the need to change any configuration. - -A queue namespace can be set using the `queue_namespace` DSL class method: - -```ruby -class SomeScheduledTaskWorker - include ApplicationWorker - - queue_namespace :cronjob - - # ... -end -``` - -Behind the scenes, this sets `SomeScheduledTaskWorker.queue` to -`cronjob:some_scheduled_task`. Commonly used namespaces have their own -concern module that can easily be included into the worker class, and that may -set other Sidekiq options besides the queue namespace. `CronjobQueue`, for -example, sets the namespace, but also disables retries. - -`bundle exec sidekiq` is namespace-aware, and listens on all -queues in a namespace (technically: all queues prefixed with the namespace name) -when a namespace is provided instead of a simple queue name in the `--queue` -(`-q`) option, or in the `:queues:` section in `config/sidekiq_queues.yml`. - -Note that adding a worker to an existing namespace should be done with care, as -the extra jobs take resources away from jobs from workers that were already -there, if the resources available to the Sidekiq process handling the namespace -are not adjusted appropriately. - -## Versioning - -Version can be specified on each Sidekiq worker class. -This is then sent along when the job is created. - -```ruby -class FooWorker - include ApplicationWorker - - version 2 - - def perform(*args) - if job_version == 2 - foo = args.first['foo'] - else - foo = args.first - end - end -end -``` - -Under this schema, any worker is expected to be able to handle any job that was -enqueued by an older version of that worker. This means that when changing the -arguments a worker takes, you must increment the `version` (or set `version 1` -if this is the first time a worker's arguments are changing), but also make sure -that the worker is still able to handle jobs that were queued with any earlier -version of the arguments. From the worker's `perform` method, you can read -`self.job_version` if you want to specifically branch on job version, or you -can read the number or type of provided arguments. - -## Idempotent Jobs - -It's known that a job can fail for multiple reasons. For example, network outages or bugs. -In order to address this, Sidekiq has a built-in retry mechanism that is -used by default by most workers within GitLab. - -It's expected that a job can run again after a failure without major side-effects for the -application or users, which is why Sidekiq encourages -jobs to be [idempotent and transactional](https://github.com/mperham/sidekiq/wiki/Best-Practices#2-make-your-job-idempotent-and-transactional). - -As a general rule, a worker can be considered idempotent if: - -- It can safely run multiple times with the same arguments. -- Application side-effects are expected to happen only once - (or side-effects of a second run do not have an effect). - -A good example of that would be a cache expiration worker. - -A job scheduled for an idempotent worker is [deduplicated](#deduplication) when -an unstarted job with the same arguments is already in the queue. - -### Ensuring a worker is idempotent - -Make sure the worker tests pass using the following shared example: - -```ruby -include_examples 'an idempotent worker' do - it 'marks the MR as merged' do - # Using subject inside this block will process the job multiple times - subject - - expect(merge_request.state).to eq('merged') - end -end -``` - -Use the `perform_multiple` method directly instead of `job.perform` (this -helper method is automatically included for workers). - -### Declaring a worker as idempotent - -```ruby -class IdempotentWorker - include ApplicationWorker - - # Declares a worker is idempotent and can - # safely run multiple times. - idempotent! - - # ... -end -``` - -It's encouraged to only have the `idempotent!` call in the top-most worker class, even if -the `perform` method is defined in another class or module. - -If the worker class isn't marked as idempotent, a cop fails. Consider skipping -the cop if you're not confident your job can safely run multiple times. - -### Deduplication - -When a job for an idempotent worker is enqueued while another -unstarted job is already in the queue, GitLab drops the second -job. The work is skipped because the same work would be -done by the job that was scheduled first; by the time the second -job executed, the first job would do nothing. - -#### Strategies - -GitLab supports two deduplication strategies: - -- `until_executing` -- `until_executed` - -More [deduplication strategies have been -suggested](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/195). If -you are implementing a worker that could benefit from a different -strategy, please comment in the issue. - -##### Until Executing - -This strategy takes a lock when a job is added to the queue, and removes that lock before the job starts. - -For example, `AuthorizedProjectsWorker` takes a user ID. When the -worker runs, it recalculates a user's authorizations. GitLab schedules -this job each time an action potentially changes a user's -authorizations. If the same user is added to two projects at the -same time, the second job can be skipped if the first job hasn't -begun, because when the first job runs, it creates the -authorizations for both projects. - -```ruby -module AuthorizedProjectUpdate - class UserRefreshOverUserRangeWorker - include ApplicationWorker - - deduplicate :until_executing - idempotent! - - # ... - end -end -``` - -##### Until Executed - -This strategy takes a lock when a job is added to the queue, and removes that lock after the job finishes. -It can be used to prevent jobs from running simultaneously multiple times. - -```ruby -module Ci - class BuildTraceChunkFlushWorker - include ApplicationWorker - - deduplicate :until_executed - idempotent! - - # ... - end -end -``` - -Also, you can pass `if_deduplicated: :reschedule_once` option to re-run a job once after -the currently running job finished and deduplication happened at least once. -This ensures that the latest result is always produced even if a race condition -happened. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/342123) for more information. - -#### Scheduling jobs in the future - -GitLab doesn't skip jobs scheduled in the future, as we assume that -the state has changed by the time the job is scheduled to -execute. Deduplication of jobs scheduled in the feature is possible -for both `until_executed` and `until_executing` strategies. - -If you do want to deduplicate jobs scheduled in the future, -this can be specified on the worker by passing `including_scheduled: true` argument -when defining deduplication strategy: - -```ruby -module AuthorizedProjectUpdate - class UserRefreshOverUserRangeWorker - include ApplicationWorker - - deduplicate :until_executing, including_scheduled: true - idempotent! - - # ... - end -end -``` - -### Setting the deduplication time-to-live (TTL) - -Deduplication depends on an idempotency key that is stored in Redis. This is normally -cleared by the configured deduplication strategy. - -However, the key can remain until its TTL in certain cases like: - -1. `until_executing` is used but the job was never enqueued or executed after the Sidekiq - client middleware was run. - -1. `until_executed` is used but the job fails to finish due to retry exhaustion, gets - interrupted the maximum number of times, or gets lost. - -The default value is 6 hours. During this time, jobs won't be enqueued even if the first -job never executed or finished. - -The TTL can be configured with: - -```ruby -class ProjectImportScheduleWorker - include ApplicationWorker - - idempotent! - deduplicate :until_executing, ttl: 5.minutes -end -``` - -Duplicate jobs can happen when the TTL is reached, so make sure you lower this only for jobs -that can tolerate some duplication. - -### Deduplication with load balancing - -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/6763) in GitLab 14.4. - -Jobs that declare either `:sticky` or `:delayed` data consistency -are eligible for database load-balancing. -In both cases, jobs are [scheduled in the future](#scheduling-jobs-in-the-future) with a short delay (1 second). -This minimizes the chance of replication lag after a write. - -If you really want to deduplicate jobs eligible for load balancing, -specify `including_scheduled: true` argument when defining deduplication strategy: - -```ruby -class DelayedIdempotentWorker - include ApplicationWorker - data_consistency :delayed - - deduplicate :until_executing, including_scheduled: true - idempotent! - - # ... -end -``` - -#### Preserve the latest WAL location for idempotent jobs - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69372) in GitLab 14.3. -> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/338350) in GitLab 14.4. -> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/338350) in GitLab 14.6. - -The deduplication always take into account the latest binary replication pointer, not the first one. -This happens because we drop the same job scheduled for the second time and the Write-Ahead Log (WAL) is lost. -This could lead to comparing the old WAL location and reading from a stale replica. - -To support both deduplication and maintaining data consistency with load balancing, -we are preserving the latest WAL location for idempotent jobs in Redis. -This way we are always comparing the latest binary replication pointer, -making sure that we read from the replica that is fully caught up. - -FLAG: -On self-managed GitLab, by default this feature is available. To hide the feature, ask an administrator to -[disable the feature flag](../administration/feature_flags.md) named `preserve_latest_wal_locations_for_idempotent_jobs`. - -This feature flag is related to GitLab development and is not intended to be used by GitLab administrators, though. -On GitLab.com, this feature is available. - -## Limited capacity worker - -It is possible to limit the number of concurrent running jobs for a worker class -by using the `LimitedCapacity::Worker` concern. - -The worker must implement three methods: - -- `perform_work`: The concern implements the usual `perform` method and calls - `perform_work` if there's any available capacity. -- `remaining_work_count`: Number of jobs that have work to perform. -- `max_running_jobs`: Maximum number of jobs allowed to run concurrently. - -```ruby -class MyDummyWorker - include ApplicationWorker - include LimitedCapacity::Worker - - def perform_work(*args) - end - - def remaining_work_count(*args) - 5 - end - - def max_running_jobs - 25 - end -end -``` - -Additional to the regular worker, a cron worker must be defined as well to -backfill the queue with jobs. the arguments passed to `perform_with_capacity` -are passed to the `perform_work` method. - -```ruby -class ScheduleMyDummyCronWorker - include ApplicationWorker - include CronjobQueue - - def perform(*args) - MyDummyWorker.perform_with_capacity(*args) - end -end -``` - -### How many jobs are running? - -It runs `max_running_jobs` at almost all times. - -The cron worker checks the remaining capacity on each execution and it -schedules at most `max_running_jobs` jobs. Those jobs on completion -re-enqueue themselves immediately, but not on failure. The cron worker is in -charge of replacing those failed jobs. - -### Handling errors and idempotence - -This concern disables Sidekiq retries, logs the errors, and sends the job to the -dead queue. This is done to have only one source that produces jobs and because -the retry would occupy a slot with a job to perform in the distant future. - -We let the cron worker enqueue new jobs, this could be seen as our retry and -back off mechanism because the job might fail again if executed immediately. -This means that for every failed job, we run at a lower capacity -until the cron worker fills the capacity again. If it is important for the -worker not to get a backlog, exceptions must be handled in `#perform_work` and -the job should not raise. - -The jobs are deduplicated using the `:none` strategy, but the worker is not -marked as `idempotent!`. - -### Metrics - -This concern exposes three Prometheus metrics of gauge type with the worker class -name as label: - -- `limited_capacity_worker_running_jobs` -- `limited_capacity_worker_max_running_jobs` -- `limited_capacity_worker_remaining_work_count` - -## Job urgency - -Jobs can have an `urgency` attribute set, which can be `:high`, -`:low`, or `:throttled`. These have the below targets: - -| **Urgency** | **Queue Scheduling Target** | **Execution Latency Requirement** | -|--------------|-----------------------------|------------------------------------| -| `:high` | 10 seconds | p50 of 1 second, p99 of 10 seconds | -| `:low` | 1 minute | Maximum run time of 5 minutes | -| `:throttled` | None | Maximum run time of 5 minutes | - -To set a job's urgency, use the `urgency` class method: - -```ruby -class HighUrgencyWorker - include ApplicationWorker - - urgency :high - - # ... -end -``` - -### Latency sensitive jobs - -If a large number of background jobs get scheduled at once, queueing of jobs may -occur while jobs wait for a worker node to be become available. This is normal -and gives the system resilience by allowing it to gracefully handle spikes in -traffic. Some jobs, however, are more sensitive to latency than others. - -In general, latency-sensitive jobs perform operations that a user could -reasonably expect to happen synchronously, rather than asynchronously in a -background worker. A common example is a write following an action. Examples of -these jobs include: - -1. A job which updates a merge request following a push to a branch. -1. A job which invalidates a cache of known branches for a project after a push - to the branch. -1. A job which recalculates the groups and projects a user can see after a - change in permissions. -1. A job which updates the status of a CI pipeline after a state change to a job - in the pipeline. - -When these jobs are delayed, the user may perceive the delay as a bug: for -example, they may push a branch and then attempt to create a merge request for -that branch, but be told in the UI that the branch does not exist. We deem these -jobs to be `urgency :high`. - -Extra effort is made to ensure that these jobs are started within a very short -period of time after being scheduled. However, in order to ensure throughput, -these jobs also have very strict execution duration requirements: - -1. The median job execution time should be less than 1 second. -1. 99% of jobs should complete within 10 seconds. - -If a worker cannot meet these expectations, then it cannot be treated as a -`urgency :high` worker: consider redesigning the worker, or splitting the -work between two different workers, one with `urgency :high` code that -executes quickly, and the other with `urgency :low`, which has no -execution latency requirements (but also has lower scheduling targets). - -### Changing a queue's urgency - -On GitLab.com, we run Sidekiq in several -[shards](https://dashboards.gitlab.net/d/sidekiq-shard-detail/sidekiq-shard-detail), -each of which represents a particular type of workload. - -When changing a queue's urgency, or adding a new queue, we need to take -into account the expected workload on the new shard. Note that, if we're -changing an existing queue, there is also an effect on the old shard, -but that always reduces work. - -To do this, we want to calculate the expected increase in total execution time -and RPS (throughput) for the new shard. We can get these values from: - -- The [Queue Detail - dashboard](https://dashboards.gitlab.net/d/sidekiq-queue-detail/sidekiq-queue-detail) - has values for the queue itself. For a new queue, we can look for - queues that have similar patterns or are scheduled in similar - circumstances. -- The [Shard Detail - dashboard](https://dashboards.gitlab.net/d/sidekiq-shard-detail/sidekiq-shard-detail) - has Total Execution Time and Throughput (RPS). The Shard Utilization - panel displays if there is currently any excess capacity for this - shard. - -We can then calculate the RPS * average runtime (estimated for new jobs) -for the queue we're changing to see what the relative increase in RPS and -execution time we expect for the new shard: - -```ruby -new_queue_consumption = queue_rps * queue_duration_avg -shard_consumption = shard_rps * shard_duration_avg - -(new_queue_consumption / shard_consumption) * 100 -``` - -If we expect an increase of **less than 5%**, then no further action is needed. - -Otherwise, please ping `@gitlab-org/scalability` on the merge request and ask -for a review. - -## Job size - -GitLab stores Sidekiq jobs and their arguments in Redis. To avoid -excessive memory usage, we compress the arguments of Sidekiq jobs -if their original size is bigger than 100KB. - -After compression, if their size still exceeds 5MB, it raises an -[`ExceedLimitError`](https://gitlab.com/gitlab-org/gitlab/-/blob/f3dd89e5e510ea04b43ffdcb58587d8f78a8d77c/lib/gitlab/sidekiq_middleware/size_limiter/exceed_limit_error.rb#L8) -error when scheduling the job. - -If this happens, rely on other means of making the data -available in Sidekiq. There are possible workarounds such as: - -- Rebuild the data in Sidekiq with data loaded from the database or - elsewhere. -- Store the data in [object storage](file_storage.md#object-storage) - before scheduling the job, and retrieve it inside the job. - -## Job data consistency strategies - -In GitLab 13.11 and earlier, Sidekiq workers would always send database queries to the primary -database node, -both for reads and writes. This ensured that data integrity -is both guaranteed and immediate, since in a single-node scenario it is impossible to encounter -stale reads even for workers that read their own writes. -If a worker writes to the primary, but reads from a replica, however, the possibility -of reading a stale record is non-zero due to replicas potentially lagging behind the primary. - -When the number of jobs that rely on the database increases, ensuring immediate data consistency -can put unsustainable load on the primary database server. We therefore added the ability to use -[Database Load Balancing for Sidekiq workers](../administration/postgresql/database_load_balancing.md). -By configuring a worker's `data_consistency` field, we can then allow the scheduler to target read replicas -under several strategies outlined below. - -## Trading immediacy for reduced primary load - -We require Sidekiq workers to make an explicit decision around whether they need to use the -primary database node for all reads and writes, or whether reads can be served from replicas. This is -enforced by a RuboCop rule, which ensures that the `data_consistency` field is set. - -When setting this field, consider the following trade-off: - -- Ensure immediately consistent reads, but increase load on the primary database. -- Prefer read replicas to add relief to the primary, but increase the likelihood of stale reads that have to be retried. - -To maintain the same behavior compared to before this field was introduced, set it to `:always`, so -database operations will only target the primary. Reasons for having to do so include workers -that mostly or exclusively perform writes, or workers that read their own writes and who might run -into data consistency issues should a stale record be read back from a replica. **Try to avoid -these scenarios, since `:always` should be considered the exception, not the rule.** - -To allow for reads to be served from replicas, we added two additional consistency modes: `:sticky` and `:delayed`. - -When you declare either `:sticky` or `:delayed` consistency, workers become eligible for database -load-balancing. In both cases, jobs are enqueued with a short delay. -This minimizes the likelihood of replication lag after a write. - -The difference is in what happens when there is replication lag after the delay: `sticky` workers -switch over to the primary right away, whereas `delayed` workers fail fast and are retried once. -If they still encounter replication lag, they also switch to the primary instead. -**If your worker never performs any writes, it is strongly advised to apply one of these consistency settings, -since it will never need to rely on the primary database node.** - -The table below shows the `data_consistency` attribute and its values, ordered by the degree to which -they prefer read replicas and will wait for replicas to catch up: - -| **Data Consistency** | **Description** | -|--------------|-----------------------------| -| `:always` | The job is required to use the primary database (default). It should be used for workers that primarily perform writes or that have strict requirements around data consistency when reading their own writes. | -| `:sticky` | The job prefers replicas, but switches to the primary for writes or when encountering replication lag. It should be used for jobs that require to be executed as fast as possible but can sustain a small initial queuing delay. | -| `:delayed` | The job prefers replicas, but switches to the primary for writes. When encountering replication lag before the job starts, the job is retried once. If the replica is still not up to date on the next retry, it switches to the primary. It should be used for jobs where delaying execution further typically does not matter, such as cache expiration or web hooks execution. | - -In all cases workers read either from a replica that is fully caught up, -or from the primary node, so data consistency is always ensured. - -To set a data consistency for a worker, use the `data_consistency` class method: - -```ruby -class DelayedWorker - include ApplicationWorker - - data_consistency :delayed - - # ... -end -``` - -### `feature_flag` property - -The `feature_flag` property allows you to toggle a job's `data_consistency`, -which permits you to safely toggle load balancing capabilities for a specific job. -When `feature_flag` is disabled, the job defaults to `:always`, which means that the job will always use the primary database. - -The `feature_flag` property does not allow the use of -[feature gates based on actors](../development/feature_flags/index.md). -This means that the feature flag cannot be toggled only for particular -projects, groups, or users, but instead, you can safely use [percentage of time rollout](../development/feature_flags/index.md). -Note that since we check the feature flag on both Sidekiq client and server, rolling out a 10% of the time, -will likely results in 1% (`0.1` `[from client]*0.1` `[from server]`) of effective jobs using replicas. - -Example: - -```ruby -class DelayedWorker - include ApplicationWorker - - data_consistency :delayed, feature_flag: :load_balancing_for_delayed_worker - - # ... -end -``` - -### Data consistency with idempotent jobs - -For [idempotent jobs](#idempotent-jobs) that declare either `:sticky` or `:delayed` data consistency, we are -[preserving the latest WAL location](#preserve-the-latest-wal-location-for-idempotent-jobs) while deduplicating, -ensuring that we read from the replica that is fully caught up. - -## Jobs with External Dependencies - -Most background jobs in the GitLab application communicate with other GitLab -services. For example, PostgreSQL, Redis, Gitaly, and Object Storage. These are considered -to be "internal" dependencies for a job. - -However, some jobs are dependent on external services in order to complete -successfully. Some examples include: - -1. Jobs which call web-hooks configured by a user. -1. Jobs which deploy an application to a k8s cluster configured by a user. - -These jobs have "external dependencies". This is important for the operation of -the background processing cluster in several ways: - -1. Most external dependencies (such as web-hooks) do not provide SLOs, and - therefore we cannot guarantee the execution latencies on these jobs. Since we - cannot guarantee execution latency, we cannot ensure throughput and - therefore, in high-traffic environments, we need to ensure that jobs with - external dependencies are separated from high urgency jobs, to ensure - throughput on those queues. -1. Errors in jobs with external dependencies have higher alerting thresholds as - there is a likelihood that the cause of the error is external. - -```ruby -class ExternalDependencyWorker - include ApplicationWorker - - # Declares that this worker depends on - # third-party, external services in order - # to complete successfully - worker_has_external_dependencies! - - # ... -end -``` - -A job cannot be both high urgency and have external dependencies. - -## CPU-bound and Memory-bound Workers - -Workers that are constrained by CPU or memory resource limitations should be -annotated with the `worker_resource_boundary` method. - -Most workers tend to spend most of their time blocked, waiting on network responses -from other services such as Redis, PostgreSQL, and Gitaly. Since Sidekiq is a -multi-threaded environment, these jobs can be scheduled with high concurrency. - -Some workers, however, spend large amounts of time _on-CPU_ running logic in -Ruby. Ruby MRI does not support true multi-threading - it relies on the -[GIL](https://thoughtbot.com/blog/untangling-ruby-threads#the-global-interpreter-lock) -to greatly simplify application development by only allowing one section of Ruby -code in a process to run at a time, no matter how many cores the machine -hosting the process has. For IO bound workers, this is not a problem, since most -of the threads are blocked in underlying libraries (which are outside of the -GIL). - -If many threads are attempting to run Ruby code simultaneously, this leads -to contention on the GIL which has the effect of slowing down all -processes. - -In high-traffic environments, knowing that a worker is CPU-bound allows us to -run it on a different fleet with lower concurrency. This ensures optimal -performance. - -Likewise, if a worker uses large amounts of memory, we can run these on a -bespoke low concurrency, high memory fleet. - -Note that memory-bound workers create heavy GC workloads, with pauses of -10-50ms. This has an impact on the latency requirements for the -worker. For this reason, `memory` bound, `urgency :high` jobs are not -permitted and fail CI. In general, `memory` bound workers are -discouraged, and alternative approaches to processing the work should be -considered. - -If a worker needs large amounts of both memory and CPU time, it should -be marked as memory-bound, due to the above restriction on high urgency -memory-bound workers. - -## Declaring a Job as CPU-bound - -This example shows how to declare a job as being CPU-bound. - -```ruby -class CPUIntensiveWorker - include ApplicationWorker - - # Declares that this worker will perform a lot of - # calculations on-CPU. - worker_resource_boundary :cpu - - # ... -end -``` - -## Determining whether a worker is CPU-bound - -We use the following approach to determine whether a worker is CPU-bound: - -- In the Sidekiq structured JSON logs, aggregate the worker `duration` and - `cpu_s` fields. -- `duration` refers to the total job execution duration, in seconds -- `cpu_s` is derived from the - [`Process::CLOCK_THREAD_CPUTIME_ID`](https://www.rubydoc.info/stdlib/core/Process:clock_gettime) - counter, and is a measure of time spent by the job on-CPU. -- Divide `cpu_s` by `duration` to get the percentage time spend on-CPU. -- If this ratio exceeds 33%, the worker is considered CPU-bound and should be - annotated as such. -- Note that these values should not be used over small sample sizes, but - rather over fairly large aggregates. - -## Feature category - -All Sidekiq workers must define a known [feature category](feature_categorization/index.md#sidekiq-workers). - -## Job weights - -Some jobs have a weight declared. This is only used when running Sidekiq -in the default execution mode - using -[`sidekiq-cluster`](../administration/operations/extra_sidekiq_processes.md) -does not account for weights. - -As we are [moving towards using `sidekiq-cluster` in -Free](https://gitlab.com/gitlab-org/gitlab/-/issues/34396), newly-added -workers do not need to have weights specified. They can use the -default weight, which is 1. - -## Worker context - -> [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/9) in GitLab 12.8. - -To have some more information about workers in the logs, we add -[metadata to the jobs in the form of an -`ApplicationContext`](logging.md#logging-context-metadata-through-rails-or-grape-requests). -In most cases, when scheduling a job from a request, this context is already -deducted from the request and added to the scheduled job. - -When a job runs, the context that was active when it was scheduled -is restored. This causes the context to be propagated to any job -scheduled from within the running job. - -All this means that in most cases, to add context to jobs, we don't -need to do anything. - -There are however some instances when there would be no context -present when the job is scheduled, or the context that is present is -likely to be incorrect. For these instances, we've added Rubocop rules -to draw attention and avoid incorrect metadata in our logs. - -As with most our cops, there are perfectly valid reasons for disabling -them. In this case it could be that the context from the request is -correct. Or maybe you've specified a context already in a way that -isn't picked up by the cops. In any case, leave a code comment -pointing to which context to use when disabling the cops. - -When you do provide objects to the context, make sure that the -route for namespaces and projects is pre-loaded. This can be done by using -the `.with_route` scope defined on all `Routable`s. - -### Cron workers - -The context is automatically cleared for workers in the cronjob queue -(`include CronjobQueue`), even when scheduling them from -requests. We do this to avoid incorrect metadata when other jobs are -scheduled from the cron worker. - -Cron workers themselves run instance wide, so they aren't scoped to -users, namespaces, projects, or other resources that should be added to -the context. - -However, they often schedule other jobs that _do_ require context. - -That is why there needs to be an indication of context somewhere in -the worker. This can be done by using one of the following methods -somewhere within the worker: - -1. Wrap the code that schedules jobs in the `with_context` helper: - - ```ruby - def perform - deletion_cutoff = Gitlab::CurrentSettings - .deletion_adjourned_period.days.ago.to_date - projects = Project.with_route.with_namespace - .aimed_for_deletion(deletion_cutoff) - - projects.find_each(batch_size: 100).with_index do |project, index| - delay = index * INTERVAL - - with_context(project: project) do - AdjournedProjectDeletionWorker.perform_in(delay, project.id) - end - end - end - ``` - -1. Use the a batch scheduling method that provides context: - - ```ruby - def schedule_projects_in_batch(projects) - ProjectImportScheduleWorker.bulk_perform_async_with_contexts( - projects, - arguments_proc: -> (project) { project.id }, - context_proc: -> (project) { { project: project } } - ) - end - ``` - - Or, when scheduling with delays: - - ```ruby - diffs.each_batch(of: BATCH_SIZE) do |diffs, index| - DeleteDiffFilesWorker - .bulk_perform_in_with_contexts(index * 5.minutes, - diffs, - arguments_proc: -> (diff) { diff.id }, - context_proc: -> (diff) { { project: diff.merge_request.target_project } }) - end - ``` - -### Jobs scheduled in bulk - -Often, when scheduling jobs in bulk, these jobs should have a separate -context rather than the overarching context. - -If that is the case, `bulk_perform_async` can be replaced by the -`bulk_perform_async_with_context` helper, and instead of -`bulk_perform_in` use `bulk_perform_in_with_context`. - -For example: - -```ruby - ProjectImportScheduleWorker.bulk_perform_async_with_contexts( - projects, - arguments_proc: -> (project) { project.id }, - context_proc: -> (project) { { project: project } } - ) -``` - -Each object from the enumerable in the first argument is yielded into 2 -blocks: - -- The `arguments_proc` which needs to return the list of arguments the - job needs to be scheduled with. - -- The `context_proc` which needs to return a hash with the context - information for the job. - -## Arguments logging - -As of GitLab 13.6, Sidekiq job arguments are logged by default, unless [`SIDEKIQ_LOG_ARGUMENTS`](../administration/troubleshooting/sidekiq.md#log-arguments-to-sidekiq-jobs) -is disabled. - -By default, the only arguments logged are numeric arguments, because -arguments of other types could contain sensitive information. To -override this, use `loggable_arguments` inside a worker with the indexes -of the arguments to be logged. (Numeric arguments do not need to be -specified here.) - -For example: - -```ruby -class MyWorker - include ApplicationWorker - - loggable_arguments 1, 3 - - # object_id will be logged as it's numeric - # string_a will be logged due to the loggable_arguments call - # string_b will be filtered from logs - # string_c will be logged due to the loggable_arguments call - def perform(object_id, string_a, string_b, string_c) - end -end -``` - -## Tests - -Each Sidekiq worker must be tested using RSpec, just like any other class. These -tests should be placed in `spec/workers`. - -## Sidekiq Compatibility across Updates - -Keep in mind that the arguments for a Sidekiq job are stored in a queue while it -is scheduled for execution. During a online update, this could lead to several -possible situations: - -1. An older version of the application publishes a job, which is executed by an - upgraded Sidekiq node. -1. A job is queued before an upgrade, but executed after an upgrade. -1. A job is queued by a node running the newer version of the application, but - executed on a node running an older version of the application. - -### Adding new workers - -On GitLab.com, we [do not currently have a Sidekiq deployment in the -canary stage](https://gitlab.com/gitlab-org/gitlab/-/issues/19239). This -means that a new worker than can be scheduled from an HTTP endpoint may -be scheduled from canary but not run on Sidekiq until the full -production deployment is complete. This can be several hours later than -scheduling the job. For some workers, this will not be a problem. For -others - particularly [latency-sensitive -jobs](#latency-sensitive-jobs) - this will result in a poor user -experience. - -This only applies to new worker classes when they are first introduced. -As we recommend [using feature flags](feature_flags/) as a general -development process, it's best to control the entire change (including -scheduling of the new Sidekiq worker) with a feature flag. - -### Changing the arguments for a worker - -Jobs need to be backward and forward compatible between consecutive versions -of the application. Adding or removing an argument may cause problems -during deployment before all Rails and Sidekiq nodes have the updated code. - -#### Deprecate and remove an argument - -**Before you remove arguments from the `perform_async` and `perform` methods.**, deprecate them. The -following example deprecates and then removes `arg2` from the `perform_async` method: - -1. Provide a default value (usually `nil`) and use a comment to mark the - argument as deprecated in the coming minor release. (Release M) - - ```ruby - class ExampleWorker - # Keep arg2 parameter for backwards compatibility. - def perform(object_id, arg1, arg2 = nil) - # ... - end - end - ``` - -1. One minor release later, stop using the argument in `perform_async`. (Release M+1) - - ```ruby - ExampleWorker.perform_async(object_id, arg1) - ``` - -1. At the next major release, remove the value from the worker class. (Next major release) - - ```ruby - class ExampleWorker - def perform(object_id, arg1) - # ... - end - end - ``` - -#### Add an argument - -There are two options for safely adding new arguments to Sidekiq workers: - -1. Set up a [multi-step deployment](#multi-step-deployment) in which the new argument is first added to the worker. -1. Use a [parameter hash](#parameter-hash) for additional arguments. This is perhaps the most flexible option. - -##### Multi-step deployment - -This approach requires multiple releases. - -1. Add the argument to the worker with a default value (Release M). - - ```ruby - class ExampleWorker - def perform(object_id, new_arg = nil) - # ... - end - end - ``` - -1. Add the new argument to all the invocations of the worker (Release M+1). - - ```ruby - ExampleWorker.perform_async(object_id, new_arg) - ``` - -1. Remove the default value (Release M+2). - - ```ruby - class ExampleWorker - def perform(object_id, new_arg) - # ... - end - end - ``` - -##### Parameter hash - -This approach doesn't require multiple releases if an existing worker already -uses a parameter hash. - -1. Use a parameter hash in the worker to allow future flexibility. - - ```ruby - class ExampleWorker - def perform(object_id, params = {}) - # ... - end - end - ``` - -### Removing workers - -Try to avoid removing workers and their queues in minor and patch -releases. - -During online update instance can have pending jobs and removing the queue can -lead to those jobs being stuck forever. If you can't write migration for those -Sidekiq jobs, please consider removing the worker in a major release only. - -### Renaming queues - -For the same reasons that removing workers is dangerous, care should be taken -when renaming queues. - -When renaming queues, use the `sidekiq_queue_migrate` helper migration method -in a **post-deployment migration**: - -```ruby -class MigrateTheRenamedSidekiqQueue < Gitlab::Database::Migration[1.0] - def up - sidekiq_queue_migrate 'old_queue_name', to: 'new_queue_name' - end - - def down - sidekiq_queue_migrate 'new_queue_name', to: 'old_queue_name' - end -end - -``` - -You must rename the queue in a post-deployment migration not in a normal -migration. Otherwise, it runs too early, before all the workers that -schedule these jobs have stopped running. See also [other examples](post_deployment_migrations.md#use-cases). +<!-- This redirect file can be deleted after <2022-04-13>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/snowplow/implementation.md b/doc/development/snowplow/implementation.md index 439485c9e73..d35413cfd5f 100644 --- a/doc/development/snowplow/implementation.md +++ b/doc/development/snowplow/implementation.md @@ -18,21 +18,27 @@ to track custom events. For the recommended frontend tracking implementation, see [Usage recommendations](#usage-recommendations). +Structured events and page views include the [`gitlab_standard`](schemas.md#gitlab_standard) +context, using the `window.gl.snowplowStandardContext` object which includes +[default data](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/views/layouts/_snowplow.html.haml) +as base. This object can be modified for any subsequent structured event fired, +although it's not recommended. + Tracking implementations must have an `action` and a `category`. You can provide additional -categories from the [structured event taxonomy](index.md#structured-event-taxonomy) with an `extra` object -that accepts key-value pairs. +properties from the [structured event taxonomy](index.md#structured-event-taxonomy), in +addition to an `extra` object that accepts key-value pairs. -| Field | Type | Default value | Description | +| Property | Type | Default value | Description | |:-----------|:-------|:---------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `category` | string | `document.body.dataset.page` | Page or subsection of a page in which events are captured. | -| `action` | string | generic | Action the user is taking. Clicks must be `click` and activations must be `activate`. For example, focusing a form field is `activate_form_input`, and clicking a button is `click_button`. | -| `data` | object | `{}` | Additional data such as `label`, `property`, `value`, `context` as described in [Structured event taxonomy](index.md#structured-event-taxonomy), and `extra` (key-value pairs object). | +| `action` | string | `'generic'` | Action the user is taking. Clicks must be `click` and activations must be `activate`. For example, focusing a form field is `activate_form_input`, and clicking a button is `click_button`. | +| `data` | object | `{}` | Additional data such as `label`, `property`, `value` as described in [Structured event taxonomy](index.md#structured-event-taxonomy), `context` for custom contexts, and `extra` (key-value pairs object). | ### Usage recommendations - Use [data attributes](#implement-data-attribute-tracking) on HTML elements that emit `click`, `show.bs.dropdown`, or `hide.bs.dropdown` events. -- Use the [Vue mixin](#implement-vue-component-tracking) for tracking custom events, or if the supported events for data attributes are not propagating. -- Use the [tracking class](#implement-raw-javascript-tracking) when tracking raw JavaScript files. +- Use the [Vue mixin](#implement-vue-component-tracking) for tracking custom events, or if the supported events for data attributes are not propagating. For example, clickable components that don't emit `click`. +- Use the [tracking class](#implement-raw-javascript-tracking) when tracking in vanilla JavaScript files. ### Implement data attribute tracking @@ -42,6 +48,9 @@ The following example shows `data-track-*` attributes assigned to a button: ```haml %button.btn{ data: { track: { action: "click_button", label: "template_preview", property: "my-template" } } } + +// or +// %button.btn{ data: { track_action: "click_button", track_label: "template_preview", track_property: "my-template" } } ``` ```html @@ -62,7 +71,7 @@ The following example shows `data-track-*` attributes assigned to a button: | `data-track-property` | false | Any additional property of the element, or object being acted on. | | `data-track-value` | false | Describes a numeric value (decimal) directly related to the event. This could be the value of an input. For example, `10` when clicking `internal` visibility. If omitted, this is the element's `value` property or `undefined`. For checkboxes, the default value is the element's checked attribute or `0` when unchecked. | | `data-track-extra` | false | A key-value pair object passed as a valid JSON string. This attribute is added to the `extra` property in our [`gitlab_standard`](schemas.md#gitlab_standard) schema. | -| `data-track-context` | false | The `context` as described in our [Structured event taxonomy](index.md#structured-event-taxonomy). | +| `data-track-context` | false | To append a custom context object, passed as a valid JSON string. | #### Event listeners @@ -74,12 +83,19 @@ If click events stop propagating, you must implement listeners and [Vue componen #### Helper methods -Use the following Ruby helper: +You can use the following Ruby helper: ```ruby tracking_attrs(label, action, property) # { data: { track_label... } } +``` +You can also use it on HAML templates: + +```haml %button{ **tracking_attrs('main_navigation', 'click_button', 'navigation') } + +// When adding additional data +// %button{ data: { platform: "...", **tracking_attrs('main_navigation', 'click_button', 'navigation') } } ``` If you use the GitLab helper method [`nav_link`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/helpers/tab_helper.rb#L76), you must wrap `html_options` under the `html_options` keyword argument. If you @@ -91,7 +107,7 @@ use the `ActionView` helper method [`link_to`](https://api.rubyonrails.org/v5.2. track_action: "click_button" }) # Good -= nav_link(controller: ['dashboard/groups', 'explore/groups'], html_options: { data: { track_label: += nav_link(controller: ['dashboard/groups', 'explore/groups'], html_options: { data: { track_label: "explore_groups", track_action: "click_button" } }) # Good (other helpers) @@ -101,63 +117,64 @@ track_action: "click_button" }) ### Implement Vue component tracking -For custom event tracking, use a Vue `mixin` in components. Vue `mixin` exposes the `Tracking.event` -static method and the `track` method. You can specify tracking options in `data` or `computed`. -These options override any defaults and allow the values to be dynamic from props or based on state. - -Several default options are passed when an event is tracked from the component: +For custom event tracking, use the [Vue mixin](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/tracking/tracking.js#L207). It exposes `Tracking.event` as the `track` method. +You can specify tracking options by creating a `tracking` data object or +computed property, and as a second parameter: `this.track('click_button', opts)`. +These options override any defaults and allow the values to be dynamic from props or based on state: -- `category`: If you don't specify, by default `document.body.dataset.page` is used. -- `label` -- `property` -- `value` +| Property | Type | Default | Example | +| -- | -- | -- | -- | +| `category` | string | `document.body.dataset.page` | `'code_quality_walkthrough'` | +| `label` | string | `''` | `'process_start_button'` | +| `property` | string | `''` | `'asc'` or `'desc'` | +| `value` | integer | `undefined` | `0`, `1`, `500` | +| `extra` | object | `{}` | `{ selectedVariant: this.variant }` | To implement Vue component tracking: -1. Import the `Tracking` library and request a `mixin`: +1. Import the `Tracking` library and call the `mixin` method: ```javascript import Tracking from '~/tracking'; - const trackingMixin = Tracking.mixin; - ``` -1. Provide categories to track the event from the component. For example, to track all events in a -component with a label, use the `label` category: + const trackingMixin = Tracking.mixin(); - ```javascript - import Tracking from '~/tracking'; - const trackingMixin = Tracking.mixin({ label: 'right_sidebar' }); + // Optionally provide default properties + // const trackingMixin = Tracking.mixin({ label: 'right_sidebar' }); ``` -1. In the component, declare the Vue `mixin`: +1. Use the mixin in the component: ```javascript export default { mixins: [trackingMixin], - // ...[component implementation]... + // Or + // mixins: [Tracking.mixin()], + // mixins: [Tracking.mixin({ label: 'right_sidebar' })], + data() { return { expanded: false, - tracking: { - label: 'left_sidebar', - }, }; }, }; ``` -1. To receive event data as a tracking object or computed property: - - Declare it in the `data` function. Use a `tracking` object when default event properties are dynamic or provided at runtime: +1. You can specify tracking options in by creating a `tracking` data object +or computed property: ```javascript export default { name: 'RightSidebar', + mixins: [Tracking.mixin()], + data() { return { + expanded: false, + variant: '', tracking: { label: 'right_sidebar', - // category: '', // property: '', // value: '', // experiment: '', @@ -165,18 +182,28 @@ component with a label, use the `label` category: }, }; }, + + // Or + // computed: { + // tracking() { + // return { + // property: this.variant, + // extra: { expanded: this.expanded }, + // }; + // }, + // }, }; ``` - - Declare it in the event data in the `track` function. This object merges with any previously provided options: +1. Call the `track` method. Tracking options can be passed as the second parameter: - ```javascript - this.track('click_button', { - label: 'right_sidebar', - }); - ``` + ```javascript + this.track('click_button', { + label: 'right_sidebar', + }); + ``` -1. Optional. Use the `track` method in a template: + Or use the `track` method in the template: ```html <template> @@ -185,55 +212,67 @@ component with a label, use the `label` category: <div v-if="expanded"> <p>Hello world!</p> - <button @click="track('click_action')">Track another event</button> + <button @click="track('click_button')">Track another event</button> </div> </div> </template> ``` -The following example shows an implementation of Vue component tracking: +#### Testing example ```javascript export default { - name: 'RightSidebar', - mixins: [Tracking.mixin({ label: 'right_sidebar' })], + name: 'CountDropdown', + + mixins: [Tracking.mixin({ label: 'count_dropdown' })], + data() { return { - expanded: false, + variant: 'counter', + count: 0, }; }, + methods: { - toggle() { - this.expanded = !this.expanded; - // Additional data will be merged, like `value` below - this.track('click_toggle', { value: Number(this.expanded) }); - } - } + handleChange({ target }) { + const { variant } = this; + + this.count = Number(target.value); + + this.track('change_value', { + value: this.count, + extra: { variant } + }); + }, + }, }; ``` -#### Testing example - ```javascript import { mockTracking } from 'helpers/tracking_helper'; // mockTracking(category, documentOverride, spyMethod) -describe('RightSidebar.vue', () => { +describe('CountDropdown.vue', () => { let trackingSpy; let wrapper; + ... + beforeEach(() => { trackingSpy = mockTracking(undefined, wrapper.element, jest.spyOn); }); - const findToggle = () => wrapper.find('[data-testid="toggle"]'); + const findDropdown = () => wrapper.find('[data-testid="dropdown"]'); - it('tracks turning off toggle', () => { - findToggle().trigger('click'); + it('tracks change event', () => { + const dropdown = findDropdown(); + dropdown.element.value = 30; + dropdown.trigger('change'); - expect(trackingSpy).toHaveBeenCalledWith(undefined, 'click_toggle', { - label: 'right_sidebar', - value: 0, + expect(trackingSpy).toHaveBeenCalledWith(undefined, 'change_value', { + value: 30, + label: 'count_dropdown', + extra: { variant: 'counter' }, }); }); }); @@ -241,7 +280,8 @@ describe('RightSidebar.vue', () => { ### Implement raw JavaScript tracking -To call custom event tracking and instrumentation directly from the JavaScript file, call the `Tracking.event` static function. +To track from a vanilla JavaScript file, use the `Tracking.event` static function +(calls [`dispatchSnowplowEvent`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/tracking/dispatch_snowplow_event.js)). The following example demonstrates tracking a click on a button by manually calling `Tracking.event`. @@ -251,7 +291,7 @@ import Tracking from '~/tracking'; const button = document.getElementById('create_from_template_button'); button.addEventListener('click', () => { - Tracking.event('dashboard:projects:index', 'click_button', { + Tracking.event(undefined, 'click_button', { label: 'create_from_template', property: 'template_preview', extra: { diff --git a/doc/development/snowplow/index.md b/doc/development/snowplow/index.md index dbc7a25075f..29f4514a21e 100644 --- a/doc/development/snowplow/index.md +++ b/doc/development/snowplow/index.md @@ -72,6 +72,7 @@ sequenceDiagram GitLab.com Snowplow Collector ->> GitLab.com Snowplow Collector: Write to disk end GitLab.com Snowplow Collector ->> S3 Bucket: Kinesis Firehose + Note over GitLab.com Snowplow Collector, S3 Bucket: Pseudonymization S3 Bucket->>Snowflake DW: Import data Snowflake DW->>Snowflake DW: Transform data using dbt Snowflake DW->>Sisense Dashboards: Data available for querying @@ -166,11 +167,24 @@ LIMIT 100 Snowplow JavaScript adds [web-specific parameters](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/snowplow-tracker-protocol/#Web-specific_parameters) to all web events by default. +## Snowplow monitoring + +For different stages in the processing pipeline, there are several tools that monitor Snowplow events tracking: + +- [Product Intelligence Grafana dashboard](https://dashboards.gitlab.net/d/product-intelligence-main/product-intelligence-product-intelligence?orgId=1) monitors backend events sent from GitLab.com instance to collectors fleet. This dashboard provides information about: + - The number of events that successfully reach Snowplow collectors. + - The number of events that failed to reach Snowplow collectors. + - The number of backend events that were sent. +- [AWS CloudWatch dashboard](https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#dashboards:name=SnowPlow;start=P3D) monitors the state of the events processing pipeline. The pipeline starts from Snowplow collectors, through to enrichers and pseudonymization, and up to persistence on S3 bucket from which events are imported to Snowflake Data Warehouse. To view this dashboard AWS access is required, follow this [instruction](https://gitlab.com/gitlab-org/growth/product-intelligence/snowplow-pseudonymization#monitoring) if you are interested in getting one. +- [SiSense dashboard](https://app.periscopedata.com/app/gitlab/417669/Snowplow-Summary-Dashboard) provides information about the number of good and bad events imported into the Data Warehouse, in addition to the total number of imported Snowplow events. + +For more information, see this [video walk-through](https://www.youtube.com/watch?v=NxPS0aKa_oU). + ## Related topics - [Snowplow data structure](https://docs.snowplowanalytics.com/docs/understanding-your-pipeline/canonical-event/) - [Our Iglu schema registry](https://gitlab.com/gitlab-org/iglu) -- [List of events used in our codebase (Event Dictionary)](https://metrics.gitlab.com/snowplow.html) +- [List of events used in our codebase (Event Dictionary)](https://metrics.gitlab.com/snowplow/) - [Product Intelligence Guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/) - [Service Ping Guide](../service_ping/index.md) - [Product Intelligence Direction](https://about.gitlab.com/direction/product-intelligence/) diff --git a/doc/development/snowplow/review_guidelines.md b/doc/development/snowplow/review_guidelines.md index 69fad1794e2..0359636380b 100644 --- a/doc/development/snowplow/review_guidelines.md +++ b/doc/development/snowplow/review_guidelines.md @@ -14,7 +14,7 @@ general best practices for code reviews, refer to our [code review guide](../cod ## Resources for reviewers - [Snowplow Guide](index.md) -- [Event Dictionary](https://metrics.gitlab.com/snowplow.html) +- [Event Dictionary](https://metrics.gitlab.com/snowplow/) ## Review process diff --git a/doc/development/snowplow/schemas.md b/doc/development/snowplow/schemas.md index eb57e7d98a5..63864c9329b 100644 --- a/doc/development/snowplow/schemas.md +++ b/doc/development/snowplow/schemas.md @@ -12,19 +12,23 @@ This page provides Snowplow schema reference for GitLab events. We are including the [`gitlab_standard` schema](https://gitlab.com/gitlab-org/iglu/-/blob/master/public/schemas/com.gitlab/gitlab_standard/jsonschema/) with every event. See [Standardize Snowplow Schema](https://gitlab.com/groups/gitlab-org/-/epics/5218) for details. -The [`StandardContext`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/tracking/standard_context.rb) class represents this schema in the application. +The [`StandardContext`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/tracking/standard_context.rb) +class represents this schema in the application. Some properties are automatically populated for [frontend](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/views/layouts/_snowplow.html.haml) +events. -| Field Name | Required | Type | Description | -|----------------|---------------------|-----------------------|---------------------------------------------------------------------------------------------| -| `project_id` | **{dotted-circle}** | integer | | -| `namespace_id` | **{dotted-circle}** | integer | | -| `user_id` | **{dotted-circle}** | integer | User database record ID attribute. This file undergoes a pseudonymization process at the collector level. | -| `context_generated_at` | **{dotted-circle}** | string (date time format) | Timestamp indicating when context was generated. | -| `environment` | **{check-circle}** | string (max 32 chars) | Name of the source environment, such as `production` or `staging` | -| `source` | **{check-circle}** | string (max 32 chars) | Name of the source application, such as `gitlab-rails` or `gitlab-javascript` | -| `plan` | **{dotted-circle}** | string (max 32 chars) | Name of the plan for the namespace, such as `free`, `premium`, or `ultimate`. Automatically picked from the `namespace`. | -| `google_analytics_id` | **{dotted-circle}** | string (max 32 chars) | Google Analytics ID, present when set from our marketing sites. | -| `extra` | **{dotted-circle}** | JSON | Any additional data associated with the event, in the form of key-value pairs | +| Field Name | Required | Default value | Type | Description | +|----------------|:-------------------:|-----------------------|--|---------------------------------------------------------------------------------------------| +| `project_id` | **{dotted-circle}** | Current project ID * | integer | | +| `namespace_id` | **{dotted-circle}** | Current group/namespace ID * | integer | | +| `user_id` | **{dotted-circle}** | Current user ID * | integer | User database record ID attribute. This file undergoes a pseudonymization process at the collector level. | +| `context_generated_at` | **{dotted-circle}** | Current timestamp | string (date time format) | Timestamp indicating when context was generated. | +| `environment` | **{check-circle}** | Current environment | string (max 32 chars) | Name of the source environment, such as `production` or `staging` | +| `source` | **{check-circle}** | Event source | string (max 32 chars) | Name of the source application, such as `gitlab-rails` or `gitlab-javascript` | +| `plan` | **{dotted-circle}** | Current namespace plan * | string (max 32 chars) | Name of the plan for the namespace, such as `free`, `premium`, or `ultimate`. Automatically picked from the `namespace`. | +| `google_analytics_id` | **{dotted-circle}** | GA ID value * | string (max 32 chars) | Google Analytics ID, present when set from our marketing sites. | +| `extra` | **{dotted-circle}** | | JSON | Any additional data associated with the event, in the form of key-value pairs | + +_\* Default value present for frontend events only_ ## Default Schema diff --git a/doc/development/snowplow/troubleshooting.md b/doc/development/snowplow/troubleshooting.md new file mode 100644 index 00000000000..75c8b306a67 --- /dev/null +++ b/doc/development/snowplow/troubleshooting.md @@ -0,0 +1,50 @@ +--- +stage: Growth +group: Product Intelligence +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Troubleshooting + +## Good events drop + +### Symptoms + +You will be alarmed via a [Sisense alert](https://app.periscopedata.com/app/gitlab/alert/Volume-of-Snowplow-Good-events/5a5f80ef34fe450da5ebb84eaa84067f/edit) that is sent to `#g_product_intelligence` Slack channel + +### Locating the problem + +First you need to identify at which stage in Snowplow the data pipeline the drop is occurring. +Start at [Snowplow dashboard](https://console.aws.amazon.com/systems-manager/resource-groups/cloudwatch?dashboard=SnowPlow®ion=us-east-1#) on CloudWatch, +if you do not have access to CloudWatch you need to create an [access request issue](https://gitlab.com/gitlab-com/team-member-epics/access-requests/-/issues/9730) first. +While on CloudWatch dashboard set time range to last 4 weeks, to get better picture of system characteristics over time. Than visit following charts: + +1. `ELB New Flow Count` and `Collector Auto Scaling Group Network In/Out` - they show in order: number of connections to collectors via load balancers and data volume (in bytes) processed by collectors. If there is drop visible there, it means less events were fired from the GitLab application. Proceed to [application layer guide](#troubleshooting-gitlab-application-layer) for more details +1. `Firehose Records to S3` - it shows how many event records were saved to S3 bucket, if there was drop on this chart but not on the charts from 1. it means that problem is located at AWS infrastructure layer, please refer to [AWS layer guide](#troubleshooting-aws-layer) +1. If drop wasn't visible on any of previous charts it means that probelm is at data warehouse layer, please refer to [data warehouse layer guide](#troubleshooting-data-warehouse-layer) + +### Troubleshooting GitLab application layer + +Drop occurring at application layer can be symptom of some issue, but it might be also a result of normal application lifecycle, intended changes done to product intelligence or experiments tracking +or even a result of a public holiday in some regions of the world with a larger user-base. To verify if there is an underlying problem to solve, you can check following things: + +1. Check `about.gitlab.com` website traffic on [Google Analytics](https://analytics.google.com/) to verify if some public holiday might impact overall use of GitLab system + 1. You may require to open an access request for Google Analytics access first eg: [access request internal issue](https://gitlab.com/gitlab-com/team-member-epics/access-requests/-/issues/1772) +1. Plot `select date(dvce_created_tstamp) , event , count(*) from legacy.snowplow_unnested_events_90 where dvce_created_tstamp > '2021-06-15' and dvce_created_tstamp < '2021-07-10' group by 1 , 2 order by 1 , 2` in SiSense to see what type of events was responsible for drop +1. Plot `select date(dvce_created_tstamp) ,se_category , count(*) from legacy.snowplow_unnested_events_90 where dvce_created_tstamp > '2021-06-15' and dvce_created_tstamp < '2021-07-31' and event = 'struct' group by 1 , 2 order by 1, 2` what events recorded the biggest drops in suspected category +1. Check if there was any MR merged that might cause reduction in reported events, pay an attention to ~"product intelligence" and ~"growth experiment" labeled MRs +1. Check (via [Grafana explore tab](https://dashboards.gitlab.net/explore) ) following Prometheus counters `gitlab_snowplow_events_total`, `gitlab_snowplow_failed_events_total` and `gitlab_snowplow_successful_events_total` to see how many events were fired correctly from GitLab.com. Example query to use `sum(rate(gitlab_snowplow_successful_events_total{env="gprd"}[5m])) / sum(rate(gitlab_snowplow_events_total{env="gprd"}[5m]))` would chart rate at which number of good events rose in comparison to events sent in total. If number drops from 1 it means that problem might be in communication between GitLab and AWS collectors fleet. +1. Check [logs in Kibana](https://log.gprd.gitlab.net/app/discover#) and filter with `{ "query": { "match_phrase": { "json.message": "failed to be reported to collector at" } } }` if there are some failed events logged + +For results about an investigation conducted into an unexpected drop in snowplow events volume, see [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/335206). + +### Troubleshooting AWS layer + +Already conducted investigations: + +- [Steep decrease of Snowplow page views](https://gitlab.com/gitlab-org/gitlab/-/issues/268009) +- [`snowplow.trx.gitlab.net` unreachable](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/5073) + +### Troubleshooting data warehouse layer + +Reach out to [Data team](https://about.gitlab.com/handbook/business-technology/data-team) to ask about current state of data warehouse. On their handbook page there is a [section with contact details](https://about.gitlab.com/handbook/business-technology/data-team/#how-to-connect-with-us) diff --git a/doc/development/stage_group_dashboards.md b/doc/development/stage_group_dashboards.md index 88e9141574e..744d049f72d 100644 --- a/doc/development/stage_group_dashboards.md +++ b/doc/development/stage_group_dashboards.md @@ -64,8 +64,8 @@ component can have 2 indicators: [customize the request Apdex](application_slis/rails_request_apdex.md), this new Apdex measurement is not yet part of the error budget. - For Sidekiq job execution, the threshold depends on the [job - urgency](sidekiq_style_guide.md#job-urgency). It is + For Sidekiq job execution, the threshold depends on the + [job urgency](sidekiq/worker_attributes.md#job-urgency). It is [currently](https://gitlab.com/gitlab-com/runbooks/-/blob/f22f40b2c2eab37d85e23ccac45e658b2c914445/metrics-catalog/services/lib/sidekiq-helpers.libsonnet#L25-38) **10 seconds** for high-urgency jobs and **5 minutes** for other jobs. diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md index 543feaa967c..405ff40ef6a 100644 --- a/doc/development/testing_guide/end_to_end/best_practices.md +++ b/doc/development/testing_guide/end_to_end/best_practices.md @@ -363,11 +363,11 @@ after(:all) do end ``` -## Tag tests that require the Administrator role +## Tag tests that require administrator access -We don't run tests that require the Administrator role against our Production environments. +We don't run tests that require administrator access against our Production environments. -When you add a new test that requires the Administrator role, apply the RSpec metadata `:requires_admin` so that the test will not be included in the test suites executed against Production and other environments on which we don't want to run those tests. +When you add a new test that requires administrator access, apply the RSpec metadata `:requires_admin` so that the test will not be included in the test suites executed against Production and other environments on which we don't want to run those tests. When running tests locally or configuring a pipeline, the environment variable `QA_CAN_TEST_ADMIN_FEATURES` can be set to `false` to skip tests that have the `:requires_admin` tag. diff --git a/doc/development/testing_guide/end_to_end/execution_context_selection.md b/doc/development/testing_guide/end_to_end/execution_context_selection.md index 8456cfa8314..0fdcf0c8c3b 100644 --- a/doc/development/testing_guide/end_to_end/execution_context_selection.md +++ b/doc/development/testing_guide/end_to_end/execution_context_selection.md @@ -120,3 +120,5 @@ Similarly to specifying that a test should only run against a specific environme test only when it runs against a specific environment. The syntax is exactly the same, except that the `only: { ... }` hash is nested in the [`quarantine: { ... }`](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests) hash. For example, `quarantine: { only: { subdomain: :staging } }` only quarantines the test when run against `staging`. + +The quarantine feature can be explicitly disabled with the `DISABLE_QUARANTINE` environment variable. This can be useful when running tests locally. diff --git a/doc/development/testing_guide/end_to_end/feature_flags.md b/doc/development/testing_guide/end_to_end/feature_flags.md index 48157a961e1..238b1ccd8f9 100644 --- a/doc/development/testing_guide/end_to_end/feature_flags.md +++ b/doc/development/testing_guide/end_to_end/feature_flags.md @@ -144,6 +144,14 @@ end After the feature flag is removed, clean up the resource class and delete the variable. All methods should use the condition procedures of the now-default state. +## Managing flakiness due to caching + +All application settings, and all feature flags, are cached inside GitLab for one minute. +All caching is disabled during testing, except on static environments. + +When a test changes a feature flag, it can cause flaky behavior if elements are visible only with an +active feature flag. To circumvent this behavior, add a wait for elements behind a feature flag. + ## Running a scenario with a feature flag enabled It's also possible to run an entire scenario with a feature flag enabled, without having to edit diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index 1fc9bc8258a..dc989acbdcc 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -101,19 +101,19 @@ This is due to technical limitations in the GitLab permission model: the ability against a protected branch is controlled by the ability to push/merge to this branch. This means that for developers to be able to trigger a pipeline for the default branch in `gitlab-org/omnibus-gitlab`/`gitlab-org/gitlab-qa`, they would need to have the -[Maintainer role](../../../user/permissions.md) for those projects. +Maintainer role for those projects. For security reasons and implications, we couldn't open up the default branch to all the Developers. Hence we created these mirrors where Developers and Maintainers are allowed to push/merge to the default branch. This problem was discovered in <https://gitlab.com/gitlab-org/gitlab-qa/-/issues/63#note_107175160> and the "mirror" work-around was suggested in <https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4717>. A feature proposal to segregate access control regarding running pipelines from ability to push/merge was also created at <https://gitlab.com/gitlab-org/gitlab/-/issues/24585>. -#### With Pipeline for Merged Results +#### With merged results pipelines -In a Pipeline for Merged Results, the pipeline runs on a new ref that contains the merge result of the source and target branch. +In a merged results pipeline, the pipeline runs on a new ref that contains the merge result of the source and target branch. However, this ref is not available to the `gitlab-qa-mirror` pipeline. -For this reason, the end-to-end tests on a Pipeline for Merged Results would use the head of the merge request source branch. +For this reason, the end-to-end tests on a merged results pipeline would use the head of the merge request source branch. ```mermaid graph LR @@ -126,7 +126,7 @@ A --> C B --> C A --> E["E2E tests"] -C --> D["Pipeline for merged results"] +C --> D["Merged results pipeline"] ``` ##### Running custom tests diff --git a/doc/development/testing_guide/end_to_end/resources.md b/doc/development/testing_guide/end_to_end/resources.md index abba1d51f07..e2b005d8a1b 100644 --- a/doc/development/testing_guide/end_to_end/resources.md +++ b/doc/development/testing_guide/end_to_end/resources.md @@ -431,11 +431,32 @@ module QA def validate_reuse_preconditions raise ResourceReuseError unless reused_name_valid? end + + # Internally we identify an instance of a reusable resource by a unique value of `@reuse_as`, but in GitLab the + # resource has one or more attributes that must also be unique. This method lists those attributes and allows the + # test framework to check that each instance of a reusable resource has values that match the associated values + # in Gitlab. + def unique_identifiers + [:name, :path] + end end end end ``` +Reusable resources aren't removed immediately when `remove_via_api!` is called. Instead, they're removed after the test +suite completes. To do so each class must be registered with `QA::Resource::ReusableCollection` in `qa/spec/spec_helper.rb` +as in the example below. Registration allows `QA::Resource::ReusableCollection` to keep track of each instance of each +registered class, and to delete them all in the `config.after(:suite)` hook. + +```ruby +config.before(:suite) do |suite| + QA::Resource::ReusableCollection.register_resource_classes do |collection| + QA::Resource::ReusableProject.register(collection) + end +end +``` + Consider some examples of how a reusable resource is used: ```ruby @@ -488,6 +509,65 @@ end # match the name specified when the project was first fabricated. ``` +### Validating reusable resources + +Reusable resources can speed up test suites by avoiding the cost of creating the same resource again and again. However, +that can cause problems if a test makes changes to a resource that prevent it from being reused as expected by later +tests. That can lead to order-dependent test failures that can be difficult to troubleshoot. + +For example, the default project created by `QA::Resource::ReusableProject` has `auto_devops_enabled` set to `false` +(inherited from `QA::Resource::Project`). If a test reuses that project and enables Auto DevOps, subsequent tests that reuse +the project will fail if they expect Auto DevOps to be disabled. + +We try to avoid that kind of trouble by validating reusable resources after a test suite. If the environment variable +`QA_VALIDATE_RESOURCE_REUSE` is set to `true` the test framework will check each reusable resource to verify that none +of the attributes they were created with have been changed. It does that by creating a new resource using the same +attributes that were used to create the original resource. It then compares the new resource to the original and raises +an error if any attributes don't match. + +#### Implementation + +When you implement a new type of reusable resource there are two `private` methods you must implement so the resource +can be validated. They are: + +- `reference_resource`: creates a new instance of the resource that can be compared with the one that was used during the tests. +- `unique_identifiers`: returns an array of attributes that allow the resource to be identified (e.g., name) and that are therefore +expected to differ when comparing the reference resource with the resource reused in the tests. + +The following example shows the implementation of those two methods in `QA::Resource::ReusableProject`. + +```ruby +# Creates a new project that can be compared to a reused project, using the attributes of the original. +# +# @return [QA::Resource] a new instance of Resource::ReusableProject that should be a copy of the original resource +def reference_resource + # These are the attributes that the reused resource was created with + attributes = self.class.resources[reuse_as][:attributes] + + # Two projects can't have the same path, and since we typically use the same value for the name and path, we assign + # a unique name and path to the reference resource. + name = "reference_resource_#{SecureRandom.hex(8)}_for_#{attributes.delete(:name)}" + + Project.fabricate_via_api! do |project| + self.class.resources[reuse_as][:attributes].each do |attribute_name, attribute_value| + project.instance_variable_set("@#{attribute_name}", attribute_value) if attribute_value + end + project.name = name + project.path = name + project.path_with_namespace = "#{project.group.full_path}/#{project.name}" + end +end + +# The attributes of the resource that should be the same whenever a test wants to reuse a project. +# +# @return [Array<Symbol>] the attribute names. +def unique_identifiers + # As noted above, path must be unique, and since we typically use the same value for both, we treat name and path + # as unique. These attributes are ignored when we compare the reference and reused resources. + [:name, :path] +end +``` + ## Where to ask for help? If you need more information, ask for help on `#quality` channel on Slack diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md index bb214976926..07f7e9582ee 100644 --- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md +++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md @@ -20,6 +20,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec | `:github` | The test requires a GitHub personal access token. | | `:group_saml` | The test requires a GitLab instance that has SAML SSO enabled at the group level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. | | `:instance_saml` | The test requires a GitLab instance that has SAML SSO enabled at the instance level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. | +| `:integrations` | This aims to test the available [integrations](../../../user/project/integrations/overview.md#integrations-listing). The test requires Docker to be installed in the run context. It will provision the containers and can be run against a local instance or using the `gitlab-qa` scenario `Test::Integration::Integrations` | | `:service_ping_disabled` | The test interacts with the GitLab configuration service ping at the instance level to turn admin setting service ping checkbox on or off. This tag will have the test run only in the `service_ping_disabled` job and must be paired with the `:orchestrated` and `:requires_admin` tags. | | `:jira` | The test requires a Jira Server. [GitLab-QA](https://gitlab.com/gitlab-org/gitlab-qa) provisions the Jira Server in a Docker container when the `Test::Integration::Jira` test scenario is run. | `:kubernetes` | The test includes a GitLab instance that is configured to be run behind an SSH tunnel, allowing a TLS-accessible GitLab. This test also includes provisioning of at least one Kubernetes cluster to test against. _This tag is often be paired with `:orchestrated`._ | diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md index ef3e0624395..7fb95769fc2 100644 --- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md +++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md @@ -141,6 +141,34 @@ docker stop gitlab-gitaly-cluster praefect postgres gitaly3 gitaly2 gitaly1 docker rm gitlab-gitaly-cluster praefect postgres gitaly3 gitaly2 gitaly1 ``` +## Tests that require a runner + +To execute tests that use a runner without errors, while creating the GitLab Docker instance the `--hostname` parameter in the Docker `run` command should be given a specific interface IP address or a non-loopback hostname accessible from the runner container. Having `localhost` (or `127.0.0.1`) as the GitLab hostname won't work (unless the GitLab Runner is created with the Docker network as `host`) + +Examples of tests which require a runner: + +- `qa/qa/specs/features/ee/browser_ui/13_secure/create_merge_request_with_secure_spec.rb` +- `qa/qa/specs/features/browser_ui/4_verify/runner/register_runner_spec.rb` + +Example: + +```shell +docker run \ + --detach \ + --hostname interface_ip_address \ + --publish 80:80 \ + --name gitlab \ + --restart always \ + --volume ~/ee_volume/config:/etc/gitlab \ + --volume ~/ee_volume/logs:/var/log/gitlab \ + --volume ~/ee_volume/data:/var/opt/gitlab \ + --shm-size 256m \ + gitlab/gitlab-ee:latest +``` + +Where `interface_ip_address` is your local network's interface IP, which you can find with the `ifconfig` command. +The same would apply to GDK running with the instance address as `localhost` too. + ## Guide to run and debug Monitor tests ### How to set up diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md index d2e68ea7715..09fc8f4d33e 100644 --- a/doc/development/testing_guide/flaky_tests.md +++ b/doc/development/testing_guide/flaky_tests.md @@ -14,14 +14,14 @@ eventually. ## Quarantined tests When a test frequently fails in `main`, -[a ~"master:broken" issue](https://about.gitlab.com/handbook/engineering/workflow/#broken-master) -should be created. +create [a ~"failure::flaky-test" issue](https://about.gitlab.com/handbook/engineering/workflow/#broken-master). + If the test cannot be fixed in a timely fashion, there is an impact on the -productivity of all the developers, so it should be placed in quarantine by -assigning the `:quarantine` metadata with the issue URL. +productivity of all the developers, so it should be quarantined by +assigning the `:quarantine` metadata with the issue URL, and add the `~"quarantined test"` label to the issue. ```ruby -it 'should succeed', quarantine: 'https://gitlab.com/gitlab-org/gitlab/-/issues/12345' do +it 'succeeds', quarantine: 'https://gitlab.com/gitlab-org/gitlab/-/issues/12345' do expect(response).to have_gitlab_http_status(:ok) end ``` @@ -32,23 +32,13 @@ This means it is skipped unless run with `--tag quarantine`: bin/rspec --tag quarantine ``` -**Before putting a test in quarantine, you should make sure that a -~"master:broken" issue exists for it so it doesn't stay in quarantine forever.** - Once a test is in quarantine, there are 3 choices: -- Should the test be fixed (that is, get rid of its flakiness)? -- Should the test be moved to a lower level of testing? -- Should the test be removed entirely (for example, because there's already a +- Fix the test (that is, get rid of its flakiness). +- Move the test to a lower level of testing. +- Remove the test entirely (for example, because there's already a lower-level test, or it's duplicating another same-level test, or it's testing - too much etc.)? - -### Quarantine tests on the CI - -Quarantined tests are run on the CI in dedicated jobs that are allowed to fail: - -- `rspec-pg-quarantine` (CE & EE) -- `rspec-pg-quarantine-ee` (EE only) + too much etc.). ## Automatic retries and flaky tests detection @@ -116,6 +106,7 @@ reproduction. - [Lazy loaded images can cause Capybara to mis-click](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18713) - [Triggering JS events before the event handlers are set up](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18742) - [Wait for the image to be lazy-loaded when asserting on a Markdown image's `src` attribute](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25408) +- [Avoid asserting against flash notice banners](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79432) #### Capybara viewport size related issues diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index eb35227a50c..d03a4976a8c 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -748,6 +748,8 @@ Consult the [official Jest docs](https://jestjs.io/docs/jest-object#mock-modules ## Running Frontend Tests +Before generating fixtures, make sure you have a running GDK instance. + For running the frontend tests, you need the following commands: - `rake frontend:fixtures` (re-)generates [fixtures](#frontend-test-fixtures). Make sure that diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index aef83109b9b..27d5ae70ed7 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -15,6 +15,7 @@ For any of the following scenarios, the `start-review-app-pipeline` job would be - for merge requests with CI config changes - for merge requests with frontend changes - for merge requests with changes to `{,ee/,jh/}{app/controllers}/**/*` +- for merge requests with changes to `{,ee/,jh/}{app/models}/**/*` - for merge requests with QA changes - for scheduled pipelines - the MR has the `pipeline:run-review-app` label set @@ -30,6 +31,8 @@ You can also manually start the `review-qa-all`: it runs the full QA suite. After the end-to-end test runs have finished, [Allure reports](https://github.com/allure-framework/allure2) are generated and published by the `allure-report-qa-smoke`, `allure-report-qa-reliable`, and `allure-report-qa-all` jobs. A comment with links to the reports are added to the merge request. +Errors can be found in the `gitlab-review-apps` Sentry project and [filterable by Review App URL](https://sentry.gitlab.net/gitlab/gitlab-review-apps/?query=url%3A%22https%3A%2F%2Fgitlab-review-require-ve-u92nn2.gitlab-review.app%2F%22) or [commit SHA](https://sentry.gitlab.net/gitlab/gitlab-review-apps/releases/6095b501da7/all-events/). + ## Performance Metrics On every [pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/125315730) in the `qa` stage, the diff --git a/doc/development/transient/prevention-patterns.md b/doc/development/transient/prevention-patterns.md index c517a6bcd54..93712b104e6 100644 --- a/doc/development/transient/prevention-patterns.md +++ b/doc/development/transient/prevention-patterns.md @@ -120,7 +120,7 @@ When there are 2 jobs being worked on at the same time, it is possible that the In this example, `Worker B` is meant to set the updated status. But `Worker A` calls `#update_state` a little too late. This can be avoided by utilizing either database locks or `Gitlab::ExclusiveLease`. This way, jobs will be -worked on one at a time. This also allows them to be marked as [idempotent](../sidekiq_style_guide.md#idempotent-jobs). +worked on one at a time. This also allows them to be marked as [idempotent](../sidekiq/idempotent_jobs.md). ### Retry mechanism handling diff --git a/doc/development/uploads.md b/doc/development/uploads.md index 6d8b951be83..14ecf38e21c 100644 --- a/doc/development/uploads.md +++ b/doc/development/uploads.md @@ -346,3 +346,85 @@ object using `UploadedFile#from_params`! This method can be unsafe to use depend passed. Instead, use the [`UploadedFile`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/uploaded_file.rb) object that [`multipart.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/middleware/multipart.rb) builds automatically for you. + +### Document Object Storage buckets and CarrierWave integration + +When using Object Storage, GitLab expects each kind of upload to maintain its own bucket in the respective +Object Storage destination. Moreover, the integration with CarrierWave is not used all the time. +The [Object Storage Working Group](https://about.gitlab.com/company/team/structure/working-groups/object-storage/) +is investigating an approach that unifies Object Storage buckets into a single one and removes CarrierWave +so as to simplify implementation and administration of uploads. + +Therefore, document new uploads here by slotting them into the following tables: + +- [Feature bucket details](#feature-bucket-details) +- [CarrierWave integration](#carrierwave-integration) + +#### Feature bucket details + +| Feature | Upload technology | Uploader | Bucket structure | +|------------------------------------------|-------------------|-----------------------|-----------------------------------------------------------------------------------------------------------| +| Job artifacts | `direct upload` | `workhorse` | `/artifacts/<proj_id_hash>/<date>/<job_id>/<artifact_id>` | +| Pipeline artifacts | `carrierwave` | `sidekiq` | `/artifacts/<proj_id_hash>/pipelines/<pipeline_id>/artifacts/<artifact_id>` | +| Live job traces | `fog` | `sidekiq` | `/artifacts/tmp/builds/<job_id>/chunks/<chunk_index>.log` | +| Job traces archive | `carrierwave` | `sidekiq` | `/artifacts/<proj_id_hash>/<date>/<job_id>/<artifact_id>/job.log` | +| Autoscale runner caching | N/A | `gitlab-runner` | `/gitlab-com-[platform-]runners-cache/???` | +| Backups | N/A | `s3cmd`, `awscli`, or `gcs` | `/gitlab-backups/???` | +| Git LFS | `direct upload` | `workhorse` | `/lsf-objects/<lfs_obj_oid[0:2]>/<lfs_obj_oid[2:2]>` | +| Design management files | `disk buffering` | `rails controller` | `/lsf-objects/<lfs_obj_oid[0:2]>/<lfs_obj_oid[2:2]>` | +| Design management thumbnails | `carrierwave` | `sidekiq` | `/uploads/design_management/action/image_v432x230/<model_id>` | +| Generic file uploads | `direct upload` | `workhorse` | `/uploads/@hashed/[0:2]/[2:4]/<hash1>/<hash2>/file` | +| Generic file uploads - personal snippets | `direct upload` | `workhorse` | `/uploads/personal_snippet/<snippet_id>/<filename>` | +| Global appearance settings | `disk buffering` | `rails controller` | `/uploads/appearance/...` | +| Topics | `disk buffering` | `rails controller` | `/uploads/projects/topic/...` | +| Avatar images | `direct upload` | `workhorse` | `/uploads/[user,group,project]/avatar/<model_id>` | +| Import/export | `direct upload` | `workhorse` | `/uploads/import_export_upload/???` | +| GitLab Migration | `carrierwave` | `sidekiq` | `/uploads/bulk_imports/???` | +| MR diffs | `carrierwave` | `sidekiq` | `/external-diffs/merge_request_diffs/mr-<mr_id>/diff-<diff_id>` | +| Package manager archives | `direct upload` | `sidekiq` | `/packages/<proj_id_hash>/packages/<pkg_segment>/files/<pkg_file_id>` | +| Package manager archives | `direct upload` | `sidekiq` | `/packages/<container_id_hash>/debian_*_component_file/<component_file_id>` | +| Package manager archives | `direct upload` | `sidekiq` | `/packages/<container_id_hash>/debian_*_distribution/<distribution_file_id>` | +| Container image cache (?) | `direct upload` | `workhorse` | `/dependency-proxy/<group_id_hash>/dependency_proxy/<group_id>/files/<proxy_id>/<blob_id or manifest_id>` | +| Terraform state files | `carrierwave` | `rails controller` | `/terraform/<proj_id_hash>/<terraform_state_id>` | +| Pages content archives | `carrierwave` | `sidekiq` | `/gitlab-gprd-pages/<proj_id_hash>/pages_deployments/<deployment_id>/` | +| Secure Files | `carrierwave` | `sidekiq` | `/ci-secure-files/<proj_id_hash>/secure_files/<secure_file_id>/` | + +#### CarrierWave integration + +| File | Carrierwave usage | Categorized | +|---------------------------------------------------------|----------------------------------------------------------------------------------|---------------------| +| `app/models/project.rb` | `include Avatarable` | :white_check_mark: | +| `app/models/projects/topic.rb` | `include Avatarable` | :white_check_mark: | +| `app/models/group.rb` | `include Avatarable` | :white_check_mark: | +| `app/models/user.rb` | `include Avatarable` | :white_check_mark: | +| `app/models/terraform/state_version.rb` | `include FileStoreMounter` | :white_check_mark: | +| `app/models/ci/job_artifact.rb` | `include FileStoreMounter` | :white_check_mark: | +| `app/models/ci/pipeline_artifact.rb` | `include FileStoreMounter` | :white_check_mark: | +| `app/models/pages_deployment.rb` | `include FileStoreMounter` | :white_check_mark: | +| `app/models/lfs_object.rb` | `include FileStoreMounter` | :white_check_mark: | +| `app/models/dependency_proxy/blob.rb` | `include FileStoreMounter` | :white_check_mark: | +| `app/models/dependency_proxy/manifest.rb` | `include FileStoreMounter` | :white_check_mark: | +| `app/models/packages/composer/cache_file.rb` | `include FileStoreMounter` | :white_check_mark: | +| `app/models/packages/package_file.rb` | `include FileStoreMounter` | :white_check_mark: | +| `app/models/concerns/packages/debian/component_file.rb` | `include FileStoreMounter` | :white_check_mark: | +| `ee/app/models/issuable_metric_image.rb` | `include FileStoreMounter` | | +| `ee/app/models/vulnerabilities/remediation.rb` | `include FileStoreMounter` | | +| `ee/app/models/vulnerabilities/export.rb` | `include FileStoreMounter` | | +| `app/models/packages/debian/project_distribution.rb` | `include Packages::Debian::Distribution` | :white_check_mark: | +| `app/models/packages/debian/group_distribution.rb` | `include Packages::Debian::Distribution` | :white_check_mark: | +| `app/models/packages/debian/project_component_file.rb` | `include Packages::Debian::ComponentFile` | :white_check_mark: | +| `app/models/packages/debian/group_component_file.rb` | `include Packages::Debian::ComponentFile` | :white_check_mark: | +| `app/models/merge_request_diff.rb` | `mount_uploader :external_diff, ExternalDiffUploader` | :white_check_mark: | +| `app/models/note.rb` | `mount_uploader :attachment, AttachmentUploader` | :white_check_mark: | +| `app/models/appearance.rb` | `mount_uploader :logo, AttachmentUploader` | :white_check_mark: | +| `app/models/appearance.rb` | `mount_uploader :header_logo, AttachmentUploader` | :white_check_mark: | +| `app/models/appearance.rb` | `mount_uploader :favicon, FaviconUploader` | :white_check_mark: | +| `app/models/project.rb` | `mount_uploader :bfg_object_map, AttachmentUploader` | | +| `app/models/import_export_upload.rb` | `mount_uploader :import_file, ImportExportUploader` | :white_check_mark: | +| `app/models/import_export_upload.rb` | `mount_uploader :export_file, ImportExportUploader` | :white_check_mark: | +| `app/models/ci/deleted_object.rb` | `mount_uploader :file, DeletedObjectUploader` | | +| `app/models/design_management/action.rb` | `mount_uploader :image_v432x230, DesignManagement::DesignV432x230Uploader` | :white_check_mark: | +| `app/models/concerns/packages/debian/distribution.rb` | `mount_uploader :signed_file, Packages::Debian::DistributionReleaseFileUploader` | :white_check_mark: | +| `app/models/bulk_imports/export_upload.rb` | `mount_uploader :export_file, ExportUploader` | :white_check_mark: | +| `ee/app/models/user_permission_export_upload.rb` | `mount_uploader :file, AttachmentUploader` | | +| `app/models/ci/secure_file.rb` | `include FileStoreMounter` | | diff --git a/doc/development/value_stream_analytics.md b/doc/development/value_stream_analytics.md index 6b2442f1c32..0d545fa8e3f 100644 --- a/doc/development/value_stream_analytics.md +++ b/doc/development/value_stream_analytics.md @@ -4,11 +4,11 @@ group: Optimize info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Value Stream Analytics development guide +# Value stream analytics development guide Value stream analytics calculates the time between two arbitrary events recorded on domain objects and provides aggregated statistics about the duration. -For information on how to configure Value Stream Analytics in GitLab, see our [analytics documentation](../user/analytics/value_stream_analytics.md). +For information on how to configure value stream analytics in GitLab, see our [analytics documentation](../user/analytics/value_stream_analytics.md). ## Stage diff --git a/doc/development/value_stream_analytics/img/object_hierarchy_example_V14_10.png b/doc/development/value_stream_analytics/img/object_hierarchy_example_V14_10.png Binary files differnew file mode 100644 index 00000000000..0e3c760fa3c --- /dev/null +++ b/doc/development/value_stream_analytics/img/object_hierarchy_example_V14_10.png diff --git a/doc/development/value_stream_analytics/value_stream_analytics_aggregated_backend.md b/doc/development/value_stream_analytics/value_stream_analytics_aggregated_backend.md new file mode 100644 index 00000000000..aef85107cd9 --- /dev/null +++ b/doc/development/value_stream_analytics/value_stream_analytics_aggregated_backend.md @@ -0,0 +1,330 @@ +--- +stage: Manage +group: Optimize +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Aggregated Value Stream Analytics + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/335391) in GitLab 14.7. + +DISCLAIMER: +This page contains information related to upcoming products, features, and functionality. +It is important to note that the information presented is for informational purposes only. +Please do not rely on this information for purchasing or planning purposes. +As with all projects, the items mentioned on this page are subject to change or delay. +The development, release, and timing of any products, features, or functionality remain at the +sole discretion of GitLab Inc. + +This page provides a high-level overview of the aggregated backend for +Value Stream Analytics (VSA). + +## Current Status + +As of 14.8 the aggregated VSA backend is used only in the `gitlab-org` group, for testing purposes +. We plan to gradually roll it out in the next major release (15.0) for the rest of the groups. + +## Motivation + +The aggregated backend aims to solve the performance limitations of the VSA feature and set it up +for long-term growth. + +Our main database is not prepared for analytical workloads. Executing long-running queries can +affect the reliability of the application. For large groups, the current +implementation (old backend) is slow and, in some cases, doesn't even load due to the configured +statement timeout (15s). + +The database queries in the old backend use the core domain models directly through +`IssuableFinders` classes: ([MergeRequestsFinder](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/finders/merge_requests_finder.rb) and [IssuesFinder](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/finders/issues_finder.rb)). +With the requested change of the [date range filters](https://gitlab.com/groups/gitlab-org/-/epics/6046), +this approach was no longer viable from the performance point of view. + +Benefits of the aggregated VSA backend: + +- Simpler database queries (fewer JOINs). +- Faster aggregations, only a single table is accessed. +- Possibility to introduce further aggregations for improving the first page load time. +- Better performance for large groups (with many sub-groups, projects, issues and, merge requests). +- Ready for database decomposition. The VSA related database tables could live in a separate +database with a minimal development effort. +- Ready for keyset pagination which can be useful for exporting the data. +- Possibility to implement more complex event definitions. + - For example, the start event can be two timestamp columns where the earliest value would be + used by the system. + - Example: `MIN(issues.created_at, issues.updated_at)` + +## How does Value Stream Analytics work? + +Value Stream Analytics calculates the duration between two timestamp columns or timestamp +expressions and runs various aggregations on the data. + +Examples: + +- Duration between the Merge Request creation time and Merge Request merge time. +- Duration between the Issue creation time and Issue close time. + +This duration is exposed in various ways: + +- Aggregation: median, average +- Listing: list the duration for individual Merge Request and Issue records + +Apart from the durations, we expose the record count within a stage. + +### Stages + +A stage represents an event pair (start and end events) with additional metadata, such as the name +of the stage. Stages are configurable by the user within the pairing rules defined in the backend. + +**Example stage: Code Review** + +- Start event identifier: Merge Request creation time +- Start event column: uses the `merge_requests.created_at` timestamp column. +- End event identifier: Merge Request merge time +- End event column: uses the `merge_request_metrics.merged_at` timestamp column. +- Stage event hash ID: a calculated hash for the pair of start and end event identifiers. + - If two stages have the same configuration of start and end events, then their stage event hash + IDs are identical. + - The stage event hash ID is later used to store the aggregated data in partitioned database tables. + +### Value streams + +Value streams are container objects for the stages. There can be multiple value streams per group +or project focusing on different aspects of the Dev Ops lifecycle. + +### Example configuration + + + +In this example, there are two independent value streams set up for two teams that are using +different development workflows within the `Test Group` (top-level namespace). + +The first value stream uses standard timestamp-based events for defining the stages. The second +value stream uses label events. + +Each value stream and stage item from the example will be persisted in the database. Notice that +the `Deployment` stage is identical for both value streams; that means that the underlying +`stage_event_hash_id` is the same for both stages. The `stage_event_hash_id` reduces +the amount of data the backend collects and plays a vital role in database partitioning. + +We expect value streams and stages to be rarely changed. When stages (start and end events) are +changed, the aggregated data gets stale. This is fixed by the periodical aggregation occurring +every day. + +### Feature availability + +The aggregated VSA feature is available on the group and project level however, the aggregated +backend is only available for Premium and Ultimate customers due to data storage and data +computation costs. Storing de-normalized, aggregated data requires significant disk space. + +## Aggregated value stream analytics architecture + +The main idea behind the aggregated VSA backend is separation: VSA database tables and queries do +not use the core domain models directly (Issue, MergeRequest). This allows us to scale and +optimize VSA independently from the other parts of the application. + +The architecture consists of two main mechanisms: + +- Periodical data collection and loading (happens in the background). +- Querying the collected data (invoked by the user). + +### Data loading + +The aggregated nature of VSA comes from the periodical data loading. The system queries the core +domain models to collect the stage and timestamp data. This data is periodically inserted into the +VSA database tables. + +High-level overview for each top-level namespace with Premium or Ultimate license: + +1. Load all stages in the group. +1. Iterate over the issues and merge requests records. +1. Based on the stage configurations (start and end event identifiers) collect the timestamp data. +1. `INSERT` or `UPDATE` the data into the VSA database tables. + +The data loading is implemented within the [`Analytics::CycleAnalytics::DataLoaderService`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/services/analytics/cycle_analytics/data_loader_service.rb) +class. There are groups containing a lot of data, so to avoid overloading the primary database, +the service performs operations in batches and enforces strict application limits: + +- Load records in batches. +- Insert records in batches. +- Stop processing when a limit is reached, schedule a background job to continue the processing +later. +- Continue processing data from a specific point. + +As of GitLab 14.7, the data loading is done manually. Once the feature is ready, the service will +be invoked periodically by the system via a cron job (this part is not implemented yet). + +#### Record iteration + +The batched iteration is implemented with the +[efficient IN operator](../database/efficient_in_operator_queries.md). The background job scans +all issues and merge request records in the group hierarchy ordered by the `updated_at` and the +`id` columns. For already aggregated groups, the `DataLoaderService` continues the aggregation +from a specific point which saves time. + +Collecting the timestamp data happens on every iteration. The `DataLoaderService` determines which +stage events are configured within the group hierarchy and builds a query that selects the +required timestamps. The stage record knows which events are configured and the events know how to +select the timestamp columns. + +Example for collected stage events: merge request merged, merge request created, merge request +closed + +Generated SQL query for loading the timestamps: + +```sql +SELECT + -- the list of columns depends on the configured stages + "merge_request_metrics"."merged_at", + "merge_requests"."created_at", + "merge_request_metrics"."latest_closed_at" + FROM "merge_requests" + LEFT OUTER JOIN "merge_request_metrics" ON "merge_request_metrics"."merge_request_id" = "merge_requests"."id" + WHERE "merge_requests"."id" IN (1, 2, 3, 4) -- ids are coming from the batching query +``` + +The `merged_at` column is located in a separate table (`merge_request_metrics`). The +`Gitlab::Analytics::CycleAnalytics::StagEvents::MergeRequestMerged` class adds itself to a scope +for loading the timestamp data without affecting the number of rows (uses `LEFT JOIN`). This +behavior is implemented for each `StageEvent` class with the `include_in` method. + +The data collection query works on the event level. It extracts the event timestamps from the +stages and ensures that we don't collect the same data multiple times. The events mentioned above +could come from the following stage configuration: + +- merge request created - merge request merged +- merge request created - merge request closed + +Other combinations might be also possible, but we prevent the ones that make no sense, for example: + +- merge request merged - merge request created + +Creation time always happens first, so this stage always reports negative duration. + +#### Data scope + +The data collection scans and processes all issues and merge requests records in the group +hierarchy, starting from the top-level group. This means that if a group only has one value stream +in a sub-group, we nevertheless collect data of all issues and merge requests in the hierarchy of +this group. This aims to simplify the data collection mechanism. Moreover, data research shows +that most group hierarchies have their stages configured on the top level. + +During the data collection process, the collected timestamp data is transformed into rows. For +each configured stage, if the start event timestamp is present, the system inserts or updates one +event record. This allows us to determine the upper limit of the inserted rows per group by +counting all issues and merge requests and multiplying the sum by the stage count. + +#### Data consistency concerns + +Due to the async nature of the data collection, data consistency issues are bound to happen. This +is a trade-off that makes the query performance significantly faster. We think that for analytical +workload a slight lag in the data is acceptable. + +Before the rollout we plan to implement some indicators on the VSA page that shows the most +recent backend activities. For example, indicators that show the last data collection timestamp +and the last consistency check timestamp. + +#### Database structure + +VSA collects data for the following domain models: `Issue` and `MergeRequest`. To keep the +aggregated data separated, we use two additional database tables: + +- `analytics_cycle_analytics_issue_stage_events` +- `analytics_cycle_analytics_merge_request_stage_events` + +Both tables are hash partitioned by the `stage_event_hash_id`. Each table uses 32 partitions. It's +an arbitrary number and it could be changed. Important is to keep the partitions under 100GB in +size (which gives the feature a lot of headroom). + +|Column|Description| +|-|-| +|`stage_event_hash_id`|partitioning key| +|`merge_request_id` or `issue_id`|reference to the domain record (Issuable)| +|`group_id`|reference to the group (de-normalization)| +|`project_id`|reference to the project| +|`milestone_id`|duplicated data from the domain record table| +|`author_id`|duplicated data from the domain record table| +|`state_id`|duplicated data from the domain record table| +|`start_event_timestamp`|timestamp derived from the stage configuration| +|`end_event_timestamp`|timestamp derived from the stage configuration| + +With accordance to the data separation requirements, the table doesn't have any foreign keys. The +consistency is ensured by a background job (eventually consistent). + +### Data querying + +The base query always includes the following filters: + +- `stage_event_hash_id` - partition key +- `project_id` or `group_id` - depending if it's a project or group level query +- `end_event_timestamp` - date range filter (last 30 days) + +Example: Selecting review stage duration for the GitLab project + +```sql +SELECT end_event_timestamp - start_event_timestamp +FROM analytics_cycle_analytics_merge_request_stage_events +WHERE +stage_event_hash_id = 16 AND -- hits a specific partition +project_id = 278964 AND +end_event_timestamp > '2022-01-01' AND end_event_timestamp < '2022-01-30' +``` + +#### Query generation + +The query backend is hidden behind the same interface that the old backend implementation uses. +Thanks to this, we can easily switch between the old and new query backends. + +- `DataCollector`: entrypoint for querying VSA data + - `BaseQueryBuilder`: provides the base `ActiveRecord` scope (filters are applied here). + - `average`: average aggregation. + - `median`: median aggregation. + - `count`: row counting. + - `records`: list of issue or merge request records. + +#### Filters + +VSA supports various filters on the base query. Most of the filters require no additional JOINs: + +|Filter name|Description| +|-|-| +|`milestone_title`|The backend translates it to `milestone_id` filter| +|`author_username`|The backend translates it to `author_id` filter| +|`project_ids`|Only used on the group-level| + +Exceptions: these filters are applied on other tables which means we `JOIN` them. + +|Filter name|Description| +|-|-| +|`label_name`|Array filter, using the `label_links` table| +|`assignee_username`|Array filter, using the `*_assignees` table| + +To fully decompose the database, the required ID values would need to be replicated in the VSA +database tables. This change could be implemented using array columns. + +### Endpoints + +The feature uses private JSON APIs for delivering the data to the frontend. On the first page load +, the following requests are invoked: + +- Initial HTML page load which is mostly empty. Some configuration data is exposed via `data` +attributes. +- `value_streams` - Load the available value streams for the given group. +- `stages` - Load the stages for the currently selected value stream. +- `median` - For each stage, request the median duration. +- `count` - For each stage, request the number of items in the stage (this is a +[limit count](../merge_request_performance_guidelines.md#badge-counters), maximum 1000 rows). +- `average_duration_chart` - Data for the duration chart. +- `summary`, `time_summary` - Top-level aggregations, most of the metrics are using different APIs/ +finders and not invoking the aggregated backend. + +When clicking on a specific stage, the `records` endpoint is invoked, which returns the related +records (paginated) for the chosen stage in a specific order. + +### Database decomposition + +By separating the query logic from the main application code, the feature is ready for database +decomposition. If we decide that VSA requires a separate database instance, then moving the +aggregated tables can be accomplished with little effort. + +A different database technology could also be used to further improve the performance of the +feature, for example [Timescale DB](https://www.timescale.com). diff --git a/doc/install/aws/eks_clusters_aws.md b/doc/install/aws/eks_clusters_aws.md index 86318467a91..fe436a03cb5 100644 --- a/doc/install/aws/eks_clusters_aws.md +++ b/doc/install/aws/eks_clusters_aws.md @@ -23,7 +23,7 @@ Using `eksctl` enables the following when building an EKS Cluster: - You have various cluster configuration options: - Selection of operating system: Amazon Linux 2, Windows, Bottlerocket - Selection of Hardware Architecture: x86, ARM, GPU - - Selection of Kubernetes version (the GitLab-managed clusters for your project's applications have [specific Kubernetes version requirements](../../user/infrastructure/clusters/connect/index.md#supported-cluster-versions)) + - Selection of Kubernetes version (the GitLab-managed clusters for your project's applications have [specific Kubernetes version requirements](../../user/clusters/agent/index.md#supported-cluster-versions)) - It can deploy high value-add items to the cluster, including: - A bastion host to keep the cluster endpoint private and possible perform performance testing. - Prometheus and Grafana for monitoring. diff --git a/doc/install/aws/gitlab_hybrid_on_aws.md b/doc/install/aws/gitlab_hybrid_on_aws.md index 2183f351efd..2c40efd5909 100644 --- a/doc/install/aws/gitlab_hybrid_on_aws.md +++ b/doc/install/aws/gitlab_hybrid_on_aws.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Provision GitLab Cloud Native Hybrid on AWS EKS **(FREE SELF)** -GitLab "Cloud Native Hybrid" is a hybrid of the cloud native technology Kubernetes (EKS) and EC2. While as much of the GitLab application as possible runs in Kubernetes or on AWS services (PaaS), the GitLab service Gitaly must still be run on Ec2. Gitaly is a layer designed to overcome limitations of the Git binaries in a horizontally scaled architecture. You can read more here about why Gitaly was built and why the limitations of Git mean that it must currently run on instance compute in [Git Characteristics That Make Horizontal Scaling Difficult](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/DESIGN.md#git-characteristics-that-make-horizontal-scaling-difficult). +GitLab "Cloud Native Hybrid" is a hybrid of the cloud native technology Kubernetes (EKS) and EC2. While as much of the GitLab application as possible runs in Kubernetes or on AWS services (PaaS), the GitLab service Gitaly must still be run on EC2. Gitaly is a layer designed to overcome limitations of the Git binaries in a horizontally scaled architecture. You can read more here about why Gitaly was built and why the limitations of Git mean that it must currently run on instance compute in [Git Characteristics That Make Horizontal Scaling Difficult](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/DESIGN.md#git-characteristics-that-make-horizontal-scaling-difficult). Amazon provides a managed Kubernetes service offering known as [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). @@ -38,13 +38,13 @@ NOTE: This automation is in **Developer Preview**. GitLab is working with AWS on resolving [the outstanding issues](https://github.com/aws-quickstart/quickstart-eks-gitlab/issues?q=is%3Aissue+is%3Aopen+%5BHL%5D) before it is fully released. You can subscribe to this issue to be notified of progress and release announcements: [AWS Quick Start for GitLab Cloud Native Hybrid on EKS Status: DEVELOPER PREVIEW](https://gitlab.com/gitlab-com/alliances/aws/public-tracker/-/issues/11).<br><br> The developer preview deploys Aurora PostgreSQL, but the release version will deploy Amazon RDS PostgreSQL due to [known issues](https://gitlab.com/gitlab-com/alliances/aws/public-tracker/-/issues?label_name%5B%5D=AWS+Known+Issue) with Aurora. All performance testing results will also be redone after this change has been made. -The [GitLab Environment Toolkit (GET)](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit/-/tree/master) is an effort made by GitLab to create a multi-cloud, multi-GitLab (Omnibus + Cloud Native Hybrid) toolkit to provision GitLab. GET is developed by GitLab developers and is open to community contributions. -It is helpful to review the [GitLab Environment Toolkit (GET) Issues](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit/-/issues) to understand if any of them may affect your provisioning plans. +The [GitLab Environment Toolkit (GET)](https://gitlab.com/gitlab-org/gitlab-environment-toolkit/-/tree/main) is an effort made by GitLab to create a multi-cloud, multi-GitLab (Omnibus + Cloud Native Hybrid) toolkit to provision GitLab. GET is developed by GitLab developers and is open to community contributions. +It is helpful to review the [GitLab Environment Toolkit (GET) Issues](https://gitlab.com/gitlab-org/gitlab-environment-toolkit/-/issues) to understand if any of them may affect your provisioning plans. -| | [AWS Quick Start for GitLab Cloud Native Hybrid on EKS](https://aws-quickstart.github.io/quickstart-eks-gitlab/) | [GitLab Environment Toolkit (GET)](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit) | +| | [AWS Quick Start for GitLab Cloud Native Hybrid on EKS](https://aws-quickstart.github.io/quickstart-eks-gitlab/) | [GitLab Environment Toolkit (GET)](https://gitlab.com/gitlab-org/gitlab-environment-toolkit) | | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | -| Licensing | [Open Source (Apache 2.0)](https://github.com/aws-quickstart/quickstart-eks-gitlab/blob/main/LICENSE.txt) | [GitLab Enterprise Edition license](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit/-/blob/main/LICENSE) ([GitLab Premium tier](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit/-/blob/main/README.md)) | -| Overview and Vision | [AWS Quick Start](https://aws.amazon.com/quickstart/) | [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit/-/blob/main/README.md) | +| Licensing | [Open Source (Apache 2.0)](https://github.com/aws-quickstart/quickstart-eks-gitlab/blob/main/LICENSE.txt) | [GitLab Enterprise Edition license](https://gitlab.com/gitlab-org/gitlab-environment-toolkit/-/blob/main/LICENSE) ([GitLab Premium tier](https://gitlab.com/gitlab-org/gitlab-environment-toolkit/-/blob/main/README.md)) | +| Overview and Vision | [AWS Quick Start](https://aws.amazon.com/quickstart/) | [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/gitlab-environment-toolkit/-/blob/main/README.md) | | GitLab Reference Architecture Compliant | Yes | Yes | | GitLab Performance Tool (GPT) Tested | Yes | Yes | | Amazon Well Architected Compliant | Yes<br />(via Quick Start program) | Critical portions <br />reviewed by AWS | diff --git a/doc/install/aws/manual_install_aws.md b/doc/install/aws/manual_install_aws.md index 531a006dbf3..26d93ea06b7 100644 --- a/doc/install/aws/manual_install_aws.md +++ b/doc/install/aws/manual_install_aws.md @@ -24,11 +24,11 @@ GitLab maintains and tests two main types of Reference Architectures. The **Omni ### Getting started for production-grade Omnibus GitLab -The Infrastructure as Code tooling [GitLab Environment Tool (GET)](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit/-/tree/master) is the best place to start for building Omnibus GitLab on AWS and most especially if you are targeting an HA setup. While it does not automate everything, it does complete complex setups like Gitaly Cluster for you. GET is open source so anyone can build on top of it and contribute improvements to it. +The Infrastructure as Code tooling [GitLab Environment Tool (GET)](https://gitlab.com/gitlab-org/gitlab-environment-toolkit/-/tree/main) is the best place to start for building Omnibus GitLab on AWS and most especially if you are targeting an HA setup. While it does not automate everything, it does complete complex setups like Gitaly Cluster for you. GET is open source so anyone can build on top of it and contribute improvements to it. ### Getting started for production-grade Cloud Native Hybrid GitLab -For the Cloud Native Hybrid architectures there are two Infrastructure as Code options which are compared in GitLab Cloud Native Hybrid on AWS EKS implementation pattern in the section [Available Infrastructure as Code for GitLab Cloud Native Hybrid](gitlab_hybrid_on_aws.md#available-infrastructure-as-code-for-gitlab-cloud-native-hybrid). It compares the [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit/-/tree/master) to the AWS Quick Start for GitLab Cloud Native Hybrid on EKS which was co-developed by GitLab and AWS. GET and the AWS Quick Start are both open source so anyone can build on top of them and contribute improvements to them. +For the Cloud Native Hybrid architectures there are two Infrastructure as Code options which are compared in GitLab Cloud Native Hybrid on AWS EKS implementation pattern in the section [Available Infrastructure as Code for GitLab Cloud Native Hybrid](gitlab_hybrid_on_aws.md#available-infrastructure-as-code-for-gitlab-cloud-native-hybrid). It compares the [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/gitlab-environment-toolkit/-/tree/main) to the AWS Quick Start for GitLab Cloud Native Hybrid on EKS which was co-developed by GitLab and AWS. GET and the AWS Quick Start are both open source so anyone can build on top of them and contribute improvements to them. ## Introduction @@ -268,7 +268,7 @@ On the EC2 dashboard, look for Load Balancer in the left navigation bar: 1. Click **Configure Health Check** and set up a health check for your EC2 instances. 1. For **Ping Protocol**, select HTTP. 1. For **Ping Port**, enter 80. - 1. For **Ping Path** - we recommend that you [use the Readiness check endpoint](../../administration/load_balancer.md#readiness-check). You'll need to add [the VPC IP Adddress Range (CIDR)](https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-security-groups.html#elb-vpc-nacl) to the [IP Allowlist](../../administration/monitoring/ip_whitelist.md) for the [Health Check endpoints](../../user/admin_area/monitoring/health_check.md) + 1. For **Ping Path** - we recommend that you [use the Readiness check endpoint](../../administration/load_balancer.md#readiness-check). You'll need to add [the VPC IP Address Range (CIDR)](https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-security-groups.html#elb-vpc-nacl) to the [IP Allowlist](../../administration/monitoring/ip_whitelist.md) for the [Health Check endpoints](../../user/admin_area/monitoring/health_check.md) 1. Keep the default **Advanced Details** or adjust them according to your needs. 1. Click **Add EC2 Instances** - don't add anything as we will create an Auto Scaling Group later to manage instances for us. 1. Click **Add Tags** and add any tags you need. @@ -588,8 +588,8 @@ Let's create an EC2 instance where we'll install Gitaly: 1. In the **Subnet** dropdown, select `gitlab-private-10.0.1.0` from the list of subnets we created earlier. 1. Double check that **Auto-assign Public IP** is set to `Use subnet setting (Disable)`. 1. Click **Add Storage**. -1. Increase the Root volume size to `20 GiB` and change the **Volume Type** to `Provisoned IOPS SSD (io1)`. (This is an arbitrary size. Create a volume big enough for your repository storage requirements.) - 1. For **IOPS** set `1000` (20 GiB x 50 IOPS). You can provision up to 50 IOPS per GiB. If you select a larger volume, increase the IOPS accordingly. Workloads where many small files are written in a serialized manner, like `git`, requires performant storage, hence the choice of `Provisoned IOPS SSD (io1)`. +1. Increase the Root volume size to `20 GiB` and change the **Volume Type** to `Provisioned IOPS SSD (io1)`. (This is an arbitrary size. Create a volume big enough for your repository storage requirements.) + 1. For **IOPS** set `1000` (20 GiB x 50 IOPS). You can provision up to 50 IOPS per GiB. If you select a larger volume, increase the IOPS accordingly. Workloads where many small files are written in a serialized manner, like `git`, requires performant storage, hence the choice of `Provisioned IOPS SSD (io1)`. 1. Click on **Add Tags** and add your tags. In our case, we'll only set `Key: Name` and `Value: Gitaly`. 1. Click on **Configure Security Group** and let's **Create a new security group**. 1. Give your security group a name and description. We'll use `gitlab-gitaly-sec-group` for both. diff --git a/doc/install/index.md b/doc/install/index.md index 44b234747dc..9ffed87fd61 100644 --- a/doc/install/index.md +++ b/doc/install/index.md @@ -31,7 +31,7 @@ install GitLab: | [Helm charts](https://docs.gitlab.com/charts/) | The cloud native Helm chart for installing GitLab and all of its components on Kubernetes. | When installing GitLab on Kubernetes, there are some trade-offs that you need to be aware of: <br/>- Administration and troubleshooting requires Kubernetes knowledge.<br/>- It can be more expensive for smaller installations. The default installation requires more resources than a single node Linux package deployment, as most services are deployed in a redundant fashion.<br/>- There are some feature [limitations to be aware of](https://docs.gitlab.com/charts/#limitations).<br/><br/> Use this method if your infrastructure is built on Kubernetes and you're familiar with how it works. The methods for management, observability, and some concepts are different than traditional deployments. | | [Docker](https://docs.gitlab.com/omnibus/docker/) | The GitLab packages, Dockerized. | Use this method if you're familiar with Docker. | | [Source](installation.md) | Install GitLab and all of its components from scratch. | Use this method if none of the previous methods are available for your platform. Useful for unsupported systems like \*BSD.| -| [GitLab Environment Toolkit (GET)](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit#documentation) | The GitLab Environment toolkit provides a set of automation tools to deploy a [reference architecture](../administration/reference_architectures/index.md) on most major cloud providers. | Customers are very welcome to trial and evaluate GET today, however be aware of [key limitations](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit#missing-features-to-be-aware-of) of the current iteration. For production environments further manual setup will be required based on your specific requirements. | +| [GitLab Environment Toolkit (GET)](https://gitlab.com/gitlab-org/gitlab-environment-toolkit#documentation) | The GitLab Environment toolkit provides a set of automation tools to deploy a [reference architecture](../administration/reference_architectures/index.md) on most major cloud providers. | Customers are very welcome to trial and evaluate GET today, however be aware of [key limitations](https://gitlab.com/gitlab-org/gitlab-environment-toolkit#missing-features-to-be-aware-of) of the current iteration. For production environments further manual setup will be required based on your specific requirements. | | [GitLab Operator](https://docs.gitlab.com/charts/installation/operator.html) | The GitLab Operator provides an installation and management method for GitLab following the [Kubernetes Operator pattern](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/). | Use the GitLab Operator to run GitLab in an [OpenShift](openshift_and_gitlab/index.md) environment. | ## Install GitLab on cloud providers diff --git a/doc/install/installation.md b/doc/install/installation.md index f405bc40f43..21d1ee84722 100644 --- a/doc/install/installation.md +++ b/doc/install/installation.md @@ -230,7 +230,7 @@ Download Ruby and compile it: ```shell mkdir /tmp/ruby && cd /tmp/ruby -curl --remote-name --progress-bar "https://cache.ruby-lang.org/pub/ruby/2.7/ruby-2.7.4.tar.gz" +curl --remote-name --location --progress-bar "https://cache.ruby-lang.org/pub/ruby/2.7/ruby-2.7.4.tar.gz" echo '3043099089608859fc8cce7f9fdccaa1f53a462457e3838ec3b25a7d609fbc5b ruby-2.7.4.tar.gz' | sha256sum -c - && tar xzf ruby-2.7.4.tar.gz cd ruby-2.7.4 @@ -250,7 +250,7 @@ page](https://go.dev/dl). # Remove former Go installation folder sudo rm -rf /usr/local/go -curl --remote-name --progress-bar "https://go.dev/dl/go1.16.10.linux-amd64.tar.gz" +curl --remote-name --location --progress-bar "https://go.dev/dl/go1.16.10.linux-amd64.tar.gz" echo '414cd18ce1d193769b9e97d2401ad718755ab47816e13b2a1cde203d263b55cf go1.16.10.linux-amd64.tar.gz' | shasum -a256 -c - && \ sudo tar -C /usr/local -xzf go1.16.10.linux-amd64.tar.gz sudo ln -sf /usr/local/go/bin/{go,gofmt} /usr/local/bin/ @@ -560,10 +560,6 @@ sudo -u git -H cp config/puma.rb.example config/puma.rb # cores you have available. You can get that number via the `nproc` command. sudo -u git -H editor config/puma.rb -# Configure Git global settings for git user -# 'autocrlf' is needed for the web editor -sudo -u git -H git config --global core.autocrlf input - # Disable 'git gc --auto' because GitLab already runs 'git gc' when needed sudo -u git -H git config --global gc.auto 0 @@ -571,6 +567,7 @@ sudo -u git -H git config --global gc.auto 0 sudo -u git -H git config --global repack.writeBitmaps true # Enable push options +# Refer to https://docs.gitlab.com/ee/user/project/push_options.html for more information. sudo -u git -H git config --global receive.advertisePushOptions true # Enable fsyncObjectFiles to reduce risk of repository corruption if the server crashes @@ -578,9 +575,10 @@ sudo -u git -H git config --global core.fsyncObjectFiles true # Configure Redis connection settings sudo -u git -H cp config/resque.yml.example config/resque.yml +sudo -u git -H cp config/cable.yml.example config/cable.yml # Change the Redis socket path if you are not using the default Debian / Ubuntu configuration -sudo -u git -H editor config/resque.yml +sudo -u git -H editor config/resque.yml config/cable.yml ``` Make sure to edit both `gitlab.yml` and `puma.rb` to match your setup. @@ -1041,9 +1039,8 @@ To use GitLab with HTTPS: 1. Update `ssl_certificate` and `ssl_certificate_key`. 1. Review the configuration file and consider applying other security and performance enhancing features. -Using a self-signed certificate is discouraged but if you must use it, follow the normal directions. Then: - -1. Generate a self-signed SSL certificate: +Using a self-signed certificate is discouraged. If you must use one, +follow the normal directions and generate a self-signed SSL certificate: ```shell mkdir -p /etc/nginx/ssl/ @@ -1052,7 +1049,12 @@ Using a self-signed certificate is discouraged but if you must use it, follow th sudo chmod o-r gitlab.key ``` -1. In the `config.yml` of GitLab Shell set `self_signed_cert` to `true`. +WARNING: +The `self_signed_cert` variable is +[deprecated and redundant](https://gitlab.com/gitlab-org/gitlab-shell/-/issues/120). +It is set to `false` by default, but still accepts self-signed certificates. Setting +this value to `true` allows any certificate to be accepted, and can make +machine-in-the-middle attacks possible. ### Enable Reply by email @@ -1142,7 +1144,7 @@ You can configure the Prometheus server in `config/gitlab.yml`: # example prometheus: enabled: true - server_address: '10.1.2.3:9090' + server_address: '10.1.2.3:9090' ``` ## Troubleshooting diff --git a/doc/install/requirements.md b/doc/install/requirements.md index 665e80e6e00..bce9702b032 100644 --- a/doc/install/requirements.md +++ b/doc/install/requirements.md @@ -331,6 +331,10 @@ NOTE: We don't support running GitLab with JavaScript disabled in the browser and have no plans of supporting that in the future because we have features such as issue boards which require JavaScript extensively. +## Security + +After installation, be sure to read and follow guidance on [maintaining a secure GitLab installation](../security/index.md). + <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/integration/datadog.md b/doc/integration/datadog.md index 4be0089703a..a9be7754cb9 100644 --- a/doc/integration/datadog.md +++ b/doc/integration/datadog.md @@ -26,7 +26,7 @@ project, group, or instance level: Copy this value, as you need it in a later step. 1. *For project-level or group-level integrations:* In GitLab, go to your project or group. 1. *For instance-level integrations:* - 1. Sign in to GitLab as a user with the [Administrator role](../user/permissions.md). + 1. Sign in to GitLab as a user with administrator access. 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Integrations**. 1. Scroll to **Add an integration**, and select **Datadog**. @@ -42,6 +42,8 @@ project, group, or instance level: 1. Optional. If you use groups of GitLab instances (such as staging and production environments), provide an **Env** name. This value is attached to each span the integration generates. +1. Optional. To define any custom tags for all spans at which the integration is being configured, + enter one tag per line in **Tags**. Each line must be in the format `key:value`. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79665) in GitLab 14.8.) 1. Optional. Select **Test settings** to test your integration. 1. Select **Save changes**. diff --git a/doc/integration/elasticsearch.md b/doc/integration/elasticsearch.md index 7356574a33e..4976f7c2664 100644 --- a/doc/integration/elasticsearch.md +++ b/doc/integration/elasticsearch.md @@ -14,22 +14,30 @@ Advanced Search provides faster search response times and [improved search featu ## Version requirements -<!-- Remember to update ee/lib/system_check/app/elasticsearch_check.rb if this changes --> +> Support for Elasticsearch 6.8 was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/350275) in GitLab 14.8 and is scheduled for removal in GitLab 15.0. -| GitLab version | Elasticsearch version | -|---------------------------------------------|-------------------------------| -| GitLab Enterprise Edition 13.9 or greater | Elasticsearch 6.8 through 7.x | -| GitLab Enterprise Edition 13.3 through 13.8 | Elasticsearch 6.4 through 7.x | -| GitLab Enterprise Edition 12.7 through 13.2 | Elasticsearch 6.x through 7.x | -| GitLab Enterprise Edition 11.5 through 12.6 | Elasticsearch 5.6 through 6.x | -| GitLab Enterprise Edition 9.0 through 11.4 | Elasticsearch 5.1 through 5.5 | -| GitLab Enterprise Edition 8.4 through 8.17 | Elasticsearch 2.4 with [Delete By Query Plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/2.4/plugins-delete-by-query.html) installed | +| GitLab version | Elasticsearch version | +|----------------------|--------------------------| +| GitLab 14.8 or later | Elasticsearch 7.x - 7.17 | +| GitLab 13.9 - 14.7 | Elasticsearch 6.8 - 7.x | +| GitLab 13.3 - 13.8 | Elasticsearch 6.4 - 7.x | +| GitLab 12.7 - 13.2 | Elasticsearch 6.x - 7.x | +| GitLab 11.5 - 12.6 | Elasticsearch 5.6 - 6.x | The Elasticsearch Integration works with supported versions of Elasticsearch and follows Elasticsearch's [End of Life Policy](https://www.elastic.co/support/eol). When we change Elasticsearch supported versions in GitLab, we announce them in [deprecation notes](https://about.gitlab.com/handbook/marketing/blog/release-posts/#deprecations) in monthly release posts before we remove them. +### Versions not supported + +GitLab does not support: + +- [Amazon's OpenSearch](https://aws.amazon.com/blogs/opensource/opensearch-1-0-launches/) +(a [fork of Elasticsearch](https://www.elastic.co/what-is/opensearch)). Use AWS Elasticsearch Service 7.10 instead. +For updates, see [issue #327560](https://gitlab.com/gitlab-org/gitlab/-/issues/327560). +- Elasticsearch 8.0. For updates, see [issue #350600](https://gitlab.com/gitlab-org/gitlab/-/issues/350600). Use Elasticsearch 7.17 instead. + ## System requirements Elasticsearch requires additional resources to those documented in the @@ -155,8 +163,7 @@ may need to set the `production -> elasticsearch -> indexer_path` setting in you ## Enable Advanced Search -For GitLab instances with more than 50GB repository data you can follow the instructions for [Indexing large -instances](#indexing-large-instances) below. +For GitLab instances with more than 50GB repository data you can follow the instructions for [how to index large instances efficiently](#how-to-index-large-instances-efficiently) below. To enable Advanced Search, you must have administrator access to GitLab: @@ -466,7 +473,7 @@ The following are some available Rake tasks: | [`sudo gitlab-rake gitlab:elastic:index`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Enables Elasticsearch indexing and run `gitlab:elastic:create_empty_index`, `gitlab:elastic:clear_index_status`, `gitlab:elastic:index_projects`, and `gitlab:elastic:index_snippets`. | | [`sudo gitlab-rake gitlab:elastic:pause_indexing`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Pauses Elasticsearch indexing. Changes are still tracked. Useful for cluster/index migrations. | | [`sudo gitlab-rake gitlab:elastic:resume_indexing`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Resumes Elasticsearch indexing. | -| [`sudo gitlab-rake gitlab:elastic:index_projects`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Iterates over all projects and queues Sidekiq jobs to index them in the background. | +| [`sudo gitlab-rake gitlab:elastic:index_projects`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Iterates over all projects, and queues Sidekiq jobs to index them in the background. It can only be used after the index is created. | | [`sudo gitlab-rake gitlab:elastic:index_projects_status`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Determines the overall status of the indexing. It is done by counting the total number of indexed projects, dividing by a count of the total number of projects, then multiplying by 100. | | [`sudo gitlab-rake gitlab:elastic:clear_index_status`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Deletes all instances of IndexStatus for all projects. Note that this command results in a complete wipe of the index, and it should be used with caution. | | [`sudo gitlab-rake gitlab:elastic:create_empty_index`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Generates empty indexes (the default index and a separate issues index) and assigns an alias for each on the Elasticsearch side only if it doesn't already exist. | @@ -511,7 +518,7 @@ When performing a search, the GitLab index uses the following scopes: | `projects` | Project data (default) | | `blobs` | Code | | `issues` | Issue data | -| `merge_requests` | Merge Request data | +| `merge_requests` | Merge request data | | `milestones` | Milestone data | | `notes` | Note data | | `snippets` | Snippet data | @@ -543,7 +550,7 @@ For basic guidance on choosing a cluster configuration you may refer to [Elastic - The `Number of Elasticsearch shards` setting usually corresponds with the number of CPUs available in your cluster. For example, if you have a 3-node cluster with 4 cores each, this means you benefit from having at least 3*4=12 shards in the cluster. It's only possible to change the shards number by using [Split index API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-split-index.html) or by reindexing to a different index with a changed number of shards. - The `Number of Elasticsearch replicas` setting should most of the time be equal to `1` (each shard has 1 replica). Using `0` is not recommended, because losing one node corrupts the index. -### Indexing large instances +### How to index large instances efficiently This section may be helpful in the event that the other [basic instructions](#enable-advanced-search) cause problems @@ -741,6 +748,86 @@ However, some larger installations may wish to tune the merge policy settings: - Do not do a [force merge](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html "Force Merge") to remove deleted documents. A warning in the [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html "Force Merge") states that this can lead to very large segments that may never get reclaimed, and can also cause significant performance or availability issues. +## Index large instances with dedicated Sidekiq nodes or processes + +Indexing a large instance can be a lengthy and resource-intensive process that has the potential +of overwhelming Sidekiq nodes and processes. This negatively affects the GitLab performance and +availability. + +As GitLab allows you to start multiple Sidekiq processes, you can create an +additional process dedicated to indexing a set of queues (or queue group). This way, you can +ensure that indexing queues always have a dedicated worker, while the rest of the queues have +another dedicated worker to avoid contention. + +For this purpose, use the [queue selector](../administration/operations/extra_sidekiq_processes.md#queue-selector) +option that allows a more general selection of queue groups using a [worker matching query](../administration/operations/extra_sidekiq_routing.md#worker-matching-query). + +To handle these two queue groups, we generally recommend one of the following two options. You can either: + +- [Use two queue groups on one single node](#single-node-two-processes). +- [Use two queue groups, one on each node](#two-nodes-one-process-for-each). + +For the steps below, consider: + +- `feature_category=global_search` as an indexing queue group with its own Sidekiq process. +- `feature_category!=global_search` as a non-indexing queue group that has its own Sidekiq process. + +### Single node, two processes + +To create both an indexing and a non-indexing Sidekiq process in one node: + +1. On your Sidekiq node, change the `/etc/gitlab/gitlab.rb` file to: + + ```ruby + sidekiq['enable'] = true + sidekiq['queue_selector'] = true + sidekiq['queue_groups'] = [ + "feature_category=global_search", + "feature_category!=global_search" + ] + ``` + +1. Save the file and [reconfigure GitLab](../administration/restart_gitlab.md) +for the changes to take effect. + +WARNING: +When starting multiple processes, the number of processes cannot exceed the number of CPU +cores you want to dedicate to Sidekiq. Each Sidekiq process can use only one CPU core, subject +to the available workload and concurrency settings. For more details, see how to +[run multiple Sidekiq processes](../administration/operations/extra_sidekiq_processes.md). + +### Two nodes, one process for each + +To handle these queue groups on two nodes: + +1. To set up the indexing Sidekiq process, on your indexing Sidekiq node, change the `/etc/gitlab/gitlab.rb` file to: + + ```ruby + sidekiq['enable'] = true + sidekiq['queue_selector'] = true + sidekiq['queue_groups'] = [ + "feature_category=global_search" + ] + ``` + +1. Save the file and [reconfigure GitLab](../administration/restart_gitlab.md) +for the changes to take effect. + +1. To set up the non-indexing Sidekiq process, on your non-indexing Sidekiq node, change the `/etc/gitlab/gitlab.rb` file to: + + ```ruby + sidekiq['enable'] = true + sidekiq['queue_selector'] = true + sidekiq['queue_groups'] = [ + "feature_category!=global_search" + ] + ``` + + to set up a non-indexing Sidekiq process. + +1. Save the file and [reconfigure GitLab](../administration/restart_gitlab.md) +for the changes to take effect. + ## Reverting to Basic Search Sometimes there may be issues with your Elasticsearch index data and as such @@ -983,6 +1070,13 @@ however searches will only surface results that can be viewed by the user. Advanced Search will honor all permission checks in the application by filtering out projects that a user does not have access to at search time. +### Indexing fails with `error: elastic: Error 429 (Too Many Requests)` + +If `ElasticCommitIndexerWorker` Sidekiq workers are failing with this error during indexing, it usually means that Elasticsearch is unable to keep up with the concurrency of indexing request. To address change the following settings: + +- To decrease the indexing throughput you can decrease `Bulk request concurrency` (see [Advanced Search settings](#advanced-search-configuration)). This is set to `10` by default, but you change it to as low as 1 to reduce the number of concurrent indexing operations. +- If changing `Bulk request concurrency` didn't help, you can use the [queue selector](../administration/operations/extra_sidekiq_processes.md#queue-selector) option to [limit indexing jobs only to specific Sidekiq nodes](#index-large-instances-with-dedicated-sidekiq-nodes-or-processes), which should reduce the number of indexing requests. + ### Access requirements for the self-managed AWS OpenSearch Service To use the self-managed AWS OpenSearch Service with GitLab, configure your instance's domain access policies diff --git a/doc/integration/github.md b/doc/integration/github.md index d8877e069b8..a265d5c67ed 100644 --- a/doc/integration/github.md +++ b/doc/integration/github.md @@ -4,198 +4,218 @@ group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Integrate your GitLab instance with GitHub **(FREE SELF)** +# Use GitHub as an authentication provider **(FREE SELF)** -You can integrate your GitLab instance with GitHub.com and GitHub Enterprise. This integration -enables users to import projects from GitHub, or sign in to your GitLab instance -with their GitHub account. +You can integrate your GitLab instance with GitHub.com and GitHub Enterprise. +You can import projects from GitHub, or sign in to GitLab +with your GitHub credentials. -## Security check +## Create an OAuth app in GitHub -Some integrations risk compromising GitLab accounts. To help mitigate this -[OAuth 2 covert redirect](https://oauth.net/advisories/2014-1-covert-redirect/) -vulnerability, append `/users/auth` to the end of the authorization callback URL. +To enable the GitHub OmniAuth provider, you need an OAuth 2.0 client ID and client +secret from GitHub: + +1. Sign in to GitHub. +1. [Create an OAuth App](https://docs.github.com/en/developers/apps/building-oauth-apps/creating-an-oauth-app) + and provide the following information: + - The URL of your GitLab instance, such as `https://gitlab.example.com`. + - The authorization callback URL, such as, `https://gitlab.example.com/users/auth`. + Include the port number if your GitLab instance uses a non-default port. + +### Check for security vulnerabilities + +For some integrations, the [OAuth 2 covert redirect](https://oauth.net/advisories/2014-1-covert-redirect/) +vulnerability can compromise GitLab accounts. +To mitigate this vulnerability, append `/users/auth` to the authorization +callback URL. However, as far as we know, GitHub does not validate the subdomain part of the `redirect_uri`. -This means that a subdomain takeover, an XSS, or an open redirect on any subdomain of +Therefore, a subdomain takeover, an XSS, or an open redirect on any subdomain of your website could enable the covert redirect attack. -## Enabling GitHub OAuth +## Enable GitHub OAuth in GitLab -To enable the GitHub OmniAuth provider, you need an OAuth 2 Client ID and Client Secret from GitHub. To get these credentials, sign into GitHub and follow their procedure for [Creating an OAuth App](https://docs.github.com/en/developers/apps/building-oauth-apps/creating-an-oauth-app). +1. [Configure the initial settings](omniauth.md#configure-initial-settings) in GitLab. -When you create an OAuth 2 app in GitHub, you need the following information: +1. Edit the GitLab configuration file using the following information: -- The URL of your GitLab instance, such as `https://gitlab.example.com`. -- The authorization callback URL; in this case, `https://gitlab.example.com/users/auth`. Include the port number if your GitLab instance uses a non-default port. + | GitHub setting | Value in the GitLab configuration file | Description | + |----------------|----------------------------------------|-------------------------| + | Client ID | `YOUR_APP_ID` | OAuth 2.0 client ID | + | Client secret | `YOUR_APP_SECRET` | OAuth 2.0 client secret | + | URL | `https://github.example.com/` | GitHub deployment URL | -See [Configure initial settings](omniauth.md#configure-initial-settings) for initial settings. + - **For Omnibus installations** -After you have configured the GitHub provider, you need the following information. You must substitute that information in the GitLab configuration file in these next steps. + 1. Open the `/etc/gitlab/gitlab.rb` file. -| Setting from GitHub | Substitute in the GitLab configuration file | Description | -|:---------------------|:---------------------------------------------|:------------| -| Client ID | `YOUR_APP_ID` | OAuth 2 Client ID | -| Client Secret | `YOUR_APP_SECRET` | OAuth 2 Client Secret | -| URL | `https://github.example.com/` | GitHub Deployment URL | + For GitHub.com, update the following section: -Follow these steps to incorporate the GitHub OAuth 2 app in your GitLab server: + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + name: "github", + # label: "Provider name", # optional label for login button, defaults to "GitHub" + app_id: "YOUR_APP_ID", + app_secret: "YOUR_APP_SECRET", + args: { scope: "user:email" } + } + ] + ``` -**For Omnibus installations** + For GitHub Enterprise, update the following section and replace + `https://github.example.com/` with your GitHub URL: -1. Edit `/etc/gitlab/gitlab.rb`: + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + name: "github", + # label: "Provider name", # optional label for login button, defaults to "GitHub" + app_id: "YOUR_APP_ID", + app_secret: "YOUR_APP_SECRET", + url: "https://github.example.com/", + args: { scope: "user:email" } + } + ] + ``` - For GitHub.com: + 1. Save the file and [reconfigure](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) + GitLab. - ```ruby - gitlab_rails['omniauth_providers'] = [ - { - name: "github", - # label: "Provider name", # optional label for login button, defaults to "GitHub" - app_id: "YOUR_APP_ID", - app_secret: "YOUR_APP_SECRET", - args: { scope: "user:email" } - } - ] - ``` + - **For installations from source** - For GitHub Enterprise: + 1. Open the `config/gitlab.yml` file. - ```ruby - gitlab_rails['omniauth_providers'] = [ - { - name: "github", - # label: "Provider name", # optional label for login button, defaults to "GitHub" - app_id: "YOUR_APP_ID", - app_secret: "YOUR_APP_SECRET", - url: "https://github.example.com/", - args: { scope: "user:email" } - } - ] - ``` + For GitHub.com, update the following section: - **Replace `https://github.example.com/` with your GitHub URL.** + ```yaml + - { name: 'github', + # label: 'Provider name', # optional label for login button, defaults to "GitHub" + app_id: 'YOUR_APP_ID', + app_secret: 'YOUR_APP_SECRET', + args: { scope: 'user:email' } } + ``` -1. Save the file and [reconfigure](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. + For GitHub Enterprise, update the following section and replace + `https://github.example.com/` with your GitHub URL: ---- - -**For installations from source** + ```yaml + - { name: 'github', + # label: 'Provider name', # optional label for login button, defaults to "GitHub" + app_id: 'YOUR_APP_ID', + app_secret: 'YOUR_APP_SECRET', + url: "https://github.example.com/", + args: { scope: 'user:email' } } + ``` -1. Navigate to your repository and edit `config/gitlab.yml`: + 1. Save the file and [restart](../administration/restart_gitlab.md#installations-from-source) + GitLab. - For GitHub.com: +1. Refresh the GitLab sign-in page. A GitHub icon should display below the + sign-in form. - ```yaml - - { name: 'github', - # label: 'Provider name', # optional label for login button, defaults to "GitHub" - app_id: 'YOUR_APP_ID', - app_secret: 'YOUR_APP_SECRET', - args: { scope: 'user:email' } } - ``` +1. Select the icon. Sign in to GitHub and authorize the GitLab application. - For GitHub Enterprise: +## Troubleshooting - ```yaml - - { name: 'github', - # label: 'Provider name', # optional label for login button, defaults to "GitHub" - app_id: 'YOUR_APP_ID', - app_secret: 'YOUR_APP_SECRET', - url: "https://github.example.com/", - args: { scope: 'user:email' } } - ``` +### Imports from GitHub Enterprise with a self-signed certificate fail - **Replace `https://github.example.com/` with your GitHub URL.** +When you import projects from GitHub Enterprise using a self-signed +certificate, the imports fail. -1. Save the file and [restart](../administration/restart_gitlab.md#installations-from-source) GitLab for the changes to take effect. +To fix this issue, you must disable SSL verification: ---- +1. Set `verify_ssl` to `false` in the configuration file. -1. Refresh the GitLab sign in page. You should now see a GitHub icon below the regular sign in form. + - **For Omnibus installations** -1. Click the icon to begin the authentication process. GitHub asks the user to sign in and authorize the GitLab application. + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + name: "github", + # label: "Provider name", # optional label for login button, defaults to "GitHub" + app_id: "YOUR_APP_ID", + app_secret: "YOUR_APP_SECRET", + url: "https://github.example.com/", + verify_ssl: false, + args: { scope: "user:email" } + } + ] + ``` -## GitHub Enterprise with self-signed Certificate + - **For installations from source** -If you are attempting to import projects from GitHub Enterprise with a self-signed -certificate and the imports are failing, you must disable SSL verification. -It should be disabled by adding `verify_ssl` to `false` in the provider configuration -and changing the global Git `sslVerify` option to `false` in the GitLab server. + ```yaml + - { name: 'github', + # label: 'Provider name', # optional label for login button, defaults to "GitHub" + app_id: 'YOUR_APP_ID', + app_secret: 'YOUR_APP_SECRET', + url: "https://github.example.com/", + verify_ssl: false, + args: { scope: 'user:email' } } + ``` -For Omnibus package: +1. Change the global Git `sslVerify` option to `false` on the GitLab server. -```ruby -gitlab_rails['omniauth_providers'] = [ - { - name: "github", - # label: "Provider name", # optional label for login button, defaults to "GitHub" - app_id: "YOUR_APP_ID", - app_secret: "YOUR_APP_SECRET", - url: "https://github.example.com/", - verify_ssl: false, - args: { scope: "user:email" } - } -] -``` + - **For Omnibus installations** -You must also disable Git SSL verification on the server hosting GitLab. + ```ruby + omnibus_gitconfig['system'] = { "http" => ["sslVerify = false"] } + ``` -```ruby -omnibus_gitconfig['system'] = { "http" => ["sslVerify = false"] } -``` + - **For installations from source** -For installation from source: + ```shell + git config --global http.sslVerify false + ``` -```yaml -- { name: 'github', - # label: 'Provider name', # optional label for login button, defaults to "GitHub" - app_id: 'YOUR_APP_ID', - app_secret: 'YOUR_APP_SECRET', - url: "https://github.example.com/", - verify_ssl: false, - args: { scope: 'user:email' } } -``` +1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) + if you installed using Omnibus, or [restart GitLab](../administration/restart_gitlab.md#installations-from-source) + if you installed from source. -You must also disable Git SSL verification on the server hosting GitLab. +### Signing in using GitHub Enterprise returns a 500 error -```shell -git config --global http.sslVerify false -``` +This error can occur because of a network connectivity issue between your +GitLab instance and GitHub Enterprise. -For the changes to take effect, [reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) if you installed -via Omnibus, or [restart GitLab](../administration/restart_gitlab.md#installations-from-source) if you installed from source. +To check for a connectivity issue: -## Troubleshooting +1. Go to the [`production.log`](../administration/logs.md#productionlog) + on your GitLab server and look for the following error: -### Error 500 when trying to sign in to GitLab via GitHub Enterprise + ``` plaintext + Faraday::ConnectionFailed (execution expired) + ``` -Check the [`production.log`](../administration/logs.md#productionlog) -on your GitLab server to obtain further details. If you are getting the error like -`Faraday::ConnectionFailed (execution expired)` in the log, there may be a connectivity issue -between your GitLab instance and GitHub Enterprise. To verify it, [start the rails console](../administration/operations/rails_console.md#starting-a-rails-console-session) -and run the commands below replacing `<github_url>` with the URL of your GitHub Enterprise instance: +1. [Start the rails console](../administration/operations/rails_console.md#starting-a-rails-console-session) + and run the following commands. Replace `<github_url>` with the URL of your + GitHub Enterprise instance: -```ruby -uri = URI.parse("https://<github_url>") # replace `GitHub-URL` with the real one here -http = Net::HTTP.new(uri.host, uri.port) -http.use_ssl = true -http.verify_mode = 1 -response = http.request(Net::HTTP::Get.new(uri.request_uri)) -``` + ```ruby + uri = URI.parse("https://<github_url>") # replace `GitHub-URL` with the real one here + http = Net::HTTP.new(uri.host, uri.port) + http.use_ssl = true + http.verify_mode = 1 + response = http.request(Net::HTTP::Get.new(uri.request_uri)) + ``` -If you are getting a similar `execution expired` error, it confirms the theory about the -network connectivity. In that case, make sure that the GitLab server is able to reach your -GitHub enterprise instance. +1. If a similar `execution expired` error is returned, this confirms the error is + caused by a connectivity issue. Make sure the GitLab server can reach + your GitHub Enterprise instance. ### Signing in using your GitHub account without a pre-existing GitLab account is not allowed -If you're getting the message `Signing in using your GitHub account without a pre-existing -GitLab account is not allowed. Create a GitLab account first, and then connect it to your -GitHub account` when signing in, in GitLab: +When you sign in to GitLab, you get the following error: + +```plaintext +Signing in using your GitHub account without a pre-existing +GitLab account is not allowed. Create a GitLab account first, +and then connect it to your GitHub account +``` + +To fix this issue, you must activate GitHub sign-in in GitLab: -1. In the top-right corner, select your avatar. +1. On the top bar, in the top right corner, select your avatar. 1. Select **Edit profile**. 1. On the left sidebar, select **Account**. 1. In the **Social sign-in** section, select **Connect to GitHub**. - -After that, you should be able to sign in via GitHub successfully. diff --git a/doc/integration/gitlab.md b/doc/integration/gitlab.md index 2dd357e50a6..74ae9bb1998 100644 --- a/doc/integration/gitlab.md +++ b/doc/integration/gitlab.md @@ -24,7 +24,12 @@ GitLab.com generates an application ID and secret key for you to use. http://your-gitlab.example.com/users/auth/gitlab/callback ``` - The first link is required for the importer and second for the authorization. + The first link is required for the importer and second for authentication. + + If you: + + - Plan to use the importer, you can leave scopes as they are. + - Only want to use this application for authentication, we recommend using a more minimal set of scopes. `read_user` is sufficient. 1. Select **Save application**. 1. You should now see an **Application ID** and **Secret**. Keep this page open as you continue @@ -57,7 +62,9 @@ GitLab.com generates an application ID and secret key for you to use. # label: "Provider name", # optional label for login button, defaults to "GitLab.com" app_id: "YOUR_APP_ID", app_secret: "YOUR_APP_SECRET", - args: { scope: "api" } + args: { scope: "read_user" # optional: defaults to the scopes of the application + , client_options: { site: "https://gitlab.example.com/api/v4" } + } } ] ``` @@ -71,7 +78,8 @@ GitLab.com generates an application ID and secret key for you to use. label: "Provider name", # optional label for login button, defaults to "GitLab.com" app_id: "YOUR_APP_ID", app_secret: "YOUR_APP_SECRET", - args: { scope: "api", client_options: { site: "https://gitlab.example.com/api/v4" } } + args: { scope: "read_user" # optional: defaults to the scopes of the application + , client_options: { site: "https://gitlab.example.com/api/v4" } } } ] ``` @@ -83,7 +91,7 @@ GitLab.com generates an application ID and secret key for you to use. # label: 'Provider name', # optional label for login button, defaults to "GitLab.com" app_id: 'YOUR_APP_ID', app_secret: 'YOUR_APP_SECRET', - args: { scope: 'api' } } + args: { "client_options": { "site": 'https://gitlab.example.com/api/v4' } } ``` Or, for installations from source to authenticate against a different GitLab instance: @@ -93,7 +101,7 @@ GitLab.com generates an application ID and secret key for you to use. label: 'Provider name', # optional label for login button, defaults to "GitLab.com" app_id: 'YOUR_APP_ID', app_secret: 'YOUR_APP_SECRET', - args: { scope: 'api', "client_options": { "site": 'https://gitlab.example.com/api/v4' } } + args: { "client_options": { "site": 'https://gitlab.example.com/api/v4' } } ``` 1. Change `'YOUR_APP_ID'` to the Application ID from the GitLab.com application page. @@ -101,7 +109,6 @@ GitLab.com generates an application ID and secret key for you to use. 1. Save the configuration file. 1. Based on how GitLab was installed, implement these changes by using the appropriate method: - - Omnibus GitLab: [reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure). - Source: [restart GitLab](../administration/restart_gitlab.md#installations-from-source). @@ -110,3 +117,20 @@ regular sign-in form. Select the icon to begin the authentication process. GitLab.com asks the user to sign in and authorize the GitLab application. If everything goes well, the user is returned to your GitLab instance and is signed in. + +## Reduce access privileges on sign in + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/337663) in GitLab 14.8 [with a flag](../administration/feature_flags.md) named `omniauth_login_minimal_scopes`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../administration/feature_flags.md) named `omniauth_login_minimal_scopes`. On GitLab.com, this feature is not available. + +If you use a GitLab instance for authentication, you can reduce access rights when an OAuth application is used for sign in. + +Any OAuth application can advertise the purpose of the application with the +authorization parameter: `gl_auth_type=login`. If the application is +configured with `api` or `read_api`, the access token is issued with +`read_user` for login, because no higher permissions are needed. + +The GitLab OAuth client is configured to pass this parameter, but other +applications can also pass it. diff --git a/doc/integration/img/enable_trello_powerup.png b/doc/integration/img/enable_trello_powerup.png Binary files differdeleted file mode 100644 index f80d0eadc0b..00000000000 --- a/doc/integration/img/enable_trello_powerup.png +++ /dev/null diff --git a/doc/integration/jenkins.md b/doc/integration/jenkins.md index bae52622966..b86643c7279 100644 --- a/doc/integration/jenkins.md +++ b/doc/integration/jenkins.md @@ -44,7 +44,7 @@ Grant a GitLab user access to the relevant GitLab projects. 1. Grant the user permission to the GitLab projects. If you're integrating Jenkins with many GitLab projects, consider granting the - user the administrator access level. Otherwise, add the user to each project + user administrator access. Otherwise, add the user to each project and grant the Developer role. ## Grant Jenkins access to the GitLab API @@ -137,6 +137,7 @@ than the [webhook integration](#configure-a-webhook). - Merge request - Tag push 1. Enter the **Jenkins server URL**. +1. Optional. Clear the **Enable SSL verification** checkbox to disable [SSL verification](../user/project/integrations/overview.md#ssl-verification). 1. Enter the **Project name**. The project name should be URL-friendly, where spaces are replaced with underscores. To ensure diff --git a/doc/integration/jenkins_deprecated.md b/doc/integration/jenkins_deprecated.md index 8da3118cf2c..57219b18047 100644 --- a/doc/integration/jenkins_deprecated.md +++ b/doc/integration/jenkins_deprecated.md @@ -19,7 +19,7 @@ This service was [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/1600) i Integration includes: - Trigger Jenkins build after push to repository -- Show build status on Merge Request page +- Show build status on Merge request page Requirements: diff --git a/doc/integration/jira/connect-app.md b/doc/integration/jira/connect-app.md index 597293ae5ca..ebe8a0f1af4 100644 --- a/doc/integration/jira/connect-app.md +++ b/doc/integration/jira/connect-app.md @@ -10,8 +10,7 @@ You can integrate GitLab and Jira Cloud using the [GitLab.com for Jira Cloud](https://marketplace.atlassian.com/apps/1221011/gitlab-com-for-jira-cloud) app in the Atlassian Marketplace. -NOTE: -Only Jira users with the administrator role can install or configure +Only Jira users with administrator access can install or configure the GitLab.com for Jira Cloud app. ## Install the GitLab.com for Jira Cloud app **(FREE SAAS)** @@ -23,7 +22,7 @@ We recommend the GitLab.com for Jira Cloud app, because data is synchronized in real time. The DVCS connector updates data only once per hour. The user configuring the GitLab.com for Jira Cloud app must have -at least the [Maintainer](../../user/permissions.md) role in the GitLab.com namespace. +at least the Maintainer role in the GitLab.com namespace. This integration method supports [Smart Commits](dvcs.md#smart-commits). @@ -43,7 +42,7 @@ To install the GitLab.com for Jira Cloud app:  1. If not already signed in to GitLab.com, you must sign in as a user with - [Maintainer](../../user/permissions.md) permissions to add namespaces. + the Maintainer role to add namespaces.  1. To open the list of available namespaces, select **Add namespace**. @@ -91,10 +90,10 @@ self-managed GitLab instances with Jira Cloud, you can either: You can configure your Atlassian Cloud instance to allow you to install applications from outside the Marketplace, which allows you to install the application: -1. Sign in to your Jira instance as a user with an Administrator role. +1. Sign in to your Jira instance as an administrator. 1. Place your Jira instance into [development mode](https://developer.atlassian.com/cloud/jira/platform/getting-started-with-connect/#step-2--enable-development-mode). -1. Sign in to your GitLab application as an [administrator](../../user/permissions.md). +1. Sign in to your GitLab application as a user with administrator access. 1. Install the GitLab application from your self-managed GitLab instance, as described in the [Atlassian developer guides](https://developer.atlassian.com/cloud/jira/platform/getting-started-with-connect/#step-3--install-and-test-your-app): 1. In your Jira instance, go to **Apps > Manage Apps** and select **Upload app**: diff --git a/doc/integration/kerberos.md b/doc/integration/kerberos.md index 04a02b8fa68..17a81419ad0 100644 --- a/doc/integration/kerberos.md +++ b/doc/integration/kerberos.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" --- diff --git a/doc/integration/mattermost/index.md b/doc/integration/mattermost/index.md index 02fe0f4ea71..6ac4faa5a35 100644 --- a/doc/integration/mattermost/index.md +++ b/doc/integration/mattermost/index.md @@ -372,19 +372,11 @@ For a complete list of upgrade notices and special considerations for older vers ## Upgrading GitLab Mattermost to 14.6 -GitLab 14.6 includes Mattermost 6.1, and also includes the migrations for Mattermost 6.0. For information about upgrading and for ways to reduce the downtime of those migrations, read the [Important Upgrade Notes](https://docs.mattermost.com/administration/important-upgrade-notes.html) for both versions. +GitLab 14.6 ships with Mattermost 6.1 including potentially long running database migrations for Mattermost 6.0. For information about upgrading and for ways to reduce the downtime caused by those migrations, read the [Important Upgrade Notes](https://docs.mattermost.com/administration/important-upgrade-notes.html) for both versions. If you need to perform any manual migrations, [connect to the bundled PostgreSQL database](#connecting-to-the-bundled-postgresql-database). NOTE: The Mattermost upgrade notes refer to different impacts when used with a PostgreSQL versus a MySQL database. The GitLab Mattermost included with the GitLab Linux packages uses a PostgreSQL database. -If you need to connect in the database to perform any manual migrations, run the following: - -```console -sudo gitlab-psql -d mattermost_production -``` - -You can then run the necessary queries that are described in the [Important Upgrade Notes](https://docs.mattermost.com/administration/important-upgrade-notes.html). - ## Upgrading GitLab Mattermost from versions prior to 11.0 With version 11.0, GitLab introduced breaking changes which affected Mattermost configuration. diff --git a/doc/integration/oauth_provider.md b/doc/integration/oauth_provider.md index ff144d9985b..adfb2fad941 100644 --- a/doc/integration/oauth_provider.md +++ b/doc/integration/oauth_provider.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -86,25 +86,24 @@ To create an application for your GitLab instance: When creating application in the **Admin Area** , you can mark it as _trusted_. The user authorization step is automatically skipped for this application. -## Expiring Access Tokens +## Expiring access tokens > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/21745) in GitLab 14.3. -By default, all new applications expire access tokens after 2 hours. In GitLab 14.2 and -earlier, OAuth access tokens had no expiration. +WARNING: +The ability to opt-out of expiring access tokens [is deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/340848). +All existing integrations should be updated to support access token refresh. -All integrations should update to support access token refresh. +Access tokens expire in two hours which means that integrations that use them must support generating new access +tokens at least every two hours. Existing: -When creating new applications, you can opt-out of expiry for backward compatibility by clearing -**Expire access tokens** when creating them. The ability to opt-out -[is deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/340848). - -Existing: - -- Applications can have expiring access tokens. Edit the application and select - **Expire access tokens** to enable them. +- Applications can have expiring access tokens: + 1. Edit the application. + 1. Select **Expire access tokens**. - Tokens must be [revoked](../api/oauth2.md#revoke-a-token) or they don't expire. +When applications are deleted, all grants and tokens associated with the application are also deleted. + ## Authorized applications Every application you authorize with your GitLab credentials is shown diff --git a/doc/integration/openid_connect_provider.md b/doc/integration/openid_connect_provider.md index e85231d1c25..1254b29bfb9 100644 --- a/doc/integration/openid_connect_provider.md +++ b/doc/integration/openid_connect_provider.md @@ -35,12 +35,15 @@ is select the `openid` scope in the application settings. ## Settings discovery -If your client allows importing OIDC settings from a discovery URL, you can use the following URL to automatically find the correct settings: +If your client allows importing OIDC settings from a discovery URL, you can use +the following URL to automatically find the correct settings for GitLab.com: ```plaintext -https://gitlab.example.com/.well-known/openid-configuration +https://gitlab.com/.well-known/openid-configuration ``` +Similar URLs can be used for other GitLab instances. + ## Shared information The following user information is shared with clients: diff --git a/doc/integration/saml.md b/doc/integration/saml.md index 61d09b4e173..95bf835147d 100644 --- a/doc/integration/saml.md +++ b/doc/integration/saml.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -367,7 +367,7 @@ The requirements are the same as the previous settings: - The IdP must pass Group information to GitLab. - GitLab must know where to look for the groups in the SAML response, as well as - which group(s) grant the user an administrator role. + which groups grant the user administrator access. ```yaml { name: 'saml', @@ -504,7 +504,7 @@ omniauth: Keep in mind that every sign in attempt redirects to the SAML server; you cannot sign in using local credentials. Ensure at least one of the -SAML users has an administrator role. +SAML users has administrator access. You may also bypass the auto sign-in feature by browsing to `https://gitlab.example.com/users/sign_in?auto_sign_in=false`. diff --git a/doc/integration/trello_power_up.md b/doc/integration/trello_power_up.md index 96150440e53..8a8952cb594 100644 --- a/doc/integration/trello_power_up.md +++ b/doc/integration/trello_power_up.md @@ -6,44 +6,25 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Trello Power-Up **(FREE)** -The GitLab Trello Power-Up enables you to seamlessly attach -GitLab **merge requests** to Trello cards. +You can use the Trello Power-Up for GitLab to attach +GitLab merge requests to Trello cards.  -## Configuring the Power-Up +## Configure the Power-Up -In order to get started, you must configure your Power-Up. +To configure a Power-Up for a Trello board: -In Trello: +1. Go to your Trello board. +1. Select **Power-Ups** and find the **GitLab** row. +1. Select **Enable**. +1. Select **Settings** (the gear icon). +1. Select **Authorize Account**. +1. Enter the [GitLab API URL](#get-the-api-url) and [personal access token](../user/profile/personal_access_tokens.md#create-a-personal-access-token) with the **API** scope. +1. Select **Save**. -1. Go to your Trello board -1. Select `Power-Ups` to see a listing of all the available Power-Ups -1. Look for a row that says `GitLab` and select the `Enable` button -1. Select the `Settings` (gear) icon -1. In the popup menu, select `Authorize Account` +## Get the API URL -In this popup, fill in your `API URL` and `Personal Access Token`. After that, you can attach any merge request to any Trello card on your selected Trello board. - -## What is my API URL? - -Your API URL should be your GitLab instance URL with `/api/v4` appended in the end of the URL. -For example, if your GitLab instance URL is `https://gitlab.com`, your API URL would be `https://gitlab.com/api/v4`. -If your instance's URL is `https://example.com`, your API URL is `https://example.com/api/v4`. - - - -## What is my Personal Access Token? - -Your GitLab personal access token enables your GitLab account to be accessed -from Trello. - -To find it in GitLab: - -1. In the top-right corner, select your avatar. -1. Select **Edit profile**. -1. On the left sidebar, select **Access Tokens**. - -Learn more about generating a personal access token in the -[Personal Access Token Documentation](../user/profile/personal_access_tokens.md). -Don't forget to check the API scope checkbox! +Your API URL is your GitLab instance URL with `/api/v4` appended at the end of the URL. +For example, if your GitLab instance URL is `https://gitlab.com`, your API URL is `https://gitlab.com/api/v4`. +If your instance URL is `https://example.com`, your API URL is `https://example.com/api/v4`. diff --git a/doc/operations/error_tracking.md b/doc/operations/error_tracking.md index 7533646aa34..6f97f002a32 100644 --- a/doc/operations/error_tracking.md +++ b/doc/operations/error_tracking.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/operations/feature_flags.md b/doc/operations/feature_flags.md index b0a180b5635..8903e99ab3b 100644 --- a/doc/operations/feature_flags.md +++ b/doc/operations/feature_flags.md @@ -168,7 +168,7 @@ target users. See the [Ruby example](#ruby-application-example) below. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/300299) in GitLab 14.4. Search your project and find any references of a feature flag in your -code so that you and clean it up when it's time to remove the feature flag. +code so that you can clean it up when it's time to remove the feature flag. To search for code references of a feature flag: @@ -345,7 +345,7 @@ import ( "log" "net/http" - "github.com/Unleash/unleash-client-go" + "github.com/Unleash/unleash-client-go/v3" ) type metricsInterface struct { diff --git a/doc/operations/incident_management/alerts.md b/doc/operations/incident_management/alerts.md index cdf7ca5c8bc..b03955edf87 100644 --- a/doc/operations/incident_management/alerts.md +++ b/doc/operations/incident_management/alerts.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -10,7 +10,7 @@ Alerts are a critical entity in your incident management workflow. They represen ## Alert List -Users with at least the Developer [role](../../user/permissions.md) can +Users with at least the Developer role can access the Alert list at **Monitor > Alerts** in your project's sidebar. The Alert list displays alerts sorted by start time, but you can change the sort order by clicking the headers in the Alert list. @@ -68,7 +68,7 @@ Alerts contain one of the following icons: ## Alert details page Navigate to the Alert details view by visiting the [Alert list](alerts.md) -and selecting an alert from the list. You need at least the Developer [role](../../user/permissions.md) +and selecting an alert from the list. You need at least the Developer role to access alerts. NOTE: @@ -96,7 +96,7 @@ instance. Prerequisite: -- You must have at least the Developer [role](../../user/permissions.md). +- You must have at least the Developer role. To view the metrics for an alert: @@ -120,7 +120,7 @@ your application's performance and how to resolve any problems. Prerequisite: -- You must have at least the Developer [role](../../user/permissions.md). +- You must have at least the Developer role. To view the logs for an alert: diff --git a/doc/operations/incident_management/escalation_policies.md b/doc/operations/incident_management/escalation_policies.md index 5f132720000..c24824e55f8 100644 --- a/doc/operations/incident_management/escalation_policies.md +++ b/doc/operations/incident_management/escalation_policies.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -17,7 +17,7 @@ where you manage [On-call schedules](oncall_schedules.md). Prerequisite: -- You must have at least the Maintainer [role](../../user/permissions.md). +- You must have at least the Maintainer role. - You must have an [on-call schedule](oncall_schedules.md). To create an escalation policy: diff --git a/doc/operations/incident_management/incidents.md b/doc/operations/incident_management/incidents.md index ada1f426dd8..e142d060d87 100644 --- a/doc/operations/incident_management/incidents.md +++ b/doc/operations/incident_management/incidents.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -21,15 +21,15 @@ You can create an incident manually or automatically. > - [Moved](https://gitlab.com/gitlab-org/monitor/monitor/-/issues/24) to GitLab Free in 13.3. > - [Permission changed](https://gitlab.com/gitlab-org/gitlab/-/issues/336624) from Guest to Reporter in GitLab 14.5. +> - Automatic application of the `incident` label [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/290964) in GitLab 14.8. If you have at least Reporter [permissions](../../user/permissions.md), you can create an incident manually from the Incidents List or the Issues List. To create an incident from the Incidents List: -1. Navigate to **Monitor > Incidents** and click **Create Incident**. -1. Create a new issue using the `incident` template available when creating it. -1. Create a new issue and assign the `incident` label to it. +1. Navigate to **Monitor > Incidents** and select **Create Incident**. +1. Create a new issue using the `incident` template.  @@ -47,15 +47,15 @@ To create an incident from the Issues List: ### Create incidents automatically **(ULTIMATE)** -With at least the Maintainer [role](../../user/permissions.md), you can enable - GitLab to create incident automatically whenever an alert is triggered: +With at least the Maintainer role, you can enable +GitLab to create incident automatically whenever an alert is triggered: 1. Navigate to **Settings > Monitor > Incidents** and expand **Incidents**. 1. Check the **Create an incident** checkbox. 1. To customize the incident, select an [issue template](../../user/project/description_templates.md#create-an-issue-template). 1. To send [an email notification](paging.md#email-notifications) to users - with the Developer [role](../../user/permissions.md), select + with the Developer role, select **Send a separate email notification to Developers**. Email notifications are also sent to users with the **Maintainer** and **Owner** roles. 1. Click **Save changes**. @@ -68,7 +68,7 @@ You can set up a webhook with PagerDuty to automatically create a GitLab inciden for each PagerDuty incident. This configuration requires you to make changes in both PagerDuty and GitLab: -1. Sign in as a user with the Maintainer [role](../../user/permissions.md). +1. Sign in as a user with the Maintainer role. 1. Navigate to **Settings > Monitor > Incidents** and expand **Incidents**. 1. Select the **PagerDuty integration** tab: @@ -276,7 +276,7 @@ templates. > - [Introduced for Prometheus Integrations](https://gitlab.com/gitlab-org/gitlab/-/issues/13401) in GitLab 12.5. > - [Introduced for HTTP Integrations](https://gitlab.com/gitlab-org/gitlab/-/issues/13402) in GitLab 13.4. -With at least the Maintainer [role](../../user/permissions.md), you can enable +With at least the Maintainer role, you can enable GitLab to close an incident automatically when a **Recovery Alert** is received: 1. Navigate to **Settings > Monitor > Incidents** and expand **Incidents**. diff --git a/doc/operations/incident_management/index.md b/doc/operations/incident_management/index.md index ff5f41e59e9..3b38d4ab427 100644 --- a/doc/operations/incident_management/index.md +++ b/doc/operations/incident_management/index.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/operations/incident_management/integrations.md b/doc/operations/incident_management/integrations.md index a8b455e05a0..fe05cb9fda0 100644 --- a/doc/operations/incident_management/integrations.md +++ b/doc/operations/incident_management/integrations.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -18,7 +18,7 @@ to use this endpoint. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/245331) in GitLab 13.5. -With at least the Maintainer [role](../../user/permissions.md), you can view the list of configured +With at least the Maintainer role, you can view the list of configured alerts integrations by navigating to **Settings > Monitor** in your project's sidebar menu, and expanding the **Alerts** section. The list displays the integration name, type, and status (enabled or disabled): @@ -36,7 +36,7 @@ Enabling the HTTP Endpoint in a GitLab projects activates it to receive alert payloads in JSON format. You can always [customize the payload](#customize-the-alert-payload-outside-of-gitlab) to your liking. -1. Sign in to GitLab as a user with the Maintainer [role](../../user/permissions.md) +1. Sign in to GitLab as a user with the Maintainer role for a project. 1. Navigate to **Settings > Monitor** in your project. 1. Expand the **Alerts** section, and in the **Select integration type** dropdown menu, @@ -53,7 +53,7 @@ In [GitLab Premium](https://about.gitlab.com/pricing/), you can create multiple unique HTTP endpoints to receive alerts from any external source in JSON format, and you can [customize the payload](#customize-the-alert-payload-outside-of-gitlab). -1. Sign in to GitLab as a user with the Maintainer [role](../../user/permissions.md) +1. Sign in to GitLab as a user with the Maintainer role for a project. 1. Navigate to **Settings > Monitor** in your project. 1. Expand the **Alerts** section. @@ -225,7 +225,7 @@ After a [project maintainer or owner](../../user/permissions.md) configures an integration, you can trigger a test alert to confirm your integration works properly. -1. Sign in as a user with at least the Developer [role](../../user/permissions.md). +1. Sign in as a user with at least the Developer role. 1. Navigate to **Settings > Monitor** in your project. 1. Click **Alerts** to expand the section. 1. Click the **{settings}** settings icon on the right side of the integration in [the list](#integrations-list). @@ -279,7 +279,7 @@ active at the same time. To enable Opsgenie integration: -1. Sign in as a user with the Maintainer or Owner [role](../../user/permissions.md). +1. Sign in as a user with the Maintainer or Owner role. 1. Navigate to **Monitor > Alerts**. 1. In the **Integrations** select box, select **Opsgenie**. 1. Select the **Active** toggle. diff --git a/doc/operations/incident_management/oncall_schedules.md b/doc/operations/incident_management/oncall_schedules.md index 2a8f0eac59c..84ab5da77df 100644 --- a/doc/operations/incident_management/oncall_schedules.md +++ b/doc/operations/incident_management/oncall_schedules.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -24,7 +24,7 @@ Set up an on-call schedule for your team to add rotations to. Prerequisite: -- You must have at least the Maintainer [role](../../user/permissions.md). +- You must have at least the Maintainer role. To create an on-call schedule: diff --git a/doc/operations/incident_management/paging.md b/doc/operations/incident_management/paging.md index 6fdf880783a..b6f77de3b4f 100644 --- a/doc/operations/incident_management/paging.md +++ b/doc/operations/incident_management/paging.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/operations/incident_management/status_page.md b/doc/operations/incident_management/status_page.md index 241112df521..fe75c1812c8 100644 --- a/doc/operations/incident_management/status_page.md +++ b/doc/operations/incident_management/status_page.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -41,7 +41,7 @@ Only AWS S3 is supported as a deploy target. Prerequisite: -- You must have at least the Maintainer [role](../../user/permissions.md). +- You must have at least the Maintainer role. To provide GitLab with the AWS account information needed to push content to your Status Page: @@ -175,5 +175,5 @@ within GitLab. Closing the issue triggers a background worker to update the GitLab Status Page website. If you -[make a published issue confidential](../../user/project/issues/confidential_issues.md#making-an-issue-confidential), +[make a published issue confidential](../../user/project/issues/confidential_issues.md#make-an-issue-confidential), GitLab unpublishes it from your GitLab Status Page website. diff --git a/doc/operations/index.md b/doc/operations/index.md index 5a83e47b556..9b988ff561d 100644 --- a/doc/operations/index.md +++ b/doc/operations/index.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/operations/metrics/alerts.md b/doc/operations/metrics/alerts.md index 712ee04e916..702ff944fc5 100644 --- a/doc/operations/metrics/alerts.md +++ b/doc/operations/metrics/alerts.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -99,8 +99,7 @@ GitLab tags each incident issue with the `incident` label automatically. If the does not yet exist, it is also created automatically. If the metric exceeds the threshold of the alert for over 5 minutes, GitLab sends -an email to all [Maintainers and Owners](../../user/permissions.md#project-members-permissions) -of the project. +an email to all Maintainers and Owners of the project. ### Recovery alerts diff --git a/doc/operations/metrics/dashboards/default.md b/doc/operations/metrics/dashboards/default.md index 295c146f0d5..3e14917209a 100644 --- a/doc/operations/metrics/dashboards/default.md +++ b/doc/operations/metrics/dashboards/default.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/operations/metrics/dashboards/develop.md b/doc/operations/metrics/dashboards/develop.md index 38f375c40a6..fc7686c8f86 100644 --- a/doc/operations/metrics/dashboards/develop.md +++ b/doc/operations/metrics/dashboards/develop.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/operations/metrics/dashboards/index.md b/doc/operations/metrics/dashboards/index.md index 9a75703a2f1..a8ca23b7002 100644 --- a/doc/operations/metrics/dashboards/index.md +++ b/doc/operations/metrics/dashboards/index.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/operations/metrics/dashboards/panel_types.md b/doc/operations/metrics/dashboards/panel_types.md index 9b015760fe9..09e969e8af6 100644 --- a/doc/operations/metrics/dashboards/panel_types.md +++ b/doc/operations/metrics/dashboards/panel_types.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/operations/metrics/dashboards/settings.md b/doc/operations/metrics/dashboards/settings.md index f4c37718c52..14da5cf4a04 100644 --- a/doc/operations/metrics/dashboards/settings.md +++ b/doc/operations/metrics/dashboards/settings.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/operations/metrics/dashboards/templating_variables.md b/doc/operations/metrics/dashboards/templating_variables.md index 8ccd334dac3..531693d032f 100644 --- a/doc/operations/metrics/dashboards/templating_variables.md +++ b/doc/operations/metrics/dashboards/templating_variables.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/operations/metrics/dashboards/variables.md b/doc/operations/metrics/dashboards/variables.md index 0008706df40..369bcd1ddeb 100644 --- a/doc/operations/metrics/dashboards/variables.md +++ b/doc/operations/metrics/dashboards/variables.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/operations/metrics/dashboards/yaml.md b/doc/operations/metrics/dashboards/yaml.md index 9d1c270388e..81f1354d3c0 100644 --- a/doc/operations/metrics/dashboards/yaml.md +++ b/doc/operations/metrics/dashboards/yaml.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/operations/metrics/dashboards/yaml_number_format.md b/doc/operations/metrics/dashboards/yaml_number_format.md index ce9e359a587..fd83bff3c08 100644 --- a/doc/operations/metrics/dashboards/yaml_number_format.md +++ b/doc/operations/metrics/dashboards/yaml_number_format.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/operations/metrics/embed.md b/doc/operations/metrics/embed.md index e84c190e08d..6a3712d8377 100644 --- a/doc/operations/metrics/embed.md +++ b/doc/operations/metrics/embed.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -53,7 +53,7 @@ The following requirements must be met for the metric to unfurl: - The `<environment_id>` must correspond to a real environment. - Prometheus must be monitoring the environment. - The GitLab instance must be configured to receive data from the environment. -- The user must be allowed access to the monitoring dashboard for the environment ([Reporter or higher](../../user/permissions.md)). +- The user must have at least the Reporter role for the monitoring dashboard for the environment. - The dashboard must have data within the last 8 hours. If all of the above are true, then the metric unfurls as seen below: diff --git a/doc/operations/metrics/embed_grafana.md b/doc/operations/metrics/embed_grafana.md index 81b1f8a3bc6..5bfb097619d 100644 --- a/doc/operations/metrics/embed_grafana.md +++ b/doc/operations/metrics/embed_grafana.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/operations/metrics/index.md b/doc/operations/metrics/index.md index f09b9f35d88..b04e19807f8 100644 --- a/doc/operations/metrics/index.md +++ b/doc/operations/metrics/index.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/operations/product_analytics.md b/doc/operations/product_analytics.md index 3ff33027042..a55cbe906a0 100644 --- a/doc/operations/product_analytics.md +++ b/doc/operations/product_analytics.md @@ -50,7 +50,7 @@ Feature.disable(:product_analytics, Project.find(<project ID>)) After enabling the feature flag for Product Analytics, you can access the user interface: -1. Sign in to GitLab as a user with at least the Reporter [role](../user/permissions.md). +1. Sign in to GitLab as a user with at least the Reporter role. 1. Navigate to **Monitor > Product Analytics**. The user interface contains: diff --git a/doc/operations/tracing.md b/doc/operations/tracing.md index 09a31c12bf4..044f6800e73 100644 --- a/doc/operations/tracing.md +++ b/doc/operations/tracing.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/public_access/public_access.md b/doc/public_access/public_access.md index e0b07abd4e9..8eee8fc9a6b 100644 --- a/doc/public_access/public_access.md +++ b/doc/public_access/public_access.md @@ -7,7 +7,7 @@ type: reference # Project and group visibility **(FREE)** -GitLab allows users with the Owner [role](../user/permissions.md) to set a project's or group's visibility as: +GitLab allows users with the Owner role to set a project's or group's visibility as: - **Public** - **Internal** @@ -24,7 +24,7 @@ Public projects can be cloned **without any** authentication over HTTPS. They are listed in the public access directory (`/public`) for all users. -**Any signed-in user** has the Guest [role](../user/permissions.md) on the repository. +**Any signed-in user** has the Guest role on the repository. NOTE: By default, `/public` is visible to unauthenticated users. However, if the @@ -39,7 +39,7 @@ Internal projects can be cloned by any signed-in user except They are also listed in the public access directory (`/public`), but only for signed-in users. Any signed-in users except [external users](../user/permissions.md#external-users) have the -Guest [role](../user/permissions.md) on the repository. +Guest role on the repository. NOTE: From July 2019, the `Internal` visibility setting is disabled for new projects, groups, @@ -57,7 +57,7 @@ They appear in the public access directory (`/public`) for project members only. Prerequisite: -- You must have the Owner [role](../user/permissions.md) for a project. +- You must have the Owner role for a project. 1. On the top bar, select **Menu > Projects** and find your project. 1. On the left sidebar, select **Settings > General**. @@ -69,7 +69,7 @@ Prerequisite: Prerequisite: -- You must have the Owner [role](../user/permissions.md) for a group. +- You must have the Owner role for a group. 1. On the top bar, select **Menu > Groups** and find your project. 1. On the left sidebar, select **Settings > General**. diff --git a/doc/push_rules/push_rules.md b/doc/push_rules/push_rules.md index c37853ffe81..3c4b0e10bf9 100644 --- a/doc/push_rules/push_rules.md +++ b/doc/push_rules/push_rules.md @@ -1,273 +1,9 @@ --- -stage: Create -group: Source Code -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" -type: reference, howto +redirect_to: '../user/project/repository/push_rules.md' +remove_date: '2022-05-10' --- -# Push rules **(PREMIUM)** +This document was moved to [another location](../user/project/repository/push_rules.md). -Gain additional control over what can and can't be pushed to your repository by using -regular expressions to reject pushes based on commit contents, branch names or file details. - -GitLab already offers [protected branches](../user/project/protected_branches.md), but there are -cases when you need some specific rules. Some common scenarios: preventing Git tag removal, or -enforcing a special format for commit messages. - -Push rules are [pre-receive Git hooks](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks) you -can enable in a user-friendly interface. They are defined either: - -- Globally if you are an administrator. -- Per project, so you can have different rules applied to different - projects depending on your needs. - -## Use cases - -Every push rule could have its own use case, but let's consider some examples. - -### Commit messages with a specific reference - -Let's assume you have the following requirements for your workflow: - -- every commit should reference a Jira issue, for example: `Refactored css. Fixes JIRA-123.` -- users should not be able to remove Git tags with `git push` - -Write a regular expression that requires the mention -of a Jira issue in the commit message, like `JIRA\-\d+`. - -Now when a user tries to push a commit with a message `Bugfix`, their push is -declined. Only pushing commits with messages like `Bugfix according to JIRA-123` -is accepted. - -### Restrict branch names - -If your company has a strict policy for branch names, you may want the branches to start -with a certain name. This approach enables different -GitLab CI/CD jobs (such as `feature`, `hotfix`, `docker`, `android`) that rely on the -branch name. - -Your developers may not remember that policy, so they might push to -various branches, and CI pipelines might not work as expected. By restricting the -branch names globally in Push Rules, such mistakes are prevented. -All branch names that don't match your push rule are rejected. - -Note that the name of your default branch is always allowed, regardless of the branch naming -regular expression (regex) specified. GitLab is configured this way -because merges typically have the default branch as their target. -If you have other target branches, include them in your regex. (See [Enabling push rules](#enabling-push-rules)). - -The default branch also defaults to being a [protected branch](../user/project/protected_branches.md), -which already limits users from pushing directly. - -Some example regular expressions you can use in push rules: - -- `^JIRA-` Branches must start with `JIRA-`. -- `-JIRA$` Branches must end with `-JIRA`. -- `^[a-z0-9\\-]{4,15}$` Branches must be between `4` and `15` characters long, - accepting only lowercase letters, numbers and dashes. - -#### Default restricted branch names - -> Introduced in GitLab 12.10. - -By default, GitLab restricts certain formats of branch names for security purposes. -40-character hexadecimal names, similar to Git commit hashes, are prohibited. - -### Custom Push Rules **(PREMIUM SELF)** - -It's possible to create custom push rules rather than the push rules available in -**Admin Area > Push Rules** by using more advanced server hooks. - -See [server hooks](../administration/server_hooks.md) for more information. - -## Enabling push rules - -You can create push rules for all new projects to inherit, but they can be overridden -at the project level or the [group level](../user/group/index.md#group-push-rules). - -To create global push rules: - -1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **Push Rules**. - -To override global push rules in a project's settings: - -1. On the top bar, select **Menu > Projects** and find your project. -1. On the left sidebar, select **Settings > Repository**. -1. Expand **Push rules**. -1. Set the rule you want. -1. Select **Save push rules**. - -The following options are available: - -| Push rule | Description | -|---------------------------------|-------------| -| Removal of tags with `git push` | Forbid users to remove Git tags with `git push`. Tags can be deleted through the web UI. | -| Check whether the commit author is a GitLab user | Restrict commits to existing GitLab users (checked against their emails). <sup>1</sup> | -| Reject unverified users | GitLab rejects any commit that was not committed by the same user as the user who pushed it, or where the committer's email address is not [confirmed](../security/user_email_confirmation.md). | -| Check whether commit is signed through GPG | Reject commit when it is not signed through GPG. Read [signing commits with GPG](../user/project/repository/gpg_signed_commits/index.md). | -| Prevent pushing secret files | GitLab rejects any files that are likely to contain secrets. See the [forbidden file names](#prevent-pushing-secrets-to-the-repository). | -| Require expression in commit messages | Only commit messages that match this regular expression are allowed to be pushed. <sup>2</sup> Leave empty to allow any commit message. Uses multiline mode, which can be disabled using `(?-m)`. | -| Reject expression in commit messages | Only commit messages that do not match this regular expression are allowed to be pushed. <sup>2</sup> Leave empty to allow any commit message. Uses multiline mode, which can be disabled using `(?-m)`. | -| Restrict by branch name | Only branch names that match this regular expression are allowed to be pushed. <sup>2</sup> Leave empty to allow all branch names. | -| Restrict by commit author's email | Only commit author's email that match this regular expression are allowed to be pushed. <sup>1</sup> <sup>2</sup> Leave empty to allow any email. | -| Prohibited file names | Any committed filenames that match this regular expression and do not already exist in the repository are not allowed to be pushed. <sup>2</sup> Leave empty to allow any filenames. See [common examples](#prohibited-file-names). | -| Maximum file size | Pushes that contain added or updated files that exceed this file size (in MB) are rejected. Set to 0 to allow files of any size. Files tracked by Git LFS are exempted. | - -1. Checks both the commit author and committer. -1. GitLab uses [RE2 syntax](https://github.com/google/re2/wiki/Syntax) for regular expressions in push rules, and you can test them at the [regex101 regex tester](https://regex101.com/). - -### Caveat to "Reject unsigned commits" push rule - -This push rule ignores commits that are authenticated and created by GitLab -(either through the UI or API). When the **Reject unsigned commits** push rule is -enabled, unsigned commits may still show up in the commit history if a commit was -created **within** GitLab itself. As expected, commits created outside GitLab and -pushed to the repository are rejected. For more information about how GitLab -plans to fix this issue, read [issue #19185](https://gitlab.com/gitlab-org/gitlab/-/issues/19185). - -#### "Reject unsigned commits" push rule disables Web IDE - -In 13.10, if a project has the "Reject unsigned commits" push rule, the user will not be allowed to -commit through GitLab Web IDE. - -To allow committing through the Web IDE on a project with this push rule, a GitLab administrator will -need to disable the feature flag `reject_unsigned_commits_by_gitlab`. This can be done through a -[rails console](../administration/operations/rails_console.md) and running: - -```ruby -Feature.disable(:reject_unsigned_commits_by_gitlab) -``` - -## Prevent pushing secrets to the repository - -> Moved to GitLab Premium in 13.9. - -Secrets, such as credential files and SSH private keys, should never be committed to a version control -system. In GitLab, you can use a predefined list of files to block those files from a -repository. Any merge request containing a file matching the list is blocked from being merged. -Files already committed to the repository are not restricted by this push rule. - -Files blocked by this rule are listed below. For a complete list of criteria, see -[`files_denylist.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/gitlab/checks/files_denylist.yml). - -- AWS CLI credential blobs: - - - `.aws/credentials` - - `aws/credentials` - - `homefolder/aws/credentials` - -- Private RSA SSH keys: - - - `/ssh/id_rsa` - - `/.ssh/personal_rsa` - - `/config/server_rsa` - - `id_rsa` - - `.id_rsa` - -- Private DSA SSH keys: - - - `/ssh/id_dsa` - - `/.ssh/personal_dsa` - - `/config/server_dsa` - - `id_dsa` - - `.id_dsa` - -- Private ed25519 SSH keys: - - - `/ssh/id_ed25519` - - `/.ssh/personal_ed25519` - - `/config/server_ed25519` - - `id_ed25519` - - `.id_ed25519` - -- Private ECDSA SSH keys: - - - `/ssh/id_ecdsa` - - `/.ssh/personal_ecdsa` - - `/config/server_ecdsa` - - `id_ecdsa` - - `.id_ecdsa` - -- Any files ending with these suffixes: - - - `*.pem` - - `*.key` - - `*.history` - - `*_history` - -### Prevent pushing secrets to all projects - -To set a global push rule to prevent pushing secrets to all projects: - -1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **Push Rules**. -1. Expand **Push rules**. -1. Select **Prevent pushing secret files**. -1. Select **Save push rules**. - -### Prevent pushing secrets to a project - -The push rule of a project overrides the global push rule. - -To prevent pushing secrets to a project: - -1. On the top bar, select **Menu > Projects** and find your project. -1. On the left sidebar, select **Settings > Repository**. -1. Expand **Push rules**. -1. Select **Prevent pushing secret files**. -1. Select **Save push rules**. - -## Prohibited file names - -> Moved to GitLab Premium in 13.9. - -Each filename contained in a Git push is compared to the regular expression in this field. Filenames in Git consist of both the file's name and any directory that may precede it. A singular regular expression can contain multiple independent matches used as exclusions. File names can be broadly matched to any location in the repository, or restricted to specific locations. Filenames can also be partial matches used to exclude file types by extension. - -The following examples make use of regex string boundary characters which match the beginning of a string (`^`), and the end (`$`). They also include instances where either the directory path or the filename can include `.` or `/`. Both of these special regex characters have to be escaped with a backslash `\\` to be used as normal characters in a match condition. - -Example: prevent pushing any `.exe` files to any location in the repository. This is an example of a partial match, which can match any filename that contains `.exe` at the end: - -```plaintext -\.exe$ -``` - -Example: prevent a specific configuration file in the repository root from being pushed: - -```plaintext -^config\.yml$ -``` - -Example: prevent a specific configuration file in a known directory from being pushed: - -```plaintext -^directory-name\/config\.yml$ -``` - -Example: prevent the specific file named `install.exe` from being pushed to any -location in the repository. The parenthesized expression `(^|\/)` matches either -a file following a directory separator or a file in the root directory of the repository: - -```plaintext -(^|\/)install\.exe$ -``` - -Example: combining all of the above in a single expression. The preceding expressions rely -on the end-of-string character `$`. We can move that part of each expression to the -end of the grouped collection of match conditions where it is appended to all matches: - -```plaintext -(\.exe|^config\.yml|^directory-name\/config\.yml|(^|\/)install\.exe)$ -``` - -<!-- ## Troubleshooting - -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. - -Each scenario can be a third-level heading, e.g. `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +<!-- This redirect file can be deleted after <2022-05-10>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md index 539908cd7bc..30cd0f8f511 100644 --- a/doc/raketasks/backup_restore.md +++ b/doc/raketasks/backup_restore.md @@ -828,6 +828,7 @@ For installations from source: The `CRON=1` environment setting directs the backup script to hide all progress output if there aren't any errors. This is recommended to reduce cron spam. +When troubleshooting backup problems, however, replace `CRON=1` with `--trace` to log verbosely. ### Limit backup lifetime for local files (prune old backups) @@ -943,7 +944,7 @@ First ensure your backup tar file is in the backup directory described in the ```shell sudo cp 11493107454_2018_04_25_10.6.4-ce_gitlab_backup.tar /var/opt/gitlab/backups/ -sudo chown git.git /var/opt/gitlab/backups/11493107454_2018_04_25_10.6.4-ce_gitlab_backup.tar +sudo chown git:git /var/opt/gitlab/backups/11493107454_2018_04_25_10.6.4-ce_gitlab_backup.tar ``` Stop the processes that are connected to the database. Leave the rest of GitLab @@ -965,6 +966,7 @@ sudo gitlab-backup restore BACKUP=11493107454_2018_04_25_10.6.4-ce ``` Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:restore` instead. +Some [known non-blocking error messages may appear](#restoring-database-backup-using-omnibus-packages-outputs-warnings). WARNING: `gitlab-rake gitlab:backup:restore` doesn't set the correct file system diff --git a/doc/raketasks/generate_sample_prometheus_data.md b/doc/raketasks/generate_sample_prometheus_data.md index f014b82cca1..cdc95c1f3cc 100644 --- a/doc/raketasks/generate_sample_prometheus_data.md +++ b/doc/raketasks/generate_sample_prometheus_data.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/security/asset_proxy.md b/doc/security/asset_proxy.md index 45c1c71158a..e4849b1b658 100644 --- a/doc/security/asset_proxy.md +++ b/doc/security/asset_proxy.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/security/crime_vulnerability.md b/doc/security/crime_vulnerability.md index 1abb0c9e918..8288f7f6a74 100644 --- a/doc/security/crime_vulnerability.md +++ b/doc/security/crime_vulnerability.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- diff --git a/doc/security/index.md b/doc/security/index.md index ab554e9135f..da3fa761f3f 100644 --- a/doc/security/index.md +++ b/doc/security/index.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments comments: false type: index @@ -30,3 +30,5 @@ type: index ## Securing your GitLab installation Consider access control features like [Sign up restrictions](../user/admin_area/settings/sign_up_restrictions.md) and [Authentication options](../topics/authentication/) to harden your GitLab instance and minimize the risk of unwanted user account creation. + +Self-hosting GitLab customers and administrators are responsible for the security of their underlying hosts, and for keeping GitLab itself up to date. It is important to [regularly patch GitLab](../policy/maintenance.md), patch your operating system and its software, and harden your hosts in accordance with vendor guidance. diff --git a/doc/security/information_exclusivity.md b/doc/security/information_exclusivity.md index 07b5a688671..0d55881c147 100644 --- a/doc/security/information_exclusivity.md +++ b/doc/security/information_exclusivity.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: concepts --- diff --git a/doc/security/password_length_limits.md b/doc/security/password_length_limits.md index 1cfff358c9d..04c3a5c99e1 100644 --- a/doc/security/password_length_limits.md +++ b/doc/security/password_length_limits.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference, howto --- diff --git a/doc/security/password_storage.md b/doc/security/password_storage.md index 6b71933b1ae..b4c2e27c952 100644 --- a/doc/security/password_storage.md +++ b/doc/security/password_storage.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- diff --git a/doc/security/passwords_for_integrated_authentication_methods.md b/doc/security/passwords_for_integrated_authentication_methods.md index 7281b310a30..d4eb16c07e7 100644 --- a/doc/security/passwords_for_integrated_authentication_methods.md +++ b/doc/security/passwords_for_integrated_authentication_methods.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- diff --git a/doc/security/project_import_decompressed_archive_size_limits.md b/doc/security/project_import_decompressed_archive_size_limits.md index 9727ba1c5f0..5082d917748 100644 --- a/doc/security/project_import_decompressed_archive_size_limits.md +++ b/doc/security/project_import_decompressed_archive_size_limits.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference, howto --- diff --git a/doc/security/rate_limits.md b/doc/security/rate_limits.md index 14fc526ca7e..a9b066631e7 100644 --- a/doc/security/rate_limits.md +++ b/doc/security/rate_limits.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference, howto --- @@ -41,6 +41,7 @@ You can set these rate limits in the Admin Area of your instance: - [Git LFS rate limits](../user/admin_area/settings/git_lfs_rate_limits.md) - [Files API rate limits](../user/admin_area/settings/files_api_rate_limits.md) - [Deprecated API rate limits](../user/admin_area/settings/deprecated_api_rate_limits.md) +- [GitLab Pages rate limits](../administration/pages/index.md#rate-limits) You can set these rate limits using the Rails console: @@ -89,7 +90,7 @@ The **rate limit** is 5 requests per minute per user. ### Users sign up -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/77835) in GitLab 14.7. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339151) in GitLab 14.7. There is a rate limit per IP address on the `/users/sign_up` endpoint. This is to mitigate attempts to misuse the endpoint. For example, to mass discover usernames or email addresses in use. @@ -98,19 +99,19 @@ The **rate limit** is 20 calls per minute per IP address. ### Update username -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/77221) in GitLab 14.7. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339152) in GitLab 14.7. -There is a rate limit on the update username action. This is enforced to mitigate misuse of the feature. For example, to mass discover +There is a rate limit on how frequently a username can be changed. This is enforced to mitigate misuse of the feature. For example, to mass discover which usernames are in use. The **rate limit** is 10 calls per minute per signed-in user. ### Username exists -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/77119) in GitLab 14.7. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/29040) in GitLab 14.7. -There is a rate limit for the internal endpoint `/users/:username/exists`, used by registration to perform a client-side validation for -uniqueness of the chosen username. This is to mitigate the risk of misuses, such as mass discovery of usernames in use. +There is a rate limit for the internal endpoint `/users/:username/exists`, used upon sign up to check if a chosen username has already been taken. +This is to mitigate the risk of misuses, such as mass discovery of usernames in use. The **rate limit** is 20 calls per minute per IP address. diff --git a/doc/security/reset_user_password.md b/doc/security/reset_user_password.md index f67b1934dc5..1940c5be73a 100644 --- a/doc/security/reset_user_password.md +++ b/doc/security/reset_user_password.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- diff --git a/doc/security/ssh_keys_restrictions.md b/doc/security/ssh_keys_restrictions.md index a7d852e2754..2e4a737f9aa 100644 --- a/doc/security/ssh_keys_restrictions.md +++ b/doc/security/ssh_keys_restrictions.md @@ -1,7 +1,7 @@ --- type: reference, howto stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -13,9 +13,9 @@ NIST). Some organizations deploying GitLab need to enforce minimum key strength, either to satisfy internal security policy or for regulatory compliance. -Similarly, certain standards groups recommend using RSA, ECDSA, or ED25519 over -the older DSA, and administrators may need to limit the allowed SSH key -algorithms. +Similarly, certain standards groups recommend using RSA, ECDSA, ED25519, +ECDSA_SK, or ED25519_SK over the older DSA, and administrators may need to +limit the allowed SSH key algorithms. GitLab allows you to restrict the allowed SSH key technology as well as specify the minimum key length for each technology: @@ -45,6 +45,8 @@ By default, the GitLab.com and self-managed settings for the - DSA SSH keys are forbidden ([since GitLab 11.0](https://about.gitlab.com/releases/2018/06/22/gitlab-11-0-released/#support-for-dsa-ssh-keys)). - ECDSA SSH keys are allowed. - ED25519 SSH keys are allowed. +- ECDSA_SK SSH keys are allowed (GitLab 14.8 and later). +- ED25519_SK SSH keys are allowed (GitLab 14.8 and later). <!-- ## Troubleshooting diff --git a/doc/security/token_overview.md b/doc/security/token_overview.md index 578bb03563f..a2119c86268 100644 --- a/doc/security/token_overview.md +++ b/doc/security/token_overview.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -44,8 +44,21 @@ are scoped to a project. As with [Personal access tokens](#personal-access-token - The GitLab registry. You can limit the scope and expiration date of project access tokens. When you -create a project access token, GitLab creates a [project bot user](../user/project/settings/project_access_tokens.md#project-bot-users). Project -bot users are service accounts and do not count as licensed seats. +create a project access token, GitLab creates a [bot user for projects](../user/project/settings/project_access_tokens.md#bot-users-for-projects). +Bot users for projects are service accounts and do not count as licensed seats. + +## Group access tokens + +[Group access tokens](../user/group/settings/group_access_tokens.md#group-access-tokens) +are scoped to a group. As with [Personal access tokens](#personal-access-tokens), you can use them to authenticate with: + +- The GitLab API. +- GitLab repositories. +- The GitLab registry. + +You can limit the scope and expiration date of group access tokens. When you +create a group access token, GitLab creates a [bot user for groups](../user/group/settings/group_access_tokens.md#bot-users-for-groups). +Bot users for groups are service accounts and do not count as licensed seats. ## Deploy tokens diff --git a/doc/security/two_factor_authentication.md b/doc/security/two_factor_authentication.md index b83d81722fa..e8bb627ccbd 100644 --- a/doc/security/two_factor_authentication.md +++ b/doc/security/two_factor_authentication.md @@ -1,7 +1,7 @@ --- type: howto stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -116,8 +116,10 @@ reactivate 2FA from scratch if they want to use it again. WARNING: This feature might not be available to you. Check the **version history** note above for details. -Two-factor authentication can be enforced for Git over SSH operations. The one-time password (OTP) -verification can be done via a GitLab Shell command: +Two-factor authentication can be enforced for Git over SSH operations. However, we recommend using +[ED25519_SK](../ssh/index.md#ed25519_sk-ssh-keys) or [ECDSA_SK](../ssh/index.md#ecdsa_sk-ssh-keys) SSH keys instead. + +The one-time password (OTP) verification can be done using a command: ```shell ssh git@<hostname> 2fa_verify diff --git a/doc/security/unlock_user.md b/doc/security/unlock_user.md index 057d4e87efa..f2ad6696b9a 100644 --- a/doc/security/unlock_user.md +++ b/doc/security/unlock_user.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- diff --git a/doc/security/user_email_confirmation.md b/doc/security/user_email_confirmation.md index 8baddaf1383..54920b15362 100644 --- a/doc/security/user_email_confirmation.md +++ b/doc/security/user_email_confirmation.md @@ -1,7 +1,7 @@ --- type: howto stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/security/user_file_uploads.md b/doc/security/user_file_uploads.md index 734a4cde7e8..e8b0c08e240 100644 --- a/doc/security/user_file_uploads.md +++ b/doc/security/user_file_uploads.md @@ -1,12 +1,14 @@ --- type: reference stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # User File Uploads **(FREE)** +> - In GitLab 14.8 and later, [authorization checks are enforced](https://gitlab.com/gitlab-org/gitlab/-/issues/26781) on media uploads. This change is being [rolled out incrementally](https://gitlab.com/gitlab-org/gitlab/-/issues/352291) on GitLab.com in 14.9. + Images that are attached to issues, merge requests, or comments do not require authentication to be viewed if they are accessed directly by URL. This direct URL contains a random 32-character ID that prevents unauthorized diff --git a/doc/security/webhooks.md b/doc/security/webhooks.md index 621e6d595bf..07b35ccebe8 100644 --- a/doc/security/webhooks.md +++ b/doc/security/webhooks.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: concepts, reference, howto --- diff --git a/doc/ssh/index.md b/doc/ssh/index.md index 6196ee5465b..35ca9a23179 100644 --- a/doc/ssh/index.md +++ b/doc/ssh/index.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" type: howto, reference --- @@ -28,6 +28,8 @@ To view the version of SSH installed on your system, run `ssh -V`. To communicate with GitLab, you can use the following SSH key types: - [ED25519](#ed25519-ssh-keys) +- [ED25519_SK](#ed25519_sk-ssh-keys) (Available in GitLab 14.8 and later.) +- [ECDSA_SK](#ecdsa_sk-ssh-keys) (Available in GitLab 14.8 and later.) - [RSA](#rsa-ssh-keys) - DSA ([Deprecated](https://about.gitlab.com/releases/2018/06/22/gitlab-11-0-released/#support-for-dsa-ssh-keys) in GitLab 11.0.) - ECDSA (As noted in [Practical Cryptography With Go](https://leanpub.com/gocrypto/read#leanpub-auto-ecdsa), the security issues related to DSA also apply to ECDSA.) @@ -42,6 +44,20 @@ suggests that [ED25519](https://ed25519.cr.yp.to/) keys are more secure and perf OpenSSH 6.5 introduced ED25519 SSH keys in 2014 and they should be available on most operating systems. +### ED25519_SK SSH keys + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78934) in GitLab 14.8. + +To use ED25519_SK SSH keys on GitLab, your local client and GitLab server +must have [OpenSSH 8.2](https://www.openssh.com/releasenotes.html#8.2) or later installed. + +### ECDSA_SK SSH keys + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78934) in GitLab 14.8. + +To use ECDSA_SK SSH keys on GitLab, your local client and GitLab server +must have [OpenSSH 8.2](https://www.openssh.com/releasenotes.html#8.2) or later installed. + ### RSA SSH keys Available documentation suggests that ED25519 is more secure than RSA. @@ -64,6 +80,8 @@ Before you create a key pair, see if a key pair already exists. | Algorithm | Public key | Private key | | --------- | ---------- | ----------- | | ED25519 (preferred) | `id_ed25519.pub` | `id_ed25519` | + | ED25519_SK | `id_ed25519_sk.pub` | `id_ed25519_sk` | + | ECDSA_SK | `id_ecdsa_sk.pub` | `id_ecdsa_sk` | | RSA (at least 2048-bit key size) | `id_rsa.pub` | `id_rsa` | | DSA (deprecated) | `id_dsa.pub` | `id_dsa` | | ECDSA | `id_ecdsa.pub` | `id_ecdsa` | @@ -177,6 +195,67 @@ OpenSSH format. ssh-keygen -o -t rsa -b 4096 -C "<comment>" ``` +## Generate an SSH key pair for a FIDO/U2F hardware security key + +To generate ED25519_SK or ECDSA_SK SSH keys, you must use OpenSSH 8.2 or later. + +1. Insert a hardware security key into your computer. +1. Open a terminal. +1. Type `ssh-keygen -t` followed by the key type and an optional comment. + This comment is included in the `.pub` file that's created. + You may want to use an email address for the comment. + + For example, for ED25519_SK: + + ```shell + ssh-keygen -t ed25519-sk -C "<comment>" + ``` + + For ECDSA_SK: + + ```shell + ssh-keygen -t ecdsa-sk -C "<comment>" + ``` + + If your security key supports FIDO2 resident keys, you can enable this when + creating your SSH key: + + ```shell + ssh-keygen -t ed25519-sk -O resident -C "<comment>" + ``` + + `-O resident` indicates that the key should be stored on the FIDO authenticator itself. + Resident key is easier to import to a new computer because it can be loaded directly + from the security key by [`ssh-add -K`](https://man.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man1/ssh-add.1#K) + or [`ssh-keygen -K`](https://man.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man1/ssh-keygen#K). + +1. Select Enter. Output similar to the following is displayed: + + ```plaintext + Generating public/private ed25519-sk key pair. + You may need to touch your authenticator to authorize key generation. + ``` + +1. Touch the button on the hardware security key. + +1. Accept the suggested filename and directory: + + ```plaintext + Enter file in which to save the key (/home/user/.ssh/id_ed25519_sk): + ``` + +1. Specify a [passphrase](https://www.ssh.com/academy/ssh/passphrase): + + ```plaintext + Enter passphrase (empty for no passphrase): + Enter same passphrase again: + ``` + +1. A confirmation is displayed, including information about where your files are stored. + +A public and private key are generated. +[Add the public SSH key to your GitLab account](#add-an-ssh-key-to-your-gitlab-account). + ## Add an SSH key to your GitLab account To use SSH with GitLab, copy your public key to your GitLab account. @@ -210,7 +289,8 @@ To use SSH with GitLab, copy your public key to your GitLab account. 1. On the left sidebar, select **SSH Keys**. 1. In the **Key** box, paste the contents of your public key. If you manually copied the key, make sure you copy the entire key, - which starts with `ssh-ed25519` or `ssh-rsa`, and may end with a comment. + which starts with `ssh-rsa`, `ssh-dss`, `ecdsa-sha2-nistp256`, `ecdsa-sha2-nistp384`, `ecdsa-sha2-nistp521`, + `ssh-ed25519`, `sk-ecdsa-sha2-nistp256@openssh.com`, or `sk-ssh-ed25519@openssh.com`, and may end with a comment. 1. In the **Title** box, type a description, like `Work Laptop` or `Home Workstation`. 1. Optional. In the **Expires at** box, select an expiration date. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/36243) in GitLab 12.9.) @@ -219,7 +299,7 @@ To use SSH with GitLab, copy your public key to your GitLab account. you from using the key. Administrators can view expiration dates and use them for guidance when [deleting keys](../user/admin_area/credentials_inventory.md#delete-a-users-ssh-key). - GitLab 14.0 and later, the expiration date is enforced. Administrators can - [allow expired keys to be used](../user/admin_area/settings/account_and_limit_settings.md#allow-expired-ssh-keys-to-be-used). + [allow expired keys to be used](../user/admin_area/settings/account_and_limit_settings.md#allow-expired-ssh-keys-to-be-used-deprecated). - GitLab checks all SSH keys at 02:00 AM UTC every day. It emails an expiration notice for all SSH keys that expire on the current date. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322637) in GitLab 13.11.) - GitLab checks all SSH keys at 01:00 AM UTC every day. It emails an expiration notice for all SSH keys that are scheduled to expire seven days from now. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322637) in GitLab 13.11.) 1. Select **Add key**. @@ -318,7 +398,8 @@ on the files make them readable to you but not accessible to others. ## Configure two-factor authentication (2FA) You can set up two-factor authentication (2FA) for -[Git over SSH](../security/two_factor_authentication.md#2fa-for-git-over-ssh-operations). +[Git over SSH](../security/two_factor_authentication.md#2fa-for-git-over-ssh-operations). We recommend using +[ED25519_SK](#ed25519_sk-ssh-keys) or [ECDSA_SK](#ecdsa_sk-ssh-keys) SSH keys. ## Use EGit on Eclipse diff --git a/doc/subscriptions/bronze_starter.md b/doc/subscriptions/bronze_starter.md index 2c66d32f669..7ca3a3ffade 100644 --- a/doc/subscriptions/bronze_starter.md +++ b/doc/subscriptions/bronze_starter.md @@ -17,9 +17,7 @@ the tiers are no longer mentioned in GitLab documentation: - [Activate GitLab EE with a license](../user/admin_area/license.md) - [Add a help message to the sign-in page](../user/admin_area/settings/help_page.md#add-a-help-message-to-the-sign-in-page) -- [Burndown and burnup charts](../user/project/milestones/burndown_and_burnup_charts.md), - including [per-project charts](../user/project/milestones/index.md#project-burndown-charts) and - [per-group charts](../user/project/milestones/index.md#group-burndown-charts) +- [Burndown and burnup charts](../user/project/milestones/burndown_and_burnup_charts.md) in the [Milestone View](../user/project/milestones/index.md#burndown-charts), - [Code owners](../user/project/code_owners.md) - Description templates: - [Setting a default template for merge requests and issues](../user/project/description_templates.md#set-a-default-template-for-merge-requests-and-issues) @@ -100,7 +98,7 @@ the tiers are no longer mentioned in GitLab documentation: - Runners: - Run pipelines in the parent project [for merge requests from a forked project](../ci/pipelines/merge_request_pipelines.md#run-pipelines-in-the-parent-project) - [Shared runners CI/CD minutes](../ci/pipelines/cicd_minutes.md) -- [Push rules](../push_rules/push_rules.md) +- [Push rules](../user/project/repository/push_rules.md) - SAML for self-managed GitLab instance: - [Administrator groups](../integration/saml.md#administrator-groups) - [Auditor groups](../integration/saml.md#auditor-groups) diff --git a/doc/subscriptions/gitlab_com/index.md b/doc/subscriptions/gitlab_com/index.md index b72fad02b3d..c94d90f61b8 100644 --- a/doc/subscriptions/gitlab_com/index.md +++ b/doc/subscriptions/gitlab_com/index.md @@ -14,7 +14,7 @@ You don't need to install anything to use GitLab SaaS, you only need to - [A subscription](https://about.gitlab.com/pricing/). - [The number of seats you want](#how-seat-usage-is-determined). -The subscription determines which features are available for your private projects. Public projects automatically get **Ultimate** tier features. +The subscription determines which features are available for your private projects. Organizations with public open source projects can actively apply to our [GitLab for Open Source Program](https://about.gitlab.com/solutions/open-source/join/). Qualifying open source projects also get 50,000 CI/CD minutes and free access to the **Ultimate** tier through the [GitLab for Open Source program](https://about.gitlab.com/solutions/open-source/). @@ -44,7 +44,7 @@ To subscribe to GitLab SaaS: Prerequisite: -- You must have the Owner [role](../../user/permissions.md) for the group. +- You must have the Owner role for the group. To see the status of your GitLab SaaS subscription: @@ -79,10 +79,12 @@ Every user is included in seat usage, with the following exceptions: - Users who are pending approval. - Members with the Guest role on an Ultimate subscription. -- GitLab-created service accounts: `Ghost User` and bots - ([`Support Bot`](../../user/project/service_desk.md#support-bot-user), - [`Project bot users`](../../user/project/settings/project_access_tokens.md#project-bot-users), and - so on.) +- GitLab-created service accounts: + - [Ghost User](../../user/profile/account/delete_account.md#associated-records). + - Bots such as: + - [Support Bot](../../user/project/service_desk.md#support-bot-user). + - [Bot users for projects](../../user/project/settings/project_access_tokens.md#bot-users-for-projects). + - [Bot users for groups](../../user/group/settings/group_access_tokens.md#bot-users-for-groups). Seat usage is reviewed [quarterly or annually](../quarterly_reconciliation.md). @@ -254,7 +256,7 @@ expiration date without a gap in available service. An invoice is generated for the renewal and available for viewing or download on the [View invoices](https://customers.gitlab.com/receipts) page. -#### Enable automatic renewal +#### Enable or disable automatic renewal To view or change automatic subscription renewal (at the same tier as the previous period), log in to the [Customers Portal](https://customers.gitlab.com/customers/sign_in), and: @@ -290,21 +292,28 @@ for more information. ### Purchase additional CI/CD minutes You can [purchase additional minutes](../../ci/pipelines/cicd_minutes.md#purchase-additional-cicd-minutes) -for your personal or group namespace. +for your personal or group namespace. CI/CD minutes are a **one-time purchase**, so they do not renew. -## Storage subscription +## Add-on subscription for additional Storage and Transfer + +NOTE: +Free namespaces are subject to a 5GB storage and 10GB transfer [soft limit](https://about.gitlab.com/pricing). Once all storage is available to view in the usage quota workflow, GitLab will automatically enforce the namespace storage limit and the project limit will be removed. This change will be announced separately. The storage and transfer add-on can be purchased to increase the limits. Projects have a free storage quota of 10 GB. To exceed this quota you must first [purchase one or -more storage subscription units](#purchase-more-storage). Each unit provides 10 GB of additional +more storage subscription units](#purchase-more-storage-and-transfer). Each unit provides 10 GB of additional storage per namespace. A storage subscription is renewed annually. For more details, see [Usage Quotas](../../user/usage_quotas.md). When the amount of purchased storage reaches zero, all projects over the free storage quota are locked. Projects can only be unlocked by purchasing more storage subscription units. -### Purchase more storage +### Purchase more storage and transfer + +You can purchase a storage subscription for your personal or group namespace. -You can purchase storage for your personal or group namespace. +NOTE: +Storage subscriptions **[renew automatically](#automatic-renewal) each year**. +You can [cancel the subscription](#enable-or-disable-automatic-renewal) to disable the automatic renewal. #### For your personal namespace diff --git a/doc/subscriptions/index.md b/doc/subscriptions/index.md index 4e4c0309c3d..3f9a1898505 100644 --- a/doc/subscriptions/index.md +++ b/doc/subscriptions/index.md @@ -150,11 +150,11 @@ To change the namespace linked to a subscription: 1. Navigate to the **Manage Purchases** page. 1. Select **Change linked namespace**. 1. Select the desired group from the **This subscription is for** dropdown. For a group to appear - here, you must have the Owner [role](../user/permissions.md) + here, you must have the Owner role for that group. 1. Select **Proceed to checkout**. -Subscription charges are calculated based on the total number of users in a group, including its subgroups and nested projects. If the total number of users exceeds the number of seats in your subscription, your account is charged for the additional users. +Subscription charges are calculated based on the total number of users in a group, including its subgroups and nested projects. If the [total number of users](gitlab_com/index.md#view-seat-usage) exceeds the number of seats in your subscription, your account is charged for the additional users and you need to pay for the overage before you can change the linked namespace. Only one namespace can be linked to a subscription. diff --git a/doc/subscriptions/self_managed/index.md b/doc/subscriptions/self_managed/index.md index 5ac2c5a5037..cb9db3673ac 100644 --- a/doc/subscriptions/self_managed/index.md +++ b/doc/subscriptions/self_managed/index.md @@ -65,10 +65,12 @@ billable user, with the following exceptions: - Users who are [pending approval](../../user/admin_area/moderate_users.md#users-pending-approval). - Members with the Guest role on an Ultimate subscription. - Users without project or group memberships on an Ultimate subscription. -- GitLab-created service accounts: `Ghost User` and bots - ([`Support Bot`](../../user/project/service_desk.md#support-bot-user), - [`Project bot users`](../../user/project/settings/project_access_tokens.md#project-bot-users), and - so on.) +- GitLab-created service accounts: + - [Ghost User](../../user/profile/account/delete_account.md#associated-records). + - Bots such as: + - [Support Bot](../../user/project/service_desk.md#support-bot-user). + - [Bot users for projects](../../user/project/settings/project_access_tokens.md#bot-users-for-projects). + - [Bot users for groups](../../user/group/settings/group_access_tokens.md#bot-users-for-groups). **Billable users** as reported in the `/admin` section is updated once per day. @@ -307,6 +309,11 @@ The **License Usage** CSV includes the following details: - Date the count was recorded - Active user count +NOTES: + +- All timestamps are displayed in UTC. +- A custom format is used for [dates](https://gitlab.com/gitlab-org/gitlab/blob/3be39f19ac3412c089be28553e6f91b681e5d739/config/initializers/date_time_formats.rb#L7) and [times](https://gitlab.com/gitlab-org/gitlab/blob/3be39f19ac3412c089be28553e6f91b681e5d739/config/initializers/date_time_formats.rb#L13) in CSV files. + ## Renew your subscription To renew your subscription, diff --git a/doc/system_hooks/system_hooks.md b/doc/system_hooks/system_hooks.md index 44429754747..71d7e7f1426 100644 --- a/doc/system_hooks/system_hooks.md +++ b/doc/system_hooks/system_hooks.md @@ -79,8 +79,12 @@ X-Gitlab-Event: System Hook "updated_at": "2012-07-21T07:38:22Z", "event_name": "project_create", "name": "StoreCloud", - "owner_email": "johnsmith@gmail.com", + "owner_email": "johnsmith@example.com", "owner_name": "John Smith", + "owners": [{ + "name": "John", + "email": "user1@example.com" + }], "path": "storecloud", "path_with_namespace": "jsmith/storecloud", "project_id": 74, @@ -96,8 +100,12 @@ X-Gitlab-Event: System Hook "updated_at": "2012-07-21T07:38:22Z", "event_name": "project_destroy", "name": "Underscore", - "owner_email": "johnsmith@gmail.com", + "owner_email": "johnsmith@example.com", "owner_name": "John Smith", + "owners": [{ + "name": "John", + "email": "user1@example.com" + }], "path": "underscore", "path_with_namespace": "jsmith/underscore", "project_id": 73, @@ -117,7 +125,11 @@ X-Gitlab-Event: System Hook "path_with_namespace": "jsmith/underscore", "project_id": 73, "owner_name": "John Smith", - "owner_email": "johnsmith@gmail.com", + "owner_email": "johnsmith@example.com", + "owners": [{ + "name": "John", + "email": "user1@example.com" + }], "project_visibility": "internal", "old_path_with_namespace": "jsmith/overscore" } @@ -138,7 +150,11 @@ Please refer to `group_rename` and `user_rename` for that case. "path_with_namespace": "scores/underscore", "project_id": 73, "owner_name": "John Smith", - "owner_email": "johnsmith@gmail.com", + "owner_email": "johnsmith@example.com", + "owners": [{ + "name": "John", + "email": "user1@example.com" + }], "project_visibility": "internal", "old_path_with_namespace": "jsmith/overscore" } @@ -152,8 +168,12 @@ Please refer to `group_rename` and `user_rename` for that case. "updated_at": "2012-07-21T07:38:22Z", "event_name": "project_update", "name": "StoreCloud", - "owner_email": "johnsmith@gmail.com", + "owner_email": "johnsmith@example.com", "owner_name": "John Smith", + "owners": [{ + "name": "John", + "email": "user1@example.com" + }], "path": "storecloud", "path_with_namespace": "jsmith/storecloud", "project_id": 74, @@ -173,7 +193,7 @@ Please refer to `group_rename` and `user_rename` for that case. "project_name": "StoreCloud", "project_path": "storecloud", "project_path_with_namespace": "jsmith/storecloud", - "user_email": "johnsmith@gmail.com", + "user_email": "johnsmith@example.com", "user_name": "John Smith", "user_username": "johnsmith", "user_id": 41, @@ -193,7 +213,7 @@ Please refer to `group_rename` and `user_rename` for that case. "project_name": "StoreCloud", "project_path": "storecloud", "project_path_with_namespace": "jsmith/storecloud", - "user_email": "johnsmith@gmail.com", + "user_email": "johnsmith@example.com", "user_name": "John Smith", "user_username": "johnsmith", "user_id": 41, @@ -213,7 +233,7 @@ Please refer to `group_rename` and `user_rename` for that case. "project_name": "StoreCloud", "project_path": "storecloud", "project_path_with_namespace": "jsmith/storecloud", - "user_email": "johnsmith@gmail.com", + "user_email": "johnsmith@example.com", "user_name": "John Smith", "user_username": "johnsmith", "user_id": 41, @@ -360,7 +380,7 @@ If the user is blocked via LDAP, `state` is `ldap_blocked`. "group_id": 78, "group_name": "StoreCloud", "group_path": "storecloud", - "user_email": "johnsmith@gmail.com", + "user_email": "johnsmith@example.com", "user_name": "John Smith", "user_username": "johnsmith", "user_id": 41 @@ -378,7 +398,7 @@ If the user is blocked via LDAP, `state` is `ldap_blocked`. "group_id": 78, "group_name": "StoreCloud", "group_path": "storecloud", - "user_email": "johnsmith@gmail.com", + "user_email": "johnsmith@example.com", "user_name": "John Smith", "user_username": "johnsmith", "user_id": 41 @@ -396,7 +416,7 @@ If the user is blocked via LDAP, `state` is `ldap_blocked`. "group_id": 78, "group_name": "StoreCloud", "group_path": "storecloud", - "user_email": "johnsmith@gmail.com", + "user_email": "johnsmith@example.com", "user_name": "John Smith", "user_username": "johnsmith", "user_id": 41 diff --git a/doc/tools/email.md b/doc/tools/email.md index 0429b9b952e..0a3e37719a4 100644 --- a/doc/tools/email.md +++ b/doc/tools/email.md @@ -26,7 +26,7 @@ For information about email notifications originating from GitLab, read 1. On the left sidebar, select **Overview > Users**. 1. Select **Send email to users**. -  +  1. Compose an email and choose where to send it (all users or users of a chosen group or project). The email body only supports plain text messages. diff --git a/doc/topics/authentication/index.md b/doc/topics/authentication/index.md index 2a301e6ff5b..9ab6235164a 100644 --- a/doc/topics/authentication/index.md +++ b/doc/topics/authentication/index.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/topics/autodevops/customize.md b/doc/topics/autodevops/customize.md index 177e10b99b9..089f9f8e7cc 100644 --- a/doc/topics/autodevops/customize.md +++ b/doc/topics/autodevops/customize.md @@ -190,6 +190,9 @@ You can override the default values in the `values.yaml` file in the `HELM_UPGRADE_VALUES_FILE` [CI/CD variable](#cicd-variables) with the path and name. +Some values can not be overridden with the options above. Settings like `replicaCount` should instead be overridden with the `REPLICAS` +[build and deployment](#build-and-deployment) CI/CD variable. Follow [this issue](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image/-/issues/31) for more information. + NOTE: For GitLab 12.5 and earlier, use the `HELM_UPGRADE_EXTRA_ARGS` variable to override the default chart values by setting `HELM_UPGRADE_EXTRA_ARGS` to `--values <my-values.yaml>`. @@ -433,7 +436,7 @@ applications. | `KUBE_NAMESPACE` | The namespace used for deployments. When using certificate-based clusters, [this value should not be overwritten directly](../../user/project/clusters/deploy_to_cluster.md#custom-namespace). | | `KUBECONFIG` | The kubeconfig to use for deployments. User-provided values take priority over GitLab-provided values. | | `PRODUCTION_REPLICAS` | Number of replicas to deploy in the production environment. Takes precedence over `REPLICAS` and defaults to 1. For zero downtime upgrades, set to 2 or greater. | -| `REPLICAS` | Number of replicas to deploy. Defaults to 1. | +| `REPLICAS` | Number of replicas to deploy. Defaults to 1. Change this variable instead of [modifying](#customize-values-for-helm-chart) `replicaCount`. | | `ROLLOUT_RESOURCE_TYPE` | Allows specification of the resource type being deployed when using a custom Helm chart. Default value is `deployment`. | | `ROLLOUT_STATUS_DISABLED` | From GitLab 12.0, used to disable rollout status check because it does not support all resource types, for example, `cronjob`. | | `STAGING_ENABLED` | Used to define a [deploy policy for staging and production environments](#deploy-policy-for-staging-and-production-environments). | @@ -478,6 +481,7 @@ The following table lists variables used to disable jobs. | `brakeman-sast` | `SAST_DISABLED` | | If the variable is present, the job isn't created. | | `bundler-audit-dependency_scanning` | `DEPENDENCY_SCANNING_DISABLED` | | If the variable is present, the job isn't created. | | `canary` | `CANARY_ENABLED` | | This manual job is created if the variable is present. | +| `cluster_image_scanning` | `CLUSTER_IMAGE_SCANNING_DISABLED` | | If the variable is present, the job isn't created. | | `code_intelligence` | `CODE_INTELLIGENCE_DISABLED` | From GitLab 13.6 | If the variable is present, the job isn't created. | | `code_quality` | `CODE_QUALITY_DISABLED` | | If the variable is present, the job isn't created. | | `container_scanning` | `CONTAINER_SCANNING_DISABLED` | | If the variable is present, the job isn't created. | diff --git a/doc/topics/cron/index.md b/doc/topics/cron/index.md index de83ec8b51b..affd746f66f 100644 --- a/doc/topics/cron/index.md +++ b/doc/topics/cron/index.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w Cron syntax is used to schedule when jobs should run. You may need to use a cron syntax string to -create a [pipeline schedule](../../api/pipeline_schedules.md#create-a-new-pipeline-schedule), +create a [pipeline schedule](../../ci/pipelines/schedules.md), or to prevent unintentional releases by setting a [deploy freeze](../../user/project/releases/index.md#prevent-unintentional-releases-by-setting-a-deploy-freeze). diff --git a/doc/topics/git/lfs/index.md b/doc/topics/git/lfs/index.md index a94caf2bf33..977f51a7211 100644 --- a/doc/topics/git/lfs/index.md +++ b/doc/topics/git/lfs/index.md @@ -31,7 +31,7 @@ Documentation for GitLab instance administrators is under [LFS administration do - Git LFS is supported in GitLab starting with version 8.2 - Git LFS must be enabled under project settings -- [Git LFS client](https://git-lfs.github.com) version 1.0.1 and up +- [Git LFS client](https://git-lfs.github.com) version 1.0.1 and up must be installed ## Known limitations diff --git a/doc/topics/git/lfs/migrate_to_git_lfs.md b/doc/topics/git/lfs/migrate_to_git_lfs.md index 2786368a9d7..32e3b6e2f72 100644 --- a/doc/topics/git/lfs/migrate_to_git_lfs.md +++ b/doc/topics/git/lfs/migrate_to_git_lfs.md @@ -46,7 +46,7 @@ Before beginning, make sure: To follow this tutorial, you need: -- The [Maintainer role](../../../user/permissions.md) for the existing Git repository +- The Maintainer role for the existing Git repository you'd like to migrate to LFS with access through the command line. - [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) and [Java Runtime Environment](https://www.java.com/en/download/manual.jsp) diff --git a/doc/topics/git/merge_conflicts.md b/doc/topics/git/merge_conflicts.md index bf69190030c..47276ccb0b2 100644 --- a/doc/topics/git/merge_conflicts.md +++ b/doc/topics/git/merge_conflicts.md @@ -23,7 +23,7 @@ comments: false 1. Fix conflicts on the `conflicts.rb` file. 1. Stage the file and continue rebasing. 1. Force push the changes. -1. Finally continue with the Merge Request. +1. Finally continue with the merge request. ```shell git checkout -b conflicts_branch diff --git a/doc/topics/gitlab_flow.md b/doc/topics/gitlab_flow.md index 7675bba8d83..3bca33b32b7 100644 --- a/doc/topics/gitlab_flow.md +++ b/doc/topics/gitlab_flow.md @@ -71,7 +71,7 @@ For example, many projects do releases but don't need to do hotfixes. ## GitHub flow as a simpler alternative In reaction to Git flow, GitHub created a simpler alternative. -[GitHub flow](https://guides.github.com/introduction/flow/index.html) has only feature branches and a `main` branch: +[GitHub flow](https://docs.github.com/en/get-started/quickstart/github-flow) has only feature branches and a `main` branch: ```mermaid graph TD @@ -221,7 +221,7 @@ If the assigned person does not feel comfortable, they can request more changes In GitLab, it is common to protect the long-lived branches, such as the `main` branch, so [most developers can't modify them](../user/permissions.md). So, if you want to merge into a protected branch, assign your merge request to someone with the -[Maintainer role](../user/permissions.md). +Maintainer role. After you merge a feature branch, you should remove it from the source control software. In GitLab, you can do this when merging. @@ -341,7 +341,7 @@ However, as discussed in [the section about rebasing](#squashing-commits-with-re Rebasing could create more work, as every time you rebase, you may need to resolve the same conflicts. Sometimes you can reuse recorded resolutions (`rerere`), but merging is better, because you only have to resolve conflicts once. -Atlassian has a more thorough explanation of the tradeoffs between merging and rebasing [on their blog](https://www.atlassian.com/blog/git/git-team-workflows-merge-or-rebase). +Atlassian has [a more thorough explanation of the tradeoffs between merging and rebasing](https://www.atlassian.com/blog/git/git-team-workflows-merge-or-rebase) on their blog. A good way to prevent creating many merge commits is to not frequently merge `main` into the feature branch. There are three reasons to merge in `main`: utilizing new code, resolving merge conflicts, and updating long-running branches. @@ -403,7 +403,7 @@ git commit -m 'Properly escape special characters in XML generation' An example of a good commit message is: "Combine templates to reduce duplicate code in the user views." The words "change," "improve," "fix," and "refactor" don't add much information to a commit message. -For more information about formatting commit messages, please see this excellent [blog post by Tim Pope](https://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html). +For more information, please see Tim Pope's excellent [note about formatting commit messages](https://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html). To add more context to a commit message, consider adding information regarding the origin of the change. For example, the URL of a GitLab issue, or a Jira issue number, diff --git a/doc/topics/release_your_application.md b/doc/topics/release_your_application.md index 3a0080fe21c..64decba33ad 100644 --- a/doc/topics/release_your_application.md +++ b/doc/topics/release_your_application.md @@ -30,8 +30,8 @@ to Kubernetes clusters using the [GitLab Agent](../user/clusters/agent/install/i #### GitOps deployments **(PREMIUM)** -With the [GitLab Agent](../user/clusters/agent/install/index.md), you can perform pull-based -deployments using Kubernetes manifests. This provides a scalable, secure, and cloud-native +With the [GitLab Agent](../user/clusters/agent/install/index.md), you can perform [pull-based +deployments of Kubernetes manifests](../user/clusters/agent/repository.md#synchronize-manifest-projects). This provides a scalable, secure, and cloud-native approach to manage Kubernetes deployments. #### Deploy to Kubernetes with the CI/CD Tunnel @@ -63,4 +63,4 @@ Use GitLab [Releases](../user/project/releases/index.md) to plan, build, and del ### Feature flags -Use [feature flags](../operations/feature_flags.md) to control and strategically roullout application deployments. +Use [feature flags](../operations/feature_flags.md) to control and strategically rollout application deployments. diff --git a/doc/tutorials/index.md b/doc/tutorials/index.md index 1f866aeb889..ac05b9945db 100644 --- a/doc/tutorials/index.md +++ b/doc/tutorials/index.md @@ -4,7 +4,7 @@ group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Get started with GitLab tutorials +# Learn GitLab with tutorials These tutorials can help you learn how to use GitLab. diff --git a/doc/update/deprecations.md b/doc/update/deprecations.md index 7388f4baf83..2f5146b8161 100644 --- a/doc/update/deprecations.md +++ b/doc/update/deprecations.md @@ -38,7 +38,7 @@ For deprecation reviewers (Technical Writers only): ## 14.0 -### NFS for Git repository storage deprecated +### NFS for Git repository storage WARNING: This feature will be changed or removed in 15.0 @@ -58,6 +58,18 @@ We encourage customers currently using NFS for Git repositories to plan their mi **Planned removal milestone: 15.0 (2022-05-22)** +### OAuth implicit grant + +WARNING: +This feature will be changed or removed in 15.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +The OAuth implicit grant authorization flow will be removed in our next major release, GitLab 15.0. Any applications that use OAuth implicit grant should switch to alternative [supported OAuth flows](https://docs.gitlab.com/ee/api/oauth2.html). + +**Planned removal milestone: 15.0 (2022-05-22)** + ## 14.2 ### Release CLI distributed as a generic package @@ -84,7 +96,7 @@ as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#brea Before updating GitLab, review the details carefully to determine if you need to make any changes to your code, settings, or workflow. -Audit events for [repository events](https://docs.gitlab.com/ee/administration/audit_events.html#repository-push) are now deprecated and will be removed in GitLab 15.0. +Audit events for [repository events](https://docs.gitlab.com/ee/administration/audit_events.html#repository-push-deprecated) are now deprecated and will be removed in GitLab 15.0. These events have always been disabled by default and had to be manually enabled with a feature flag. Enabling them can cause too many events to be generated which can @@ -143,24 +155,28 @@ Note that we are not deprecating the Kerberos SPNEGO integration, only the old p ### Certificate-based integration with Kubernetes WARNING: -This feature will be changed or removed in 15.0 +This feature will be changed or removed in 15.6 as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). Before updating GitLab, review the details carefully to determine if you need to make any changes to your code, settings, or workflow. [We are deprecating the certificate-based integration with Kubernetes](https://about.gitlab.com/blog/2021/11/15/deprecating-the-cert-based-kubernetes-integration/). -The timeline of removal of the integration from the product is not yet planned and we will communicate -more details as they emerge. The certificate-based integration will continue to receive security and +The timeline of removal of the integration from the product is planned to happen in two steps, starting with milestone 15.0 and finishing in GitLab version 15.6. + +In 15.0, we plan to introduce a feature flag that will allow GitLab Self-Managed customers to keep the certificate-based integration enabled, it will be disabled by default. We plan to remove this feature flag together with the underlying code in GitLab version 15.6. +The certificate-based integration will continue to receive security and critical fixes, and features built on the integration will continue to work with supported Kubernetes -versions. We will provide migration plans in a future iteration. See [the list of features affected by this deprecation](https://docs.gitlab.com/ee/user/infrastructure/clusters/#deprecated-features). -For updates and details, follow this [epic](https://gitlab.com/groups/gitlab-org/configure/-/epics/8). +versions until the final removal in 15.6. For a more robust, secure, forthcoming, and reliable integration with Kubernetes, we recommend the use of the -[Kubernetes Agent](https://docs.gitlab.com/ee/user/clusters/agent/) to connect Kubernetes clusters with GitLab. +[Kubernetes Agent](https://docs.gitlab.com/ee/user/clusters/agent/) to connect Kubernetes clusters with GitLab. +We provide [migration plans](https://docs.gitlab.com/ee/user/infrastructure/clusters/migrate_to_gitlab_agent.html) in the documentation. -**Planned removal milestone: 15.0 (2022-05-22)** +For updates and details around this deprecation, follow this [epic](https://gitlab.com/groups/gitlab-org/configure/-/epics/8). + +**Planned removal milestone: 15.6 (2022-11-22)** -### Converting an instance (shared) runner to a project (specific) runner is deprecated +### Converting an instance (shared) runner to a project (specific) runner WARNING: This feature will be changed or removed in 15.0 @@ -172,7 +188,7 @@ In GitLab 15.0, we will remove the feature that enables you to convert an instan **Planned removal milestone: 15.0 (2022-05-22)** -### Deprecate `Versions` on base `PackageType` +### Known host required for GitLab Runner SSH executor WARNING: This feature will be changed or removed in 15.0 @@ -180,13 +196,13 @@ as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#brea Before updating GitLab, review the details carefully to determine if you need to make any changes to your code, settings, or workflow. -As part of the work to create a [Package Registry GraphQL API](https://gitlab.com/groups/gitlab-org/-/epics/6318), the Package group deprecated the `Version` type for the basic `PackageType` type and moved it to [`PackageDetailsType`](https://docs.gitlab.com/ee/api/graphql/reference/index.html#packagedetailstype). +In [GitLab 14.3](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/3074), we added a configuration setting in the GitLab Runner `config.toml` file. This setting, [`[runners.ssh.disable_strict_host_key_checking]`](https://docs.gitlab.com/runner/executors/ssh.html#security), controls whether or not to use strict host key checking with the SSH executor. -In milestone 15.0, we will completely remove `Version` from `PackageType`. +In GitLab 15.0 and later, the default value for this configuration option will change from `true` to `false`. This means that strict host key checking will be enforced when using the GitLab Runner SSH executor. **Planned removal milestone: 15.0 (2022-05-22)** -### Deprecate support for SLES 12 SP2 +### Must explicitly assign `AuthenticationType` for `[runners.cache.s3]` WARNING: This feature will be changed or removed in 15.0 @@ -194,11 +210,13 @@ as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#brea Before updating GitLab, review the details carefully to determine if you need to make any changes to your code, settings, or workflow. -Long term service and support (LTSS) for SUSE Linux Enterprise Server (SLES) 12 SP2 [ended on March 31, 2021](https://www.suse.com/lifecycle/). The CA certificates on SP2 include the expired DST root certificate, and it's not getting new CA certificate package updates. We have implemented some [workarounds](https://gitlab.com/gitlab-org/gitlab-omnibus-builder/-/merge_requests/191), but we will not be able to continue to keep the build running properly. +In GitLab 15.0 and later, to access the AWS S3 cache, you must specify the `AuthenticationType` for [`[runners.cache.s3]`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnerscaches3-section). The `AuthenticationType` must be `IAM` or `credentials`. + +Prior to 14.5, if you did not define the `AuthenticationType`, GitLab Runner chose a type for you. **Planned removal milestone: 15.0 (2022-05-22)** -### Known host required for GitLab Runner SSH executor +### Package pipelines in API payload is paginated WARNING: This feature will be changed or removed in 15.0 @@ -206,13 +224,13 @@ as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#brea Before updating GitLab, review the details carefully to determine if you need to make any changes to your code, settings, or workflow. -In [GitLab 14.3](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/3074), we added a configuration setting in the GitLab Runner `config.toml` file. This setting, [`[runners.ssh.disable_strict_host_key_checking]`](https://docs.gitlab.com/runner/executors/ssh.html#security), controls whether or not to use strict host key checking with the SSH executor. +A request to the API for `/api/v4/projects/:id/packages` returns a paginated result of packages. Each package lists all of its pipelines in this response. This is a performance concern, as it's possible for a package to have hundreds or thousands of associated pipelines. -In GitLab 15.0 and later, the default value for this configuration option will change from `true` to `false`. This means that strict host key checking will be enforced when using the GitLab Runner SSH executor. +In milestone 15.0, we will remove the `pipelines` attribute from the API response. **Planned removal milestone: 15.0 (2022-05-22)** -### Must explicitly assign `AuthenticationType` for `[runners.cache.s3]` +### REST and GraphQL API Runner status will not return `paused` WARNING: This feature will be changed or removed in 15.0 @@ -220,13 +238,17 @@ as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#brea Before updating GitLab, review the details carefully to determine if you need to make any changes to your code, settings, or workflow. -In GitLab 15.0 and later, to access the AWS S3 cache, you must specify the `AuthenticationType` for [`[runners.cache.s3]`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnerscaches3-section). The `AuthenticationType` must be `IAM` or `credentials`. +The GitLab Runner REST and GraphQL API endpoints will not return `paused` or `active` as a status in GitLab 15.0. -Prior to 14.5, if you did not define the `AuthenticationType`, GitLab Runner chose a type for you. +A runner's status will only relate to runner contact status, such as: +`online`, `offline`, or `not_connected`. Status `paused` or `active` will no longer appear. + +When checking if a runner is `paused`, API users are advised to check the boolean attribute +`paused` to be `true` instead. When checking if a runner is `active`, check if `paused` is `false`. **Planned removal milestone: 15.0 (2022-05-22)** -### Package pipelines in API payload is paginated +### Support for SLES 12 SP2 WARNING: This feature will be changed or removed in 15.0 @@ -234,13 +256,11 @@ as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#brea Before updating GitLab, review the details carefully to determine if you need to make any changes to your code, settings, or workflow. -A request to the API for `/api/v4/projects/:id/packages` returns a paginated result of packages. Each package lists all of its pipelines in this response. This is a performance concern, as it's possible for a package to have hundreds or thousands of associated pipelines. - -In milestone 15.0, we will remove the `pipelines` attribute from the API response. +Long term service and support (LTSS) for SUSE Linux Enterprise Server (SLES) 12 SP2 [ended on March 31, 2021](https://www.suse.com/lifecycle/). The CA certificates on SP2 include the expired DST root certificate, and it's not getting new CA certificate package updates. We have implemented some [workarounds](https://gitlab.com/gitlab-org/gitlab-omnibus-builder/-/merge_requests/191), but we will not be able to continue to keep the build running properly. **Planned removal milestone: 15.0 (2022-05-22)** -### REST API Runner will not contain `paused` +### Update to the Container Registry group-level API WARNING: This feature will be changed or removed in 15.0 @@ -248,17 +268,13 @@ as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#brea Before updating GitLab, review the details carefully to determine if you need to make any changes to your code, settings, or workflow. -The GitLab Runner REST and GraphQL API endpoints will not return `paused` or `active` as a status in GitLab 15.0. - -A runner's status will only relate to runner contact status, such as: -`online`, `offline`, or `not_connected`. Status `paused` or `active` will no longer appear. +In milestone 15.0, support for the `tags` and `tags_count` parameters will be removed from the Container Registry API that [gets registry repositories from a group](../api/container_registry.md#within-a-group). -When checking if a runner is `paused`, API users are advised to check the boolean attribute -`active` to be `false` instead. When checking if a runner is `active`, check if `active` is `true`. +The `GET /groups/:id/registry/repositories` endpoint will remain, but won't return any info about tags. To get the info about tags, you can use the existing `GET /registry/repositories/:id` endpoint, which will continue to support the `tags` and `tag_count` options as it does today. The latter must be called once per image repository. **Planned removal milestone: 15.0 (2022-05-22)** -### Removal of `defaultMergeCommitMessageWithDescription` GraphQL API field +### Value Stream Analytics filtering calculation change WARNING: This feature will be changed or removed in 15.0 @@ -266,11 +282,13 @@ as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#brea Before updating GitLab, review the details carefully to determine if you need to make any changes to your code, settings, or workflow. -The GraphQL API field `defaultMergeCommitMessageWithDescription` has been deprecated and will be removed in GitLab 15.0. For projects with a commit message template set, it will ignore the template. +We are changing how the date filter works in Value Stream Analytics. Instead of filtering by the time that the issue or merge request was created, the date filter will filter by the end event time of the given stage. This will result in completely different figures after this change has rolled out. + +If you monitor Value Stream Analytics metrics and rely on the date filter, to avoid losing data, you must save the data prior to this change. **Planned removal milestone: 15.0 (2022-05-22)** -### Removal of `promote-db` command from `gitlab-ctl` +### `Versions` on base `PackageType` WARNING: This feature will be changed or removed in 15.0 @@ -278,11 +296,13 @@ as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#brea Before updating GitLab, review the details carefully to determine if you need to make any changes to your code, settings, or workflow. -In GitLab 14.5, we introduced the command `gitlab-ctl promote` to promote any Geo secondary node to a primary during a failover. This command replaces `gitlab-ctl promote-db` which is used to promote database nodes in multi-node Geo secondary sites. `gitlab-ctl promote-db` will continue to function as-is and be available until GitLab 15.0. We recommend that Geo customers begin testing the new `gitlab-ctl promote` command in their staging environments and incorporating the new command in their failover procedures. +As part of the work to create a [Package Registry GraphQL API](https://gitlab.com/groups/gitlab-org/-/epics/6318), the Package group deprecated the `Version` type for the basic `PackageType` type and moved it to [`PackageDetailsType`](https://docs.gitlab.com/ee/api/graphql/reference/index.html#packagedetailstype). + +In milestone 15.0, we will completely remove `Version` from `PackageType`. **Planned removal milestone: 15.0 (2022-05-22)** -### Removal of `promote-to-primary-node` command from `gitlab-ctl` +### `defaultMergeCommitMessageWithDescription` GraphQL API field WARNING: This feature will be changed or removed in 15.0 @@ -290,11 +310,11 @@ as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#brea Before updating GitLab, review the details carefully to determine if you need to make any changes to your code, settings, or workflow. -In GitLab 14.5, we introduced the command `gitlab-ctl promote` to promote any Geo secondary node to a primary during a failover. This command replaces `gitlab-ctl promote-to-primary-node` which was only usable for single-node Geo sites. `gitlab-ctl promote-to-primary-node` will continue to function as-is and be available until GitLab 15.0. We recommend that Geo customers begin testing the new `gitlab-ctl promote` command in their staging environments and incorporating the new command in their failover procedures. +The GraphQL API field `defaultMergeCommitMessageWithDescription` has been deprecated and will be removed in GitLab 15.0. For projects with a commit message template set, it will ignore the template. **Planned removal milestone: 15.0 (2022-05-22)** -### Remove the `:dependency_proxy_for_private_groups` feature flag +### `dependency_proxy_for_private_groups` feature flag WARNING: This feature will be changed or removed in 15.0 @@ -308,7 +328,7 @@ In milestone 15.0, we will remove the feature flag entirely. Moving forward, you **Planned removal milestone: 15.0 (2022-05-22)** -### Remove the `pipelines` field from the `version` field +### `pipelines` field from the `version` field WARNING: This feature will be changed or removed in 15.0 @@ -325,7 +345,7 @@ To mitigate possible performance problems, we will remove the `versions` field's **Planned removal milestone: 15.0 (2022-05-22)** -### Update to the Container Registry group-level API +### `promote-db` command from `gitlab-ctl` WARNING: This feature will be changed or removed in 15.0 @@ -333,13 +353,11 @@ as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#brea Before updating GitLab, review the details carefully to determine if you need to make any changes to your code, settings, or workflow. -In milestone 15.0, support for the `tags` and `tags_count` parameters will be removed from the Container Registry API that [gets registry repositories from a group](../api/container_registry.md#within-a-group). - -The `GET /groups/:id/registry/repositories` endpoint will remain, but won't return any info about tags. To get the info about tags, you can use the existing `GET /registry/repositories/:id` endpoint, which will continue to support the `tags` and `tag_count` options as it does today. The latter must be called once per image repository. +In GitLab 14.5, we introduced the command `gitlab-ctl promote` to promote any Geo secondary node to a primary during a failover. This command replaces `gitlab-ctl promote-db` which is used to promote database nodes in multi-node Geo secondary sites. `gitlab-ctl promote-db` will continue to function as-is and be available until GitLab 15.0. We recommend that Geo customers begin testing the new `gitlab-ctl promote` command in their staging environments and incorporating the new command in their failover procedures. **Planned removal milestone: 15.0 (2022-05-22)** -### Value Stream Analytics filtering calculation change +### `promote-to-primary-node` command from `gitlab-ctl` WARNING: This feature will be changed or removed in 15.0 @@ -347,9 +365,7 @@ as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#brea Before updating GitLab, review the details carefully to determine if you need to make any changes to your code, settings, or workflow. -We are changing how the date filter works in Value Stream Analytics. Instead of filtering by the time that the issue or merge request was created, the date filter will filter by the end event time of the given stage. This will result in completely different figures after this change has rolled out. - -If you monitor Value Stream Analytics metrics and rely on the date filter, to avoid losing data, you must save the data prior to this change. +In GitLab 14.5, we introduced the command `gitlab-ctl promote` to promote any Geo secondary node to a primary during a failover. This command replaces `gitlab-ctl promote-to-primary-node` which was only usable for single-node Geo sites. `gitlab-ctl promote-to-primary-node` will continue to function as-is and be available until GitLab 15.0. We recommend that Geo customers begin testing the new `gitlab-ctl promote` command in their staging environments and incorporating the new command in their failover procedures. **Planned removal milestone: 15.0 (2022-05-22)** @@ -389,21 +405,7 @@ In GitLab 15.0 we are going to limit the number of characters in CI/CD job names **Planned removal milestone: 15.0 (2022-05-22)** -### Deprecate `pipelines` fields in the Package GraphQL types - -WARNING: -This feature will be changed or removed in 15.0 -as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). -Before updating GitLab, review the details carefully to determine if you need to make any -changes to your code, settings, or workflow. - -As part of the work to create a [Package Registry GraphQL API](https://gitlab.com/groups/gitlab-org/-/epics/6318), the Package group deprecated the `pipelines` fields in all Package-related GraphQL types. As of GitLab 14.6, the `pipelines` field is deprecated in [`Package`](https://docs.gitlab.com/ee/api/graphql/reference/index.html#package) and [`PackageDetailsType`](https://docs.gitlab.com/ee/api/graphql/reference/index.html#packagedetailstype) due to scalability and performance concerns. - -In milestone 15.0, we will completely remove `pipelines` from `Package` and `PackageDetailsType`. You can follow and contribute to work on a replacement in the epic [GitLab-#7214](https://gitlab.com/groups/gitlab-org/-/epics/7214). - -**Planned removal milestone: 15.0 (2022-05-22)** - -### Deprecate legacy approval status names from License Compliance API +### Legacy approval status names from License Compliance API WARNING: This feature will be changed or removed in 15.0 @@ -417,7 +419,7 @@ If you are using our License Compliance API you should stop using the `approved` **Planned removal milestone: 15.0 (2022-05-22)** -### Deprecation of Runner status `not_connected` API value +### Runner status `not_connected` API value WARNING: This feature will be changed or removed in 15.0 @@ -432,7 +434,7 @@ Runners that have never contacted the GitLab instance will also return `stale` i **Planned removal milestone: 15.0 (2022-05-22)** -### Deprecation of bundler-audit Dependency Scanning tool +### `pipelines` fields in the Package GraphQL types WARNING: This feature will be changed or removed in 15.0 @@ -440,13 +442,13 @@ as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#brea Before updating GitLab, review the details carefully to determine if you need to make any changes to your code, settings, or workflow. -As of 14.6 bundler-audit is being deprecated from Dependency Scanning. It will continue to be in our CI/CD template while deprecated. We are removing bundler-audit from Dependency Scanning on May 22, 2022 in 15.0. After this removal Ruby scanning functionality will not be affected as it is still being covered by Gemnasium. +As part of the work to create a [Package Registry GraphQL API](https://gitlab.com/groups/gitlab-org/-/epics/6318), the Package group deprecated the `pipelines` fields in all Package-related GraphQL types. As of GitLab 14.6, the `pipelines` field is deprecated in [`Package`](https://docs.gitlab.com/ee/api/graphql/reference/index.html#package) and [`PackageDetailsType`](https://docs.gitlab.com/ee/api/graphql/reference/index.html#packagedetailstype) due to scalability and performance concerns. -If you have explicitly excluded bundler-audit using DS_EXCLUDED_ANALYZERS you will need to clean up (remove the reference) in 15.0. If you have customized your pipeline's Dependency Scanning configuration, for example to edit the `bundler-audit-dependency_scanning` job, you will want to switch to gemnasium-dependency_scanning before removal in 15.0, to prevent your pipeline from failing. If you have not used the DS_EXCLUDED_ANALYZERS to reference bundler-audit, or customized your template specifically for bundler-audit, you will not need to take action. +In milestone 15.0, we will completely remove `pipelines` from `Package` and `PackageDetailsType`. You can follow and contribute to work on a replacement in the epic [GitLab-#7214](https://gitlab.com/groups/gitlab-org/-/epics/7214). **Planned removal milestone: 15.0 (2022-05-22)** -### Remove `type` and `types` keyword in CI/CD configuration +### `type` and `types` keyword in CI/CD configuration WARNING: This feature will be changed or removed in 15.0 @@ -472,6 +474,20 @@ which isn't being used in GitLab anymore. **Planned removal milestone: 15.0 (2022-05-22)** +### bundler-audit Dependency Scanning tool + +WARNING: +This feature will be changed or removed in 15.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +As of 14.6 bundler-audit is being deprecated from Dependency Scanning. It will continue to be in our CI/CD template while deprecated. We are removing bundler-audit from Dependency Scanning on May 22, 2022 in 15.0. After this removal Ruby scanning functionality will not be affected as it is still being covered by Gemnasium. + +If you have explicitly excluded bundler-audit using DS_EXCLUDED_ANALYZERS you will need to clean up (remove the reference) in 15.0. If you have customized your pipeline's Dependency Scanning configuration, for example to edit the `bundler-audit-dependency_scanning` job, you will want to switch to gemnasium-dependency_scanning before removal in 15.0, to prevent your pipeline from failing. If you have not used the DS_EXCLUDED_ANALYZERS to reference bundler-audit, or customized your template specifically for bundler-audit, you will not need to take action. + +**Planned removal milestone: 15.0 (2022-05-22)** + ## 14.7 ### Container scanning schemas below 14.0.0 @@ -604,21 +620,6 @@ It is now considered deprecated, and will be removed in GitLab 15.0. **Planned removal milestone: 15.0 (2022-05-22)** -### Removal of Static Site Editor - -The Static Site Editor will no longer be available starting in GitLab 15.0. Improvements to the Markdown editing experience across GitLab will deliver smiliar benefit but with a wider reach. Incoming requests to the Static Site Editor will be redirected to the Web IDE. Current users of the Static Site Editor can view the [documentation](https://docs.gitlab.com/ee/user/project/static_site_editor/) for more information, including how to remove the configuration files from existing projects. - -**Planned removal milestone: 15.0 (2022-05-22)** - -### Removal of `artifacts:report:cobertura` keyword - -Currently, test coverage visualizations in GitLab only support Cobertura reports. Starting 15.0, the -`artifacts:report:cobertura` keyword will be replaced by -[`artifacts:reports:coverage_report`](https://gitlab.com/gitlab-org/gitlab/-/issues/344533). Cobertura will be the -only supported report file in 15.0, but this is the first step towards GitLab supporting other report types. - -**Planned removal milestone: 15.0 (2022-05-22)** - ### SAST schemas below 14.0.0 [SAST report schemas](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/releases) @@ -685,6 +686,12 @@ to serve the Sidekiq metrics, similar to the way Sidekiq will behave in 15.0. **Planned removal milestone: 15.0 (2022-05-22)** +### Static Site Editor + +The Static Site Editor will no longer be available starting in GitLab 15.0. Improvements to the Markdown editing experience across GitLab will deliver smiliar benefit but with a wider reach. Incoming requests to the Static Site Editor will be redirected to the Web IDE. Current users of the Static Site Editor can view the [documentation](https://docs.gitlab.com/ee/user/project/static_site_editor/) for more information, including how to remove the configuration files from existing projects. + +**Planned removal milestone: 15.0 (2022-05-22)** + ### Tracing in GitLab WARNING: @@ -697,14 +704,699 @@ Tracing in GitLab is an integration with Jaeger, an open-source end-to-end distr **Planned removal milestone: 15.0 (2022-05-22)** +### `artifacts:report:cobertura` keyword + +Currently, test coverage visualizations in GitLab only support Cobertura reports. Starting 15.0, the +`artifacts:report:cobertura` keyword will be replaced by +[`artifacts:reports:coverage_report`](https://gitlab.com/gitlab-org/gitlab/-/issues/344533). Cobertura will be the +only supported report file in 15.0, but this is the first step towards GitLab supporting other report types. + +**Planned removal milestone: 15.0 (2022-05-22)** + ### merged_by API field +WARNING: +This feature will be changed or removed in 15.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + The `merged_by` field in the [merge request API](https://docs.gitlab.com/ee/api/merge_requests.html#list-merge-requests) is being deprecated and will be removed in GitLab 15.0. This field is being replaced with the `merge_user` field (already present in GraphQL) which more correctly identifies who merged a merge request when performing actions (merge when pipeline succeeds, add to merge train) other than a simple merge. **Planned removal milestone: 15.0 (2022-05-22)** ## 14.8 +### Changes to the `CI_JOB_JWT` + +WARNING: +This feature will be changed or removed in 15.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +The `CI_JOB_JWT` will be updated to support a wider variety of cloud providers. It will be changed to match [`CI_JOB_JWT_V2`](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html), but this change may not be backwards compatible for all users, including Hashicorp Vault users. To maintain the current behavior, users can switch to using `CI_JOB_JWT_V1`, or update their configuration in GitLab 15.0 to use the improved `CI_JOB_JWT`. + +**Planned removal milestone: 15.0 ()** + +### Configurable Gitaly `per_repository` election strategy + +Configuring the `per_repository` Gitaly election strategy is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/352612). +`per_repository` has been the only option since GitLab 14.0. + +This change is part of regular maintenance to keep our codebase clean. + +**Planned removal milestone: 14.9 (2022-03-22)** + +### Container Network and Host Security + +WARNING: +This feature will be changed or removed in 15.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +All functionality related to GitLab's Container Network Security and Container Host Security categories is deprecated in GitLab 14.8 and scheduled for removal in GitLab 15.0. Users who need a replacement for this functionality are encouraged to evaluate the following open source projects as potential solutions that can be installed and managed outside of GitLab: [AppArmor](https://gitlab.com/apparmor/apparmor), [Cilium](https://github.com/cilium/cilium), [Falco](https://github.com/falcosecurity/falco), [FluentD](https://github.com/fluent/fluentd), [Pod Security Admission](https://kubernetes.io/docs/concepts/security/pod-security-admission/). To integrate these technologies into GitLab, add the desired Helm charts into your copy of the [Cluster Management Project Template](https://docs.gitlab.com/ee/user/clusters/management_project_template.html). Deploy these Helm charts in production by calling commands through the GitLab [Secure CI/CD Tunnel](https://docs.gitlab.com/ee/user/clusters/agent/repository.html#run-kubectl-commands-using-the-cicd-tunnel). + +As part of this change, the following specific capabilities within GitLab are now deprecated, and are scheduled for removal in GitLab 15.0: + +- The **Security & Compliance > Threat Monitoring** page. +- The `Network Policy` security policy type, as found on the **Security & Compliance > Policies** page. +- The ability to manage integrations with the following technologies through GitLab: AppArmor, Cilium, Falco, FluentD, and Pod Security Policies. +- All APIs related to the above functionality. + +For additional context, or to provide feedback regarding this change, please reference our open [deprecation issue](https://gitlab.com/groups/gitlab-org/-/epics/7476). + +**Planned removal milestone: 15.0 (2022-05-22)** + +### Dependency Scanning Python 3.9 and 3.6 image deprecation + +WARNING: +This feature will be changed or removed in 15.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +For those using Dependency Scanning for Python projects, we are deprecating the default `gemnasium-python:2` image which uses Python 3.6 as well as the custom `gemnasium-python:2-python-3.9` image which uses Python 3.9. The new default image as of GitLab 15.0 will be for Python 3.9 as it is a [supported version](https://endoflife.date/python) and 3.6 [is no longer supported](https://endoflife.date/python). + +For users using Python 3.9 or 3.9-compatible projects, you should not need to take action and dependency scanning should begin to work in GitLab 15.0. If you wish to test the new container now please run a test pipeline in your project with this container (which will be removed in 15.0). Use the Python 3.9 image: + +```yaml +gemnasium-python-dependency_scanning: + image: + name: registry.gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python:2-python-3.9 +``` + +For users using Python 3.6, as of GitLab 15.0 you will no longer be able to use the default template for dependency scanning. You will need to switch to use the deprecated `gemnasium-python:2` analyzer image. If you are impacted by this please comment in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/351503) so we can extend the removal if needed. + +For users using the 3.9 special exception image, you must instead use the default value and no longer override your container. To verify if you are using the 3.9 special exception image, check your `.gitlab-ci.yml` file for the following reference: + +```yaml +gemnasium-python-dependency_scanning: + image: + name: registry.gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python:2-python-3.9 +``` + +**Planned removal milestone: 15.0 (2021-05-22)** + +### Deprecate Geo Admin UI Routes + +In GitLab 13.0, we introduced new project and design replication details routes in the Geo Admin UI. These routes are `/admin/geo/replication/projects` and `/admin/geo/replication/designs`. We kept the legacy routes and redirected them to the new routes. In GitLab 15.0, we will remove support for the legacy routes `/admin/geo/projects` and `/admin/geo/designs`. Please update any bookmarks or scripts that may use the legacy routes. + +**Planned removal milestone: 15.0 (2022-05-22)** + +### Deprecate custom Geo:db:* Rake tasks + +In GitLab 14.8, we are [replacing the `geo:db:*` Rake tasks with built-in tasks](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/77269/diffs) that are now possible after [switching the Geo tracking database to use Rails' 6 support of multiple databases](https://gitlab.com/groups/gitlab-org/-/epics/6458). +The following `geo:db:*` tasks will be replaced with their corresponding `db:*:geo` tasks: + +- `geo:db:drop` -> `db:drop:geo` +- `geo:db:create` -> `db:create:geo` +- `geo:db:setup` -> `db:setup:geo` +- `geo:db:migrate` -> `db:migrate:geo` +- `geo:db:rollback` -> `db:rollback:geo` +- `geo:db:version` -> `db:version:geo` +- `geo:db:reset` -> `db:reset:geo` +- `geo:db:seed` -> `db:seed:geo` +- `geo:schema:load:geo` -> `db:schema:load:geo` +- `geo:db:schema:dump` -> `db:schema:dump:geo` +- `geo:db:migrate:up` -> `db:migrate:up:geo` +- `geo:db:migrate:down` -> `db:migrate:down:geo` +- `geo:db:migrate:redo` -> `db:migrate:redo:geo` +- `geo:db:migrate:status` -> `db:migrate:status:geo` +- `geo:db:test:prepare` -> `db:test:prepare:geo` +- `geo:db:test:load` -> `db:test:load:geo` +- `geo:db:test:purge` -> `db:test:purge:geo` + +**Planned removal milestone: 15.0 (2022-05-22)** + +### Deprecate feature flag PUSH_RULES_SUPERSEDE_CODE_OWNERS + +WARNING: +This feature will be changed or removed in 15.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +The feature flag `PUSH_RULES_SUPERSEDE_CODE_OWNERS` is being removed in GitLab 15.0. Upon its removal, push rules will supersede CODEOWNERS. The CODEOWNERS feature will no longer be available for access control. + +**Planned removal milestone: 15.0 (2022-05-22)** + +### Deprecate legacy Gitaly configuration methods + +WARNING: +This feature will be changed or removed in 15.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +Using environment variables `GIT_CONFIG_SYSTEM` and `GIT_CONFIG_GLOBAL` to configure Gitaly is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/352609). +These variables are being replaced with standard [`config.toml` Gitaly configuration](https://docs.gitlab.com/ee/administration/gitaly/reference.html). + +GitLab instances that use `GIT_CONFIG_SYSTEM` and `GIT_CONFIG_GLOBAL` to configure Gitaly should switch to configuring using +`config.toml`. + +**Planned removal milestone: 15.0 (2022-05-22)** + +### Elasticsearch 6.8 + +WARNING: +This feature will be changed or removed in 15.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +Elasticsearch 6.8 is deprecated in GitLab 14.8 and scheduled for removal in GitLab 15.0. +Customers using Elasticsearch 6.8 need to upgrade their Elasticsearch version to 7.x prior to upgrading to GitLab 15.0. +We recommend using the latest version of Elasticsearch 7 to benefit from all Elasticsearch improvements. + +Elasticsearch 6.8 is also incompatible with Amazon OpenSearch, which we [plan to support in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/327560). + +**Planned removal milestone: 15.0 (2022-05-22)** + +### External status check API breaking changes + +WARNING: +This feature will be changed or removed in 15.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +The [external status check API](https://docs.gitlab.com/ee/api/status_checks.html) was originally implemented to +support pass-by-default requests to mark a status check as passing. Pass-by-default requests are now deprecated. +Specifically, the following are deprecated: + +- Requests that do not contain the `status` field. +- Requests that have the `status` field set to `approved`. + +Beginning in GitLab 15.0, status checks will only be updated to a passing state if the `status` field is both present +and set to `pass`. Requests that: + +- Do not contain the `status` field will be rejected with a `422` error. For more information, see [the relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/338827). +- Contain any value other than `pass` will cause the status check to fail. For more information, see [the relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/339039). + +To align with this change, API calls to list external status checks will also return the value of `pass` rather than +`approved` for status checks that have passed. + +**Planned removal milestone: 15.0 (2022-05-22)** + +### GraphQL ID and GlobalID compatibility + +WARNING: +This feature will be changed or removed in 15.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +We are removing a non-standard extension to our GraphQL processor, which we added for backwards compatibility. This extension modifies the validation of GraphQL queries, allowing the use of the `ID` type for arguments where it would normally be rejected. +Some arguments originally had the type `ID`. These were changed to specific +kinds of `ID`. This change may be a breaking change if you: + +- Use GraphQL. +- Use the `ID` type for any argument in your query signatures. + +Some field arguments still have the `ID` type. These are typically for +IID values, or namespace paths. An example is `Query.project(fullPath: ID!)`. + +For a list of affected and unaffected field arguments, +see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/352832). + +You can test if this change affects you by validating +your queries locally, using schema data fetched from a GitLab server. +You can do this by using the GraphQL explorer tool for the relevant GitLab +instance. For example: `https://gitlab.com/-/graphql-explorer`. + +For example, the following query illustrates the breaking change: + +```graphql +# a query using the deprecated type of Query.issue(id:) +# WARNING: This will not work after GitLab 15.0 +query($id: ID!) { + deprecated: issue(id: $id) { + title, description + } +} +``` + +The query above will not work after GitLab 15.0 is released, because the type +of `Query.issue(id:)` is actually `IssueID!`. + +Instead, you should use one of the following two forms: + +```graphql +# This will continue to work +query($id: IssueID!) { + a: issue(id: $id) { + title, description + } + b: issue(id: "gid://gitlab/Issue/12345") { + title, description + } +} +``` + +This query works now, and will continue to work after GitLab 15.0. +You should convert any queries in the first form (using `ID` as a named type in the signature) +to one of the other two forms (using the correct appropriate type in the signature, or using +an inline argument expression). + +**Planned removal milestone: 15.0 (2022-04-22)** + +### OAuth tokens without expiration + +WARNING: +This feature will be changed or removed in 15.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +By default, all new applications expire access tokens after 2 hours. In GitLab 14.2 and earlier, OAuth access tokens +had no expiration. In GitLab 15.0, an expiry will be automatically generated for any existing token that does not +already have one. + +You should [opt in](https://docs.gitlab.com/ee/integration/oauth_provider.html#expiring-access-tokens) to expiring +tokens before GitLab 15.0 is released: + +1. Edit the application. +1. Select **Expire access tokens** to enable them. Tokens must be revoked or they don’t expire. + +**Planned removal milestone: 15.0 (2022-05-22)** + +### Optional enforcement of PAT expiration + +WARNING: +This feature will be changed or removed in 15.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +The feature to disable enforcement of PAT expiration is unusual from a security perspective. +We have become concerned that this unusual feature could create unexpected behavior for users. +Unexpected behavior in a security feature is inherently dangerous, so we have decided to remove this feature. + +**Planned removal milestone: 15.0 (2022-05-22)** + +### Optional enforcement of SSH expiration + +WARNING: +This feature will be changed or removed in 15.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +The feature to disable enforcement of SSH expiration is unusual from a security perspective. +We have become concerned that this unusual feature could create unexpected behavior for users. +Unexpected behavior in a security feature is inherently dangerous, so we have decided to remove this feature. + +**Planned removal milestone: 15.0 (2022-05-22)** + +### Out-of-the-box SAST support for Java 8 + +WARNING: +This feature will be changed or removed in 15.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +The [GitLab SAST SpotBugs analyzer](https://gitlab.com/gitlab-org/security-products/analyzers/spotbugs) scans [Java, Scala, Groovy, and Kotlin code](https://docs.gitlab.com/ee/user/application_security/sast/#supported-languages-and-frameworks) for security vulnerabilities. +For technical reasons, the analyzer must first compile the code before scanning. +Unless you use the [pre-compilation strategy](https://docs.gitlab.com/ee/user/application_security/sast/#pre-compilation), the analyzer attempts to automatically compile your project's code. + +In GitLab versions prior to 15.0, the analyzer image includes Java 8 and Java 11 runtimes to facilitate compilation. + +In GitLab 15.0, we will: + +- Remove Java 8 from the analyzer image to reduce the size of the image. +- Add Java 17 to the analyzer image to make it easier to compile with Java 17. + +If you rely on Java 8 being present in the analyzer environment, you must take action as detailed in the [deprecation issue for this change](https://gitlab.com/gitlab-org/gitlab/-/issues/352549#breaking-change). + +**Planned removal milestone: 15.0 (2022-05-22)** + +### Querying Usage Trends via the `instanceStatisticsMeasurements` GraphQL node + +WARNING: +This feature will be changed or removed in 15.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +The `instanceStatisticsMeasurements` GraphQL node has been renamed to `usageTrendsMeasurements` in 13.10 and the old field name has been marked as deprecated. To fix the existing GraphQL queries, replace `instanceStatisticsMeasurements` with `usageTrendsMeasurements`. + +**Planned removal milestone: 15.0 (2022-05-22)** + +### REST API Runner will not accept `status` filter values of `active` or `paused` + +WARNING: +This feature will be changed or removed in 16.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +The GitLab Runner REST endpoints will stop accepting `paused` or `active` as a status value in GitLab 16.0. + +A runner's status will only relate to runner contact status, such as: `online`, `offline`. +Status values `paused` or `active` will no longer be accepted and will be replaced by the `paused` query parameter. + +When checking for paused runners, API users are advised to specify `paused=true` as the query parameter. +When checking for active runners, specify `paused=false`. + +**Planned removal milestone: 16.0 (2023-04-22)** + +### REST API endpoint to list group runners no longer accepts `project_type` value for `type` argument + +WARNING: +This feature will be changed or removed in 16.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +The `GET /groups/:id/runners?type=project_type` endpoint will be removed in GitLab 16.0. The endpoint always returned an empty collection. + +**Planned removal milestone: 16.0 (2023-04-22)** + +### REST and GraphQL API Runner usage of `active` replaced by `paused` + +WARNING: +This feature will be changed or removed in 16.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +Occurrences of the `active` identifier in the GitLab Runner REST and GraphQL API endpoints will be +renamed to `paused` in GitLab 16.0, namely: + +- GraphQL API: + - the `CiRunner` property; + - the `RunnerUpdateInput` input type for the `runnerUpdate` mutation; + - the `runners` and `Group.runners` queries. +- REST API: + - endpoints taking or returning `active` properties, such as: + - `GET /runners` + - `GET /runners/all` + - `GET /runners/:id` / `PUT /runners/:id` + - `PUT --form "active=false" /runners/:runner_id` + - `GET /projects/:id/runners` / `POST /projects/:id/runners` + - `GET /groups/:id/runners` + +The 16.0 release of the GitLab Runner will start using the `paused` property when registering runners, and therefore +will only be compatible with GitLab 16.0 and later. Until 16.0, GitLab will accept the deprecated `active` flag from +existing runners. + +**Planned removal milestone: 16.0 (2023-04-22)** + +### Reminder: support for NFS repository storage + +WARNING: +This feature will be changed or removed in 15.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +As [announced](https://about.gitlab.com/releases/2021/06/22/gitlab-14-0-released/#nfs-for-git-repository-storage-deprecated) at the +release of GitLab 14.0, technical support for NFS storage for Git repositories is being removed. Please see our official +[Statement of Support](https://about.gitlab.com/support/statement-of-support.html#gitaly-and-nfs) for additional information. + +We encourage customers currently using NFS for Git repositories to plan their migration by reviewing our documentation on +[migrating to Gitaly Cluster](https://docs.gitlab.com/ee/administration/gitaly/#migrating-to-gitaly-cluster). + +Gitaly Cluster offers tremendous benefits for our customers such as: + +- [Variable replication factors](https://docs.gitlab.com/ee/administration/gitaly/praefect.html#configure-replication-factor) +- [Strong consistency](https://docs.gitlab.com/ee/administration/gitaly/#strong-consistency) +- [Distributed read capabilities](https://docs.gitlab.com/ee/administration/gitaly/#distributed-reads) + +**Planned removal milestone: 15.0 (2022-05-22)** + +### Request profiling + +WARNING: +This feature will be changed or removed in 15.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +[Request profiling](https://docs.gitlab.com/ee/administration/monitoring/performance/request_profiling.html) is deprecated in GitLab 14.8 and scheduled for removal in GitLab 15.0. + +We're working on [consolidating our profiling tools](https://gitlab.com/groups/gitlab-org/-/epics/7327) and making them more easily accessible. +We [evaluated](https://gitlab.com/gitlab-org/gitlab/-/issues/350152) the use of this feature and we found that it is not widely used. +It also depends on a few third-party gems that are not actively maintained anymore, have not been updated for the latest version of Ruby, or crash frequently when profiling heavy page loads. + +For more information, check the [summary section of the deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/352488#deprecation-summary). + +**Planned removal milestone: 15.0 (2022-05-22)** + +### Required pipeline configurations in Premium tier + +WARNING: +This feature will be changed or removed in 15.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +The [required pipeline configuration](https://docs.gitlab.com/ee/user/admin_area/settings/continuous_integration.html#required-pipeline-configuration) feature is deprecated in GitLab 14.8 for Premium customers and is scheduled for removal in GitLab 15.0. This feature is not deprecated for GitLab Ultimate customers. + +This change to move the feature to GitLab's Ultimate tier is intended to help our features better align with our [pricing philosophy](https://about.gitlab.com/company/pricing/#three-tiers) as we see demand for this feature originating primarily from executives. + +This change will also help GitLab remain consistent in its tiering strategy with the other related Ultimate-tier features of: +[Security policies](https://docs.gitlab.com/ee/user/application_security/policies/) and [compliance framework pipelines](https://docs.gitlab.com/ee/user/project/settings/index.html#compliance-pipeline-configuration). + +**Planned removal milestone: 15.0 (2022-05-22)** + +### Retire-JS Dependency Scanning tool + +WARNING: +This feature will be changed or removed in 15.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +As of 14.8 the retire.js job is being deprecated from Dependency Scanning. It will continue to be included in our CI/CD template while deprecated. We are removing retire.js from Dependency Scanning on May 22, 2022 in GitLab 15.0. JavaScript scanning functionality will not be affected as it is still being covered by Gemnasium. + +If you have explicitly excluded retire.js using DS_EXCLUDED_ANALYZERS you will need to clean up (remove the reference) in 15.0. If you have customized your pipeline's Dependency Scanning configuration related to the `retire-js-dependency_scanning` job you will want to switch to gemnasium-dependency_scanning before the removal in 15.0, to prevent your pipeline from failing. If you have not used the DS_EXCLUDED_ANALYZERS to reference retire.js, or customized your template specifically for retire.js, you will not need to take action. + +**Planned removal milestone: 15.0 (2022-05-22)** + +### SAST analyzer consolidation and CI/CD template changes + +WARNING: +This feature will be changed or removed in 15.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +GitLab SAST uses various [analyzers](https://docs.gitlab.com/ee/user/application_security/sast/analyzers/) to scan code for vulnerabilities. + +We are reducing the number of analyzers used in GitLab SAST as part of our long-term strategy to deliver a better and more consistent user experience. +Streamlining the set of analyzers will also enable faster [iteration](https://about.gitlab.com/handbook/values/#iteration), better [results](https://about.gitlab.com/handbook/values/#results), and greater [efficiency](https://about.gitlab.com/handbook/values/#results) (including a reduction in CI runner usage in most cases). + +In GitLab 15.0, GitLab SAST will no longer use the following analyzers: + +- [ESLint](https://gitlab.com/gitlab-org/security-products/analyzers/eslint) (JavaScript, TypeScript, React) +- [Gosec](https://gitlab.com/gitlab-org/security-products/analyzers/gosec) (Go) +- [Bandit](https://gitlab.com/gitlab-org/security-products/analyzers/bandit) (Python) + +These analyzers will be removed from the [GitLab-managed SAST CI/CD template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/SAST.gitlab-ci.yml) and replaced with the [Semgrep-based analyzer](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep). +They will no longer receive routine updates, except for security issues. +We will not delete container images previously published for these analyzers; any such change would be announced as a [deprecation, removal, or breaking change announcement](https://about.gitlab.com/handbook/marketing/blog/release-posts/#deprecations-removals-and-breaking-changes). + +We will also remove Java from the scope of the [SpotBugs](https://gitlab.com/gitlab-org/security-products/analyzers/spotbugs) analyzer and replace it with the [Semgrep-based analyzer](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep). +This change will make it simpler to scan Java code; compilation will no longer be required. +This change will be reflected in the automatic language detection portion of the [GitLab-managed SAST CI/CD template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/SAST.gitlab-ci.yml). + +If you applied customizations to any of the affected analyzers, you must take action as detailed in the [deprecation issue for this change](https://gitlab.com/gitlab-org/gitlab/-/issues/352554#breaking-change). + +**Planned removal milestone: 15.0 (2022-05-22)** + +### SAST support for .NET 2.1 + +WARNING: +This feature will be changed or removed in 15.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +The GitLab SAST Security Code Scan analyzer scans .NET code for security vulnerabilities. +For technical reasons, the analyzer must first build the code to scan it. + +In GitLab versions prior to 15.0, the default analyzer image (version 2) includes support for: + +- .NET 2.1 +- .NET 3.0 and .NET Core 3.0 +- .NET Core 3.1 +- .NET 5.0 + +In GitLab 15.0, we will change the default major version for this analyzer from version 2 to version 3. This change: + +- Adds [severity values for vulnerabilities](https://gitlab.com/gitlab-org/gitlab/-/issues/350408) along with [other new features and improvements](https://gitlab.com/gitlab-org/security-products/analyzers/security-code-scan/-/blob/master/CHANGELOG.md). +- Removes .NET 2.1 support. +- Adds support for .NET 6.0, Visual Studio 2019, and Visual Studio 2022. + +Version 3 was [announced in GitLab 14.6](https://about.gitlab.com/releases/2021/12/22/gitlab-14-6-released/#sast-support-for-net-6) and made available as an optional upgrade. + +If you rely on .NET 2.1 support being present in the analyzer image by default, you must take action as detailed in the [deprecation issue for this change](https://gitlab.com/gitlab-org/gitlab/-/issues/352553#breaking-change). + +**Planned removal milestone: 15.0 (2022-05-22)** + +### Secret Detection configuration variables deprecated + +To make it simpler and more reliable to [customize GitLab Secret Detection](https://docs.gitlab.com/ee/user/application_security/secret_detection/#customizing-settings), we're deprecating some of the variables that you could previously set in your CI/CD configuration. + +The following variables currently allow you to customize the options for historical scanning, but interact poorly with the [GitLab-managed CI/CD template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/Secret-Detection.gitlab-ci.yml) and are now deprecated: + +- `SECRET_DETECTION_COMMIT_FROM` +- `SECRET_DETECTION_COMMIT_TO` +- `SECRET_DETECTION_COMMITS` +- `SECRET_DETECTION_COMMITS_FILE` + +The `SECRET_DETECTION_ENTROPY_LEVEL` previously allowed you to configure rules that only considered the entropy level of strings in your codebase, and is now deprecated. +This type of entropy-only rule created an unacceptable number of incorrect results (false positives) and is no longer supported. + +In GitLab 15.0, we'll update the Secret Detection [analyzer](https://docs.gitlab.com/ee/user/application_security/terminology/#analyzer) to ignore these deprecated options. +You'll still be able to configure historical scanning of your commit history by setting the [`SECRET_DETECTION_HISTORIC_SCAN` CI/CD variable](https://docs.gitlab.com/ee/user/application_security/secret_detection/#available-cicd-variables). + +For further details, see [the deprecation issue for this change](https://gitlab.com/gitlab-org/gitlab/-/issues/352565). + +**Planned removal milestone: 15.0 (2022-05-22)** + +### Secure and Protect analyzer images published in new location + +WARNING: +This feature will be changed or removed in 15.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +GitLab uses various [analyzers](https://docs.gitlab.com/ee/user/application_security/terminology/#analyzer) to [scan for security vulnerabilities](https://docs.gitlab.com/ee/user/application_security/). +Each analyzer is distributed as a container image. + +Starting in GitLab 14.8, new versions of GitLab Secure and Protect analyzers are published to a new registry location under `registry.gitlab.com/security-products`. + +We will update the default value of [GitLab-managed CI/CD templates](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates/Security) to reflect this change: + +- For all analyzers except Container Scanning, we will update the variable `SECURE_ANALYZERS_PREFIX` to the new image registry location. +- For Container Scanning, the default image address is already updated. There is no `SECURE_ANALYZERS_PREFIX` variable for Container Scanning. + +In a future release, we will stop publishing images to `registry.gitlab.com/gitlab-org/security-products/analyzers`. +Once this happens, you must take action if you manually pull images and push them into a separate registry. This is commonly the case for [offline deployments](https://docs.gitlab.com/ee/user/application_security/offline_deployments/index.html). +Otherwise, you won't receive further updates. + +See the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/352564) for more details. + +**Planned removal milestone: 15.0 (2022-05-22)** + +### Secure and Protect analyzer major version update + +WARNING: +This feature will be changed or removed in 15.00 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +The Secure and Protect stages will be bumping the major versions of their analyzers in tandem with the GitLab 15.0 release. This major bump will enable a clear delineation for analyzers, between: + +- Those released prior to May 22, 2022, which generate reports that _are not_ subject to stringent schema validation. +- Those released after May 22, 2022, which generate reports that _are_ subject to stringent schema validation. + +If you are not using the default inclusion templates, or have pinned your analyzer version(s) you will need to update your CI/CD job definition to either remove the pinned version or to update the latest major version. +Users of GitLab 12.0-14.10 will continue to experience analyzer updates as normal until the release of GitLab 15.0, following which all newly fixed bugs and newly released features in the new major versions of the analyzers will not be available in the deprecated versions because we do not backport bugs and new features as per our [maintenance policy](https://docs.gitlab.com/ee/policy/maintenance.html). As required security patches will be backported within the latest 3 minor releases. +Specifically, the following are being deprecated and will no longer be updated after 15.0 GitLab release: + +- API Security: version 1 +- Container Scanning: version 4 +- Coverage-guided fuzz testing: version 2 +- Dependency Scanning: version 2 +- Dynamic Application Security Testing (DAST): version 2 +- License Scanning: version 3 +- Secret Detection: version 3 +- Static Application Security Testing (SAST): version 2, except security-code-scan which is version 3 + - `bandit`: version 2 + - `brakeman`: version 2 + - `eslint`: version 2 + - `flawfinder`: version 2 + - `gosec`: version 3 + - `kubesec`: version 2 + - `mobsf`: version 2 + - `nodejs-scan`: version 2 + - `phpcs-security-audit`: version 2 + - `pmd-apex`: version 2 + - `security-code-scan`: version 3 + - `semgrep`: version 2 + - `sobelow`: version 2 + - `spotbugs`: version 2 + +**Planned removal milestone: 15.00 ()** + +### Support for gRPC-aware proxy deployed between Gitaly and rest of GitLab + +WARNING: +This feature will be changed or removed in 15.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +Although not recommended or documented, it was possible to deploy a gRPC-aware proxy between Gitaly and +the rest of GitLab. For example, NGINX and Envoy. The ability to deploy a gRPC-aware proxy is +[deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/352517). If you currently use a gRPC-aware proxy for +Gitaly connections, you should change your proxy configuration to use TCP or TLS proxying (OSI layer 4) instead. + +Gitaly Cluster became incompatible with gRPC-aware proxies in GitLab 13.12. Now all GitLab installations will be incompatible with +gRPC-aware proxies, even without Gitaly Cluster. + +By sending some of our internal RPC traffic through a custom protocol (instead of gRPC) we +increase throughput and reduce Go garbage collection latency. For more information, see +the [relevant epic](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/463). + +**Planned removal milestone: 15.0 (2022-05-22)** + +### Test coverage project CI/CD setting + +WARNING: +This feature will be changed or removed in 15.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +To simplify setting a test coverage pattern, in GitLab 15.0 the +[project setting for test coverage parsing](https://docs.gitlab.com/ee/ci/pipelines/settings.html#add-test-coverage-results-to-a-merge-request-deprecated) +is being removed. + +Instead, using the project’s `.gitlab-ci.yml`, provide a regular expression with the `coverage` keyword to set +testing coverage results in merge requests. + +**Planned removal milestone: 15.0 (2022-05-22)** + +### Vulnerability Check + +WARNING: +This feature will be changed or removed in 15.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +The vulnerability check feature is deprecated in GitLab 14.8 and scheduled for removal in GitLab 15.0. We encourage you to migrate to the new security approvals feature instead. You can do so by navigating to **Security & Compliance > Policies** and creating a new Scan Result Policy. + +The new security approvals feature is similar to vulnerability check. For example, both can require approvals for MRs that contain security vulnerabilities. However, security approvals improve the previous experience in several ways: + +- Users can choose who is allowed to edit security approval rules. An independent security or compliance team can therefore manage rules in a way that prevents development project maintainers from modifying the rules. +- Multiple rules can be created and chained together to allow for filtering on different severity thresholds for each scanner type. +- A two-step approval process can be enforced for any desired changes to security approval rules. +- A single set of security policies can be applied to multiple development projects to allow for ease in maintaining a single, centralized ruleset. + +**Planned removal milestone: 15.0 (2022-05-22)** + +### `CI_BUILD_*` predefined variables + +WARNING: +This feature will be changed or removed in 15.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +The predefined CI/CD variables that start with `CI_BUILD_*` were deprecated in GitLab 9.0, and will be removed in GitLab 15.0. If you still use these variables, be sure to change to the current [`CI_JOB_*` predefined variables](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html) which are identical (except for the updated name). + +**Planned removal milestone: 15.0 (2022-05-22)** + ### `fixup!` commit messages setting draft status of associated Merge Request The use of `fixup!` as a commit message to trigger draft status @@ -716,3 +1408,30 @@ Support for `fixup!` is now considered deprecated, and will be removed in GitLab 15.0. **Planned removal milestone: 15.0 (2022-06-22)** + +### `projectFingerprint` in `PipelineSecurityReportFinding` GraphQL + +WARNING: +This feature will be changed or removed in 15.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +The `projectFingerprint` field in the [PipelineSecurityReportFinding](https://docs.gitlab.com/ee/api/graphql/reference/index.html#pipelinesecurityreportfinding) +GraphQL object is being deprecated. This field contains a "fingerprint" of security findings used to determine uniqueness. +The method for calculating fingerprints has changed, resulting in different values. Going forward, the new values will be +exposed in the UUID field. Data previously available in the projectFingerprint field will eventually be removed entirely. + +**Planned removal milestone: 15.0 (2022-05-22)** + +### `started` iterations API field + +WARNING: +This feature will be changed or removed in 15.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +The `started` field in the [iterations API](https://docs.gitlab.com/ee/api/iterations.html#list-project-iterations) is being deprecated and will be removed in GitLab 15.0. This field is being replaced with the `current` field (already available) which aligns with the naming for other time-based entities, such as milestones. + +**Planned removal milestone: 15.0 (2022-05-22)** diff --git a/doc/update/index.md b/doc/update/index.md index 3a17d3c01d7..45b5f36bc5a 100644 --- a/doc/update/index.md +++ b/doc/update/index.md @@ -98,7 +98,7 @@ that can process jobs in the `background_migration` queue. ```shell sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' -sudo gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigrationJob.pending' +sudo gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigrationJob.pending.count' ``` **For installations from source:** @@ -106,7 +106,7 @@ sudo gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigrati ```shell cd /home/git/gitlab sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' -sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::Database::BackgroundMigrationJob.pending' +sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::Database::BackgroundMigrationJob.pending.count' ``` ### Batched background migrations @@ -269,9 +269,9 @@ Additional steps between the mentioned versions are possible. We list the minima | Target version | Your version | Supported upgrade path | Note | | -------------- | ------------ | ---------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | -| `14.2.6` | `13.10.2` | `13.10.2` -> `13.12.12` -> `14.0.11` -> `14.1.8` -> `14.2.6` | Three intermediate versions are required: `13.12`, `14.0`, and `14.1`, then `14.2.6`. | -| `14.1.8` | `13.9.2` | `13.9.2` -> `13.12.12` -> `14.0.11` -> `14.1.8` | Two intermediate versions are required: `13.12` and `14.0`, then `14.1.8`. | -| `13.12.10` | `12.9.2` | `12.9.2` -> `12.10.14` -> `13.0.14` -> `13.1.11` -> `13.8.8` -> `13.12.10` | Four intermediate versions are required: `12.10`, `13.0`, `13.1` and `13.8.8`, then `13.12.10`. | +| `14.2.6` | `13.10.2` | `13.10.2` -> `13.12.15` -> `14.0.11` -> `14.1.8` -> `14.2.6` | Three intermediate versions are required: `13.12`, `14.0`, and `14.1`, then `14.2.6`. | +| `14.1.8` | `13.9.2` | `13.9.2` -> `13.12.15` -> `14.0.11` -> `14.1.8` | Two intermediate versions are required: `13.12` and `14.0`, then `14.1.8`. | +| `13.12.15` | `12.9.2` | `12.9.2` -> `12.10.14` -> `13.0.14` -> `13.1.11` -> `13.8.8` -> `13.12.15` | Four intermediate versions are required: `12.10`, `13.0`, `13.1` and `13.8.8`, then `13.12.15`. | | `13.2.10` | `11.5.0` | `11.5.0` -> `11.11.8` -> `12.0.12` -> `12.1.17` -> `12.10.14` -> `13.0.14` -> `13.1.11` -> `13.2.10` | Six intermediate versions are required: `11.11`, `12.0`, `12.1`, `12.10`, `13.0` and `13.1`, then `13.2.10`. | | `12.10.14` | `11.3.4` | `11.3.4` -> `11.11.8` -> `12.0.12` -> `12.1.17` -> `12.10.14` | Three intermediate versions are required: `11.11`, `12.0` and `12.1`, then `12.10.14`. | | `12.9.5` | `10.4.5` | `10.4.5` -> `10.8.7` -> `11.11.8` -> `12.0.12` -> `12.1.17` -> `12.9.5` | Four intermediate versions are required: `10.8`, `11.11`, `12.0` and `12.1`, then `12.9.5`. | @@ -351,7 +351,8 @@ or [init scripts](upgrading_from_source.md#configure-sysv-init-script) by [follo In 14.5 we introduce a set of migrations that wrap up this process by making sure that all remaining jobs over the `merge_request_diff_commits` table are completed. These jobs will have already been processed in most cases so that no extra time is necessary during an upgrade to 14.5. - But if there are remaining jobs, the deployment may take a few extra minutes to complete. + However, if there are remaining jobs or you haven't already upgraded to 14.1, + the deployment may take multiple hours to complete. All merge request diff commits will automatically incorporate these changes, and there are no additional requirements to perform the upgrade. @@ -439,6 +440,11 @@ for how to proceed. with self-managed installations, and ensure 14.0.0-14.0.4 installations do not encounter issues with [batched background migrations](#batched-background-migrations). +- Upgrading to GitLab [14.5](#1450) (or later) may take a lot longer if you do not upgrade to at least 14.1 + first. The 14.1 merge request diff commits database migration can take hours to run, but runs in the + background while GitLab is in use. GitLab instances upgraded directly from 14.0 to 14.5 or later must + run the migration in the foreground and therefore take a lot longer to complete. + - See [Maintenance mode issue in GitLab 13.9 to 14.4](#maintenance-mode-issue-in-gitlab-139-to-144). ### 14.0.0 @@ -676,7 +682,7 @@ When [Maintenance mode](../administration/maintenance_mode/index.md) is enabled, Users who were signed in before Maintenance mode was enabled will continue to be signed in. If the admin who enabled Maintenance mode loses their session, then they will not be able to disable Maintenance mode via the UI. In that case, you can [disable Maintenance mode via the API or Rails console](../administration/maintenance_mode/#disable-maintenance-mode). -[This bug](https://gitlab.com/gitlab-org/gitlab/-/issues/329261) was fixed in GitLab 14.5.0, and is expected to be backported to GitLab 14.3 and 14.4. +[This bug](https://gitlab.com/gitlab-org/gitlab/-/issues/329261) was fixed in GitLab 14.5.0 and backported into 14.4.3 and 14.3.5. ## Miscellaneous diff --git a/doc/update/removals.md b/doc/update/removals.md index 94ce815a110..9a28adf1f96 100644 --- a/doc/update/removals.md +++ b/doc/update/removals.md @@ -30,24 +30,70 @@ For removal reviewers (Technical Writers only): ## 14.0 +### Auto Deploy CI template v1 + +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +In GitLab 14.0, we will update the [Auto Deploy](https://docs.gitlab.com/ee/topics/autodevops/stages.html#auto-deploy) CI template to the latest version. This includes new features, bug fixes, and performance improvements with a dependency on the v2 [auto-deploy-image](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image). Auto Deploy CI template v1 is deprecated going forward. + +Since the v1 and v2 versions are not backward-compatible, your project might encounter an unexpected failure if you already have a deployed application. Follow the [upgrade guide](https://docs.gitlab.com/ee/topics/autodevops/upgrading_auto_deploy_dependencies.html#upgrade-guide) to upgrade your environments. You can also start using the latest template today by following the [early adoption guide](https://docs.gitlab.com/ee/topics/autodevops/upgrading_auto_deploy_dependencies.html#early-adopters). + ### Breaking changes to Terraform CI template +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + GitLab 14.0 renews the Terraform CI template to the latest version. The new template is set up for the GitLab Managed Terraform state, with a dependency on the GitLab `terraform-images` image, to provide a good user experience around GitLab's Infrastructure-as-Code features. The current stable and latest templates are not compatible, and the current latest template becomes the stable template beginning with GitLab 14.0, your Terraform pipeline might encounter an unexpected failure if you run a custom `init` job. ### Code Quality RuboCop support changed +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + By default, the Code Quality feature has not provided support for Ruby 2.6+ if you're using the Code Quality template. To better support the latest versions of Ruby, the default RuboCop version is updated to add support for Ruby 2.4 through 3.0. As a result, support for Ruby 2.1, 2.2, and 2.3 is removed. You can re-enable support for older versions by [customizing your configuration](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality.html#rubocop-errors). Relevant Issue: [Default `codeclimate-rubocop` engine does not support Ruby 2.6+](https://gitlab.com/gitlab-org/ci-cd/codequality/-/issues/28) ### Container Scanning Engine Clair +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + Clair, the default container scanning engine, was deprecated in GitLab 13.9 and is removed from GitLab 14.0 and replaced by Trivy. We advise customers who are customizing variables for their container scanning job to [follow these instructions](https://docs.gitlab.com/ee/user/application_security/container_scanning/#change-scanners) to ensure that their container scanning jobs continue to work. +### DAST default template stages + +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +In GitLab 14.0, we've removed the stages defined in the current `DAST.gitlab-ci.yml` template to avoid the situation where the template overrides manual changes made by DAST users. We're making this change in response to customer issues where the stages in the template cause problems when used with customized DAST configurations. Because of this removal, `gitlab-ci.yml` configurations that do not specify a `dast` stage must be updated to include this stage. + ### DAST environment variable renaming and removal +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + GitLab 13.8 renamed multiple environment variables to support their broader usage in different workflows. In GitLab 14.0, the old variables have been permanently removed and will no longer work. Any configurations using these variables must be updated to the new variable names. Any scans using these variables in GitLab 14.0 and later will fail to be configured correctly. These variables are: - `DAST_AUTH_EXCLUDE_URLS` becomes `DAST_EXCLUDE_URLS`. @@ -61,23 +107,61 @@ GitLab 13.8 renamed multiple environment variables to support their broader usag ### Default Browser Performance testing job renamed in GitLab 14.0 +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + Browser Performance Testing has run in a job named `performance` by default. With the introduction of [Load Performance Testing](https://docs.gitlab.com/ee/user/project/merge_requests/load_performance_testing.html) in GitLab 13.2, this naming could be confusing. To make it clear which job is running [Browser Performance Testing](https://docs.gitlab.com/ee/user/project/merge_requests/browser_performance_testing.html), the default job name is changed from `performance` to `browser_performance` in the template in GitLab 14.0. Relevant Issue: [Rename default Browser Performance Testing job](https://gitlab.com/gitlab-org/gitlab/-/issues/225914) ### Default DAST spider begins crawling at target URL +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + In GitLab 14.0, DAST has removed the current method of resetting the scan to the hostname when starting to spider. Prior to GitLab 14.0, the spider would not begin at the specified target path for the URL but would instead reset the URL to begin crawling at the host root. GitLab 14.0 changes the default for the new variable `DAST_SPIDER_START_AT_HOST` to `false` to better support users' intention of beginning spidering and scanning at the specified target URL, rather than the host root URL. This change has an added benefit: scans can take less time, if the specified path does not contain links to the entire site. This enables easier scanning of smaller sections of an application, rather than crawling the entire app during every scan. ### Default branch name for new repositories now `main` +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + Every Git repository has an initial branch, which is named `master` by default. It's the first branch to be created automatically when you create a new repository. Future [Git versions](https://lore.kernel.org/git/pull.656.v4.git.1593009996.gitgitgadget@gmail.com/) will change the default branch name in Git from `master` to `main`. In coordination with the Git project and the broader community, [GitLab has changed the default branch name](https://gitlab.com/gitlab-org/gitlab/-/issues/223789) for new projects on both our SaaS (GitLab.com) and self-managed offerings starting with GitLab 14.0. This will not affect existing projects. GitLab has already introduced changes that allow you to change the default branch name both at the [instance level](https://docs.gitlab.com/ee/user/project/repository/branches/default.html#instance-level-custom-initial-branch-name) (for self-managed users) and at the [group level](https://docs.gitlab.com/ee/user/group/#use-a-custom-name-for-the-initial-branch) (for both SaaS and self-managed users). We encourage you to make use of these features to set default branch names on new projects. For more information, check out our [blog post](https://about.gitlab.com/blog/2021/03/10/new-git-default-branch-name/). -### Deprecated GraphQL fields have been removed +### Dependency Scanning + +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +As mentioned in [13.9](https://about.gitlab.com/releases/2021/02/22/gitlab-13-9-released/#deprecations-for-dependency-scanning) and [this blog post](https://about.gitlab.com/blog/2021/02/08/composition-analysis-14-deprecations-and-removals/) several removals for Dependency Scanning take effect. + +Previously, to exclude a DS analyzer, you needed to remove it from the default list of analyzers, and use that to set the `DS_DEFAULT_ANALYZERS` variable in your project’s CI template. We determined it should be easier to avoid running a particular analyzer without losing the benefit of newly added analyzers. As a result, we ask you to migrate from `DS_DEFAULT_ANALYZERS` to `DS_EXCLUDED_ANALYZERS` when it is available. Read about it in [issue #287691](https://gitlab.com/gitlab-org/gitlab/-/issues/287691). + +Previously, to prevent the Gemnasium analyzers to fetch the advisory database at runtime, you needed to set the `GEMNASIUM_DB_UPDATE` variable. However, this is not documented properly, and its naming is inconsistent with the equivalent `BUNDLER_AUDIT_UPDATE_DISABLED` variable. As a result, we ask you to migrate from `GEMNASIUM_DB_UPDATE` to `GEMNASIUM_UPDATE_DISABLED` when it is available. Read about it in [issue #215483](https://gitlab.com/gitlab-org/gitlab/-/issues/215483). + +### Deprecated GraphQL fields + +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. In accordance with our [GraphQL deprecation and removal process](https://docs.gitlab.com/ee/api/graphql/#deprecation-process), the following fields that were deprecated prior to 13.7 are [fully removed in 14.0](https://gitlab.com/gitlab-org/gitlab/-/issues/267966): @@ -88,19 +172,55 @@ In accordance with our [GraphQL deprecation and removal process](https://docs.gi - `DeprecatedMutations (concern**)` - `AddAwardEmoji`, `RemoveAwardEmoji`, `ToggleAwardEmoji` - `EE::Types::DeprecatedMutations (concern***)` - `Mutations::Pipelines::RunDastScan`, `Mutations::Vulnerabilities::Dismiss`, `Mutations::Vulnerabilities::RevertToDetected` -### Deprecations for Dependency Scanning +### DevOps Adoption API Segments -As mentioned in [13.9](https://about.gitlab.com/releases/2021/02/22/gitlab-13-9-released/#deprecations-for-dependency-scanning) and [this blog post](https://about.gitlab.com/blog/2021/02/08/composition-analysis-14-deprecations-and-removals/) several removals for Dependency Scanning take effect. +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. -Previously, to exclude a DS analyzer, you needed to remove it from the default list of analyzers, and use that to set the `DS_DEFAULT_ANALYZERS` variable in your project’s CI template. We determined it should be easier to avoid running a particular analyzer without losing the benefit of newly added analyzers. As a result, we ask you to migrate from `DS_DEFAULT_ANALYZERS` to `DS_EXCLUDED_ANALYZERS` when it is available. Read about it in [issue #287691](https://gitlab.com/gitlab-org/gitlab/-/issues/287691). +The first release of the DevOps Adoption report had a concept of **Segments**. Segments were [quickly removed from the report](https://gitlab.com/groups/gitlab-org/-/epics/5251) because they introduced an additional layer of complexity on top of **Groups** and **Projects**. Subsequent iterations of the DevOps Adoption report focus on comparing adoption across groups rather than segments. GitLab 14.0 removes all references to **Segments** [from the GraphQL API](https://gitlab.com/gitlab-org/gitlab/-/issues/324414) and replaces them with **Enabled groups**. -Previously, to prevent the Gemnasium analyzers to fetch the advisory database at runtime, you needed to set the `GEMNASIUM_DB_UPDATE` variable. However, this is not documented properly, and its naming is inconsistent with the equivalent `BUNDLER_AUDIT_UPDATE_DISABLED` variable. As a result, we ask you to migrate from `GEMNASIUM_DB_UPDATE` to `GEMNASIUM_UPDATE_DISABLED` when it is available. Read about it in [issue #215483](https://gitlab.com/gitlab-org/gitlab/-/issues/215483). +### Disk source configuration for GitLab Pages + +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +GitLab Pages [API-based configuration](https://docs.gitlab.com/ee/administration/pages/#gitlab-api-based-configuration) has been available since GitLab 13.0. It replaces the unsupported `disk` source configuration removed in GitLab 14.0, which can no longer be chosen. You should stop using `disk` source configuration, and move to `gitlab` for an API-based configuration. To migrate away from the 'disk' source configuration, set `gitlab_pages['domain_config_source'] = "gitlab"` in your `/etc/gitlab/gitlab.rb` file. We recommend you migrate before updating to GitLab 14.0, to identify and troubleshoot any potential problems before upgrading. + +### Experimental prefix in Sidekiq queue selector options + +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +GitLab supports a [queue selector](https://docs.gitlab.com/ee/administration/operations/extra_sidekiq_processes.html#queue-selector) to run only a subset of background jobs for a given process. When it was introduced, this option had an 'experimental' prefix (`experimental_queue_selector` in Omnibus, `experimentalQueueSelector` in Helm charts). + +As announced in the [13.6 release post](https://about.gitlab.com/releases/2020/11/22/gitlab-13-6-released/#sidekiq-cluster-queue-selector-configuration-option-has-been-renamed), the 'experimental' prefix is no longer supported. Instead, `queue_selector` for Omnibus and `queueSelector` in Helm charts should be used. ### External Pipeline Validation Service Code Changes +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + For self-managed instances using the experimental [external pipeline validation service](https://docs.gitlab.com/ee/administration/external_pipeline_validation.html), the range of error codes GitLab accepts will be reduced. Currently, pipelines are invalidated when the validation service returns a response code from `400` to `499`. In GitLab 14.0 and later, pipelines will be invalidated for the `406: Not Accepted` response code only. -### Geo Foreign Data Wrapper settings removed +### Geo Foreign Data Wrapper settings + +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. As [announced in GitLab 13.3](https://about.gitlab.com/releases/2020/08/22/gitlab-13-3-released/#geo-foreign-data-wrapper-settings-deprecated), the following configuration settings in `/etc/gitlab/gitlab.rb` have been removed in 14.0: @@ -109,236 +229,448 @@ As [announced in GitLab 13.3](https://about.gitlab.com/releases/2020/08/22/gitla - `geo_postgresql['fdw_external_password']` - `gitlab-_rails['geo_migrated_local_files_clean_up_worker_cron']` -### GitLab OAuth implicit grant deprecation +### GitLab OAuth implicit grant + +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. GitLab is deprecating the [OAuth 2 implicit grant flow](https://docs.gitlab.com/ee/api/oauth2.html#implicit-grant-flow) as it has been removed for [OAuth 2.1](https://oauth.net/2.1/). -Beginning in 14.0, new applications can't be created with the OAuth 2 implicit grant flow. Existing OAuth implicit grant flows are no longer supported in 14.4. Migrate your existing applications to other supported [OAuth2 flows](https://docs.gitlab.com/ee/api/oauth2.html#supported-oauth2-flows) before release 14.4. +Migrate your existing applications to other supported [OAuth2 flows](https://docs.gitlab.com/ee/api/oauth2.html#supported-oauth2-flows). ### GitLab Runner helper image in GitLab.com Container Registry +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + In 14.0, we are now pulling the GitLab Runner [helper image](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#helper-image) from the GitLab Container Registry instead of Docker Hub. Refer to [issue #27218](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27218) for details. ### GitLab Runner installation to ignore the `skel` directory +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + In GitLab Runner 14.0, the installation process will ignore the `skel` directory by default when creating the user home directory. Refer to [issue #4845](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4845) for details. -### Gitaly Cluster SQL primary elector has been removed +### Gitaly Cluster SQL primary elector + +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. Now that Praefect supports a [primary election strategy](https://docs.gitlab.com/ee/administration/gitaly/praefect.html#repository-specific-primary-nodes) for each repository, we have removed the `sql` election strategy. The `per_repository` election strategy is the new default, which is automatically used if no election strategy was specified. If you had configured the `sql` election strategy, you must follow the [migration instructions](https://docs.gitlab.com/ee/administration/gitaly/praefect.html#migrate-to-repository-specific-primary-gitaly-nodes) before upgrading to 14.0. +### Global `SAST_ANALYZER_IMAGE_TAG` in SAST CI template + +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +With the maturity of GitLab Secure scanning tools, we've needed to add more granularity to our release process. Previously, GitLab shared a major version number for [all analyzers and tools](https://docs.gitlab.com/ee/user/application_security/sast/#supported-languages-and-frameworks). This requires all tools to share a major version, and prevents the use of [semantic version numbering](https://semver.org/). In GitLab 14.0, SAST removes the `SAST_ANALYZER_IMAGE_TAG` global variable in our [managed `SAST.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/SAST.gitlab-ci.yml) CI template, in favor of the analyzer job variable setting the `major.minor` tag in the SAST vendored template. + +Each analyzer job now has a scoped `SAST_ANALYZER_IMAGE_TAG` variable, which will be actively managed by GitLab and set to the `major` tag for the respective analyzer. To pin to a specific version, [change the variable value to the specific version tag](https://docs.gitlab.com/ee/user/application_security/sast/#pinning-to-minor-image-version). +If you override or maintain custom versions of `SAST.gitlab-ci.yml`, update your CI templates to stop referencing the global `SAST_ANALYZER_IMAGE_TAG`, and move it to a scoped analyzer job tag. We strongly encourage [inheriting and overriding our managed CI templates](https://docs.gitlab.com/ee/user/application_security/sast/#overriding-sast-jobs) to future-proof your CI templates. This change allows you to more granularly control future analyzer updates with a pinned `major.minor` version. +This deprecation and removal changes our [previously announced plan](https://about.gitlab.com/releases/2021/02/22/gitlab-13-9-released/#pin-static-analysis-analyzers-and-tools-to-minor-versions) to pin the Static Analysis tools. + +### Hardcoded `master` in CI/CD templates + +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +Our CI/CD templates have been updated to no longer use hard-coded references to a `master` branch. In 14.0, they all use a variable that points to your project's configured default branch instead. If your CI/CD pipeline relies on our built-in templates, verify that this change works with your current configuration. For example, if you have a `master` branch and a different default branch, the updates to the templates may cause changes to your pipeline behavior. For more information, [read the issue](https://gitlab.com/gitlab-org/gitlab/-/issues/324131). + ### Helm v2 support +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + Helm v2 was [officially deprecated](https://helm.sh/blog/helm-v2-deprecation-timeline/) in November of 2020, with the `stable` repository being [de-listed from the Helm Hub](https://about.gitlab.com/blog/2020/11/09/ensure-auto-devops-work-after-helm-stable-repo/) shortly thereafter. With the release of GitLab 14.0, which will include the 5.0 release of the [GitLab Helm chart](https://docs.gitlab.com/charts/), Helm v2 will no longer be supported. Users of the chart should [upgrade to Helm v3](https://helm.sh/docs/topics/v2_v3_migration/) to deploy GitLab 14.0 and later. -### Legacy feature flags removed +### Legacy DAST domain validation + +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +The legacy method of DAST Domain Validation for CI/CD scans was deprecated in GitLab 13.8, and is removed in GitLab 14.0. This method of domain validation only disallows scans if the `DAST_FULL_SCAN_DOMAIN_VALIDATION_REQUIRED` environment variable is set to `true` in the `gitlab-ci.yml` file, and a `Gitlab-DAST-Permission` header on the site is not set to `allow`. This two-step method required users to opt in to using the variable before they could opt out from using the header. + +For more information, see the [removal issue](https://gitlab.com/gitlab-org/gitlab/-/issues/293595). + +### Legacy feature flags + +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. Legacy feature flags became read-only in GitLab 13.4. GitLab 14.0 removes support for legacy feature flags, so you must migrate them to the [new version](https://docs.gitlab.com/ee/operations/feature_flags.html). You can do this by first taking a note (screenshot) of the legacy flag, then deleting the flag through the API or UI (you don't need to alter the code), and finally create a new Feature Flag with the same name as the legacy flag you deleted. Also, make sure the strategies and environments match the deleted flag. We created a [video tutorial](https://www.youtube.com/watch?v=CAJY2IGep7Y) to help with this migration. -### Legacy storage removed +### Legacy fields from DAST report + +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +As a part of the migration to a common report format for all of the Secure scanners in GitLab, DAST is making changes to the DAST JSON report. Certain legacy fields were deprecated in 13.8 and have been completely removed in 14.0. These fields are `@generated`, `@version`, `site`, and `spider`. This should not affect any normal DAST operation, but does affect users who consume the JSON report in an automated way and use these fields. Anyone affected by these changes, and needs these fields for business reasons, is encouraged to open a new GitLab issue and explain the need. + +For more information, see [the removal issue](https://gitlab.com/gitlab-org/gitlab/-/issues/33915). + +### Legacy storage + +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. As [announced in GitLab 13.0](https://about.gitlab.com/releases/2020/05/22/gitlab-13-0-released/#planned-removal-of-legacy-storage-in-14.0), [legacy storage](https://docs.gitlab.com/ee/administration/repository_storage_types.html#legacy-storage) has been removed in GitLab 14.0. +### License Compliance + +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +In 13.0, we deprecated the License-Management CI template and renamed it License-Scanning. We have been providing backward compatibility by warning users of the old template to switch. Now in 14.0, we are completely removing the License-Management CI template. Read about it in [issue #216261](https://gitlab.com/gitlab-org/gitlab/-/issues/216261) or [this blog post](https://about.gitlab.com/blog/2021/02/08/composition-analysis-14-deprecations-and-removals/). + ### Limit projects returned in `GET /groups/:id/` +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + To improve performance, we are limiting the number of projects returned from the `GET /groups/:id/` API call to 100. A complete list of projects can still be retrieved with the `GET /groups/:id/projects` API call. ### Make `pwsh` the default shell for newly-registered Windows Runners +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + In GitLab Runner 13.2, PowerShell Core support was added to the Shell executor. In 14.0, PowerShell Core, `pwsh` is now the default shell for newly-registered Windows runners. Windows `CMD` will still be available as a shell option for Windows runners. Refer to [issue #26419](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/26419) for details. ### Migrate from `SAST_DEFAULT_ANALYZERS` to `SAST_EXCLUDED_ANALYZERS` +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + Until GitLab 13.9, if you wanted to avoid running one particular GitLab SAST analyzer, you needed to remove it from the [long string of analyzers in the `SAST.gitlab-ci.yml` file](https://gitlab.com/gitlab-org/gitlab/-/blob/390afc431e7ce1ac253b35beb39f19e49c746bff/lib/gitlab/ci/templates/Security/SAST.gitlab-ci.yml#L12) and use that to set the [`SAST_DEFAULT_ANALYZERS`](https://docs.gitlab.com/ee/user/application_security/sast/#docker-images) variable in your project's CI file. If you did this, it would exclude you from future new analyzers because this string hard codes the list of analyzers to execute. We avoid this problem by inverting this variable's logic to exclude, rather than choose default analyzers. Beginning with 13.9, [we migrated](https://gitlab.com/gitlab-org/gitlab/-/blob/14fed7a33bfdbd4663d8928e46002a5ef3e3282c/lib/gitlab/ci/templates/Security/SAST.gitlab-ci.yml#L13) to `SAST_EXCLUDED_ANALYZERS` in our `SAST.gitlab-ci.yml` file. We encourage anyone who uses a [customized SAST configuration](https://docs.gitlab.com/ee/user/application_security/sast/#customizing-the-sast-settings) in their project CI file to migrate to this new variable. If you have not overridden `SAST_DEFAULT_ANALYZERS`, no action is needed. The CI/CD variable `SAST_DEFAULT_ANALYZERS` has been removed in GitLab 14.0, which released on June 22, 2021. -### New Terraform template version +### Off peak time mode configuration for Docker Machine autoscaling -As we continuously [develop GitLab's Terraform integrations](https://gitlab.com/gitlab-org/gitlab/-/issues/325312), to minimize customer disruption, we maintain two GitLab CI/CD templates for Terraform: - -- The ["latest version" template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Terraform.latest.gitlab-ci.yml), which is updated frequently between minor releases of GitLab (such as 13.10, 13.11, etc). -- The ["major version" template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Terraform.gitlab-ci.yml), which is updated only at major releases (such as 13.0, 14.0, etc). +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. -At every major release of GitLab, the "latest version" template becomes the "major version" template, inheriting the "latest template" setup. -As we have added many new features to the Terraform integration, the new setup for the "major version" template can be considered a breaking change. - -The latest template supports the [Terraform Merge Request widget](https://docs.gitlab.com/ee/user/infrastructure/mr_integration.html) and -doesn't need additional setup to work with the [GitLab managed Terraform state](https://docs.gitlab.com/ee/user/infrastructure/terraform_state.html). - -To check the new changes, see the [new "major version" template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Terraform.gitlab-ci.yml). +In GitLab Runner 13.0, [issue #5069](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/5069), we introduced new timing options for the GitLab Docker Machine executor. In GitLab Runner 14.0, we have removed the old configuration option, [off peak time mode](https://docs.gitlab.com/runner/configuration/autoscale.html#off-peak-time-mode-configuration-deprecated). ### OpenSUSE Leap 15.1 +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + Support for [OpenSUSE Leap 15.1 is being deprecated](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5135). Support for 15.1 will be dropped in 14.0. We are now providing support for openSUSE Leap 15.2 packages. ### PostgreSQL 11 support +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + PostgreSQL 12 will be the minimum required version in GitLab 14.0. It offers [significant improvements](https://www.postgresql.org/about/news/postgresql-12-released-1976/) to indexing, partitioning, and general performance benefits. Starting in GitLab 13.7, all new installations default to version 12. From GitLab 13.8, single-node instances are automatically upgraded as well. If you aren't ready to upgrade, you can [opt out of automatic upgrades](https://docs.gitlab.com/omnibus/settings/database.html#opt-out-of-automatic-postgresql-upgrades). -### Removal of deprecated `trace` parameter from `jobs` API endpoint +### Redundant timestamp field from DORA metrics API payload -GitLab Runner was updated in GitLab 13.4 to internally stop passing the `trace` parameter to the `/api/jobs/:id` endpoint. GitLab 14.0 deprecates the `trace` parameter entirely for all other requests of this endpoint. Make sure your [GitLab Runner version matches your GitLab version](https://docs.gitlab.com/runner/#gitlab-runner-versions) to ensure consistent behavior. +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. -### Removal of legacy fields from DAST report - -As a part of the migration to a common report format for all of the Secure scanners in GitLab, DAST is making changes to the DAST JSON report. Certain legacy fields were deprecated in 13.8 and have been completely removed in 14.0. These fields are `@generated`, `@version`, `site`, and `spider`. This should not affect any normal DAST operation, but does affect users who consume the JSON report in an automated way and use these fields. Anyone affected by these changes, and needs these fields for business reasons, is encouraged to open a new GitLab issue and explain the need. +The [deployment frequency project-level API](https://docs.gitlab.com/ee/api/dora4_project_analytics.html#list-project-deployment-frequencies) endpoint has been deprecated in favor of the [DORA 4 API](https://docs.gitlab.com/ee/api/dora/metrics.html), which consolidates all the metrics under one API with the specific metric as a required field. As a result, the timestamp field, which doesn't allow adding future extensions and causes performance issues, will be removed. With the old API, an example response would be `{ "2021-03-01": 3, "date": "2021-03-01", "value": 3 }`. The first key/value (`"2021-03-01": 3`) will be removed and replaced by the last two (`"date": "2021-03-01", "value": 3`). -For more information, see [the removal issue](https://gitlab.com/gitlab-org/gitlab/-/issues/33915). +### Release description in the Tags API -### Removal of release description in the Tags API +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. GitLab 14.0 removes support for the release description in the Tags API. You can no longer add a release description when [creating a new tag](https://docs.gitlab.com/ee/api/tags.html#create-a-new-tag). You also can no longer [create](https://docs.gitlab.com/ee/api/tags.html#create-a-new-release) or [update](https://docs.gitlab.com/ee/api/tags.html#update-a-release) a release through the Tags API. Please migrate to use the [Releases API](https://docs.gitlab.com/ee/api/releases/#create-a-release) instead. -### Removals for License Compliance +### Ruby version changed in `Ruby.gitlab-ci.yml` -In 13.0, we deprecated the License-Management CI template and renamed it License-Scanning. We have been providing backward compatibility by warning users of the old template to switch. Now in 14.0, we are completely removing the License-Management CI template. Read about it in [issue #216261](https://gitlab.com/gitlab-org/gitlab/-/issues/216261) or [this blog post](https://about.gitlab.com/blog/2021/02/08/composition-analysis-14-deprecations-and-removals/). +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. -### Remove DAST default template stages +By default, the `Ruby.gitlab-ci.yml` file has included Ruby 2.5. -In GitLab 14.0, we've removed the stages defined in the current `DAST.gitlab-ci.yml` template to avoid the situation where the template overrides manual changes made by DAST users. We're making this change in response to customer issues where the stages in the template cause problems when used with customized DAST configurations. Because of this removal, `gitlab-ci.yml` configurations that do not specify a `dast` stage must be updated to include this stage. +To better support the latest versions of Ruby, the template is changed to use `ruby:latest`, which is currently 3.0. To better understand the changes in Ruby 3.0, please reference the [Ruby-lang.org release announcement](https://www.ruby-lang.org/en/news/2020/12/25/ruby-3-0-0-released/). -### Remove SAST analyzer `SAST_GOSEC_CONFIG` variable in favor of custom rulesets +Relevant Issue: [Updates Ruby version 2.5 to 3.0](https://gitlab.com/gitlab-org/gitlab/-/issues/329160) -With the release of [SAST Custom Rulesets](https://docs.gitlab.com/ee/user/application_security/sast/#customize-rulesets) in GitLab 13.5 we allow greater flexibility in configuration options for our Go analyzer (GoSec). As a result we no longer plan to support our less flexible [`SAST_GOSEC_CONFIG`](https://docs.gitlab.com/ee/user/application_security/sast/#analyzer-settings) analyzer setting. This variable was deprecated in GitLab 13.10. -GitLab 14.0 removes the old `SAST_GOSEC_CONFIG variable`. If you use or override `SAST_GOSEC_CONFIG` in your CI file, update your SAST CI configuration or pin to an older version of the GoSec analyzer. We strongly encourage [inheriting and overriding our managed CI templates](https://docs.gitlab.com/ee/user/application_security/sast/#overriding-sast-jobs) to future-proof your CI templates. +### SAST analyzer `SAST_GOSEC_CONFIG` variable -### Remove Ubuntu 19.10 (Eoan Ermine) package +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. -Ubuntu 19.10 (Eoan Ermine) reached end of life on Friday, July 17, 2020. In GitLab Runner 14.0, Ubuntu 19.10 (Eoan Ermine) is no longer available from our package distribution. Refer to [issue #26036](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/26036) for details. +With the release of [SAST Custom Rulesets](https://docs.gitlab.com/ee/user/application_security/sast/#customize-rulesets) in GitLab 13.5 we allow greater flexibility in configuration options for our Go analyzer (GoSec). As a result we no longer plan to support our less flexible [`SAST_GOSEC_CONFIG`](https://docs.gitlab.com/ee/user/application_security/sast/#analyzer-settings) analyzer setting. This variable was deprecated in GitLab 13.10. +GitLab 14.0 removes the old `SAST_GOSEC_CONFIG variable`. If you use or override `SAST_GOSEC_CONFIG` in your CI file, update your SAST CI configuration or pin to an older version of the GoSec analyzer. We strongly encourage [inheriting and overriding our managed CI templates](https://docs.gitlab.com/ee/user/application_security/sast/#overriding-sast-jobs) to future-proof your CI templates. -### Remove `/usr/lib/gitlab-runner` symlink from package +### Service Templates -In GitLab Runner 13.3, a symlink was added from `/user/lib/gitlab-runner/gitlab-runner` to `/usr/bin/gitlab-runner`. In 14.0, the symlink has been removed and the runner is now installed in `/usr/bin/gitlab-runner`. Refer to [issue #26651](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/26651) for details. +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. -### Remove `?w=1` URL parameter to ignore whitespace changes +Service Templates are [removed in GitLab 14.0](https://gitlab.com/groups/gitlab-org/-/epics/5672). They were used to apply identical settings to a large number of projects, but they only did so at the time of project creation. -To create a consistent experience for users based on their preferences, support for toggling whitespace changes via URL parameter has been removed in GitLab 14.0. +While they solved part of the problem, _updating_ those values later proved to be a major pain point. [Project Integration Management](https://docs.gitlab.com/ee/user/admin_area/settings/project_integration_management.html) solves this problem by enabling you to create settings at the Group or Instance level, and projects within that namespace inheriting those settings. -### Remove `FF_RESET_HELPER_IMAGE_ENTRYPOINT` feature flag +### Success and failure for finished build metric conversion -In 14.0, we have deactivated the `FF_RESET_HELPER_IMAGE_ENTRYPOINT` feature flag. Refer to issue [#26679](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/26679) for details. +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. -### Remove `FF_SHELL_EXECUTOR_USE_LEGACY_PROCESS_KILL` feature flag +In GitLab Runner 13.5, we introduced `failed` and `success` states for a job. To support Prometheus rules, we chose to convert `success/failure` to `finished` for the metric. In 14.0, the conversion has now been removed. Refer to [issue #26900](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/26900) for details. -In [GitLab Runner 13.1](https://docs.gitlab.com/runner/executors/shell.html#gitlab-131-and-later), [issue #3376](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3376), we introduced `sigterm` and then `sigkill` to a process in the Shell executor. We also introduced a new feature flag, `FF_SHELL_EXECUTOR_USE_LEGACY_PROCESS_KILL`, so you can use the previous process termination sequence. In GitLab Runner 14.0, [issue #6413](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6413), the feature flag has been removed. +### Terraform template version -### Remove `FF_USE_GO_CLOUD_WITH_CACHE_ARCHIVER` feature flag +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. -GitLab Runner 14.0 removes the `FF_USE_GO_CLOUD_WITH_CACHE_ARCHIVER` feature flag. Refer to [issue #27175](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27175) for details. +As we continuously [develop GitLab's Terraform integrations](https://gitlab.com/gitlab-org/gitlab/-/issues/325312), to minimize customer disruption, we maintain two GitLab CI/CD templates for Terraform: -### Remove `secret_detection_default_branch` job +- The ["latest version" template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Terraform.latest.gitlab-ci.yml), which is updated frequently between minor releases of GitLab (such as 13.10, 13.11, etc). +- The ["major version" template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Terraform.gitlab-ci.yml), which is updated only at major releases (such as 13.0, 14.0, etc). -To ensure Secret Detection was scanning both default branches and feature branches, we introduced two separate secret detection CI jobs (`secret_detection_default_branch` and `secret_detection`) in our managed [`Secret-Detection.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/Secret-Detection.gitlab-ci.yml) template. These two CI jobs created confusion and complexity in the CI rules logic. This deprecation moves the `rule` logic into the `script` section, which then determines how the `secret_detection` job is run (historic, on a branch, commits, etc). -If you override or maintain custom versions of `SAST.gitlab-ci.yml` or `Secret-Detection.gitlab-ci.yml`, you must update your CI templates. We strongly encourage [inheriting and overriding our managed CI templates](https://docs.gitlab.com/ee/user/application_security/secret_detection/#custom-settings-example) to future-proof your CI templates. GitLab 14.0 no longer supports the old `secret_detection_default_branch` job. +At every major release of GitLab, the "latest version" template becomes the "major version" template, inheriting the "latest template" setup. +As we have added many new features to the Terraform integration, the new setup for the "major version" template can be considered a breaking change. -### Remove disk source configuration for GitLab Pages +The latest template supports the [Terraform Merge Request widget](https://docs.gitlab.com/ee/user/infrastructure/mr_integration.html) and +doesn't need additional setup to work with the [GitLab managed Terraform state](https://docs.gitlab.com/ee/user/infrastructure/terraform_state.html). -GitLab Pages [API-based configuration](https://docs.gitlab.com/ee/administration/pages/#gitlab-api-based-configuration) has been available since GitLab 13.0. It replaces the unsupported `disk` source configuration removed in GitLab 14.0, which can no longer be chosen. You should stop using `disk` source configuration, and move to `gitlab` for an API-based configuration. To migrate away from the 'disk' source configuration, set `gitlab_pages['domain_config_source'] = "gitlab"` in your `/etc/gitlab/gitlab.rb` file. We recommend you migrate before updating to GitLab 14.0, to identify and troubleshoot any potential problems before upgrading. +To check the new changes, see the [new "major version" template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Terraform.gitlab-ci.yml). -### Remove legacy DAST domain validation +### Ubuntu 16.04 support -The legacy method of DAST Domain Validation for CI/CD scans was deprecated in GitLab 13.8, and is removed in GitLab 14.0. This method of domain validation only disallows scans if the `DAST_FULL_SCAN_DOMAIN_VALIDATION_REQUIRED` environment variable is set to `true` in the `gitlab-ci.yml` file, and a `Gitlab-DAST-Permission` header on the site is not set to `allow`. This two-step method required users to opt in to using the variable before they could opt out from using the header. +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. -For more information, see the [removal issue](https://gitlab.com/gitlab-org/gitlab/-/issues/293595). +Ubuntu 16.04 [reached end-of-life in April 2021](https://ubuntu.com/about/release-cycle), and no longer receives maintenance updates. We strongly recommend users to upgrade to a newer release, such as 20.04. -### Remove off peak time mode configuration for Docker Machine autoscaling +GitLab 13.12 will be the last release with Ubuntu 16.04 support. -In GitLab Runner 13.0, [issue #5069](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/5069), we introduced new timing options for the GitLab Docker Machine executor. In GitLab Runner 14.0, we have removed the old configuration option, [off peak time mode](https://docs.gitlab.com/runner/configuration/autoscale.html#off-peak-time-mode-configuration-deprecated). +### Ubuntu 19.10 (Eoan Ermine) package -### Remove redundant timestamp field from DORA metrics API payload +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. -The [deployment frequency project-level API](https://docs.gitlab.com/ee/api/dora4_project_analytics.html#list-project-deployment-frequencies) endpoint has been deprecated in favor of the [DORA 4 API](https://docs.gitlab.com/ee/api/dora/metrics.html), which consolidates all the metrics under one API with the specific metric as a required field. As a result, the timestamp field, which doesn't allow adding future extensions and causes performance issues, will be removed. With the old API, an example response would be `{ "2021-03-01": 3, "date": "2021-03-01", "value": 3 }`. The first key/value (`"2021-03-01": 3`) will be removed and replaced by the last two (`"date": "2021-03-01", "value": 3`). +Ubuntu 19.10 (Eoan Ermine) reached end of life on Friday, July 17, 2020. In GitLab Runner 14.0, Ubuntu 19.10 (Eoan Ermine) is no longer available from our package distribution. Refer to [issue #26036](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/26036) for details. -### Remove success and failure for finished build metric conversion +### Unicorn in GitLab self-managed -In GitLab Runner 13.5, we introduced `failed` and `success` states for a job. To support Prometheus rules, we chose to convert `success/failure` to `finished` for the metric. In 14.0, the conversion has now been removed. Refer to [issue #26900](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/26900) for details. +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. -### Remove support for Windows Server 1903 image +[Support for Unicorn](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6078) has been removed in GitLab 14.0 in favor of Puma. [Puma has a multi-threaded architecture](https://docs.gitlab.com/ee/administration/operations/puma.html) which uses less memory than a multi-process application server like Unicorn. On GitLab.com, we saw a 40% reduction in memory consumption by using Puma. -In 14.0, we have removed Windows Server 1903. Microsoft ended support for this version on 2020-08-12. Refer to [issue #27551](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27551) for details. +### WIP merge requests renamed 'draft merge requests' -### Remove support for Windows Server 1909 image +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. -In 14.0, we have removed Windows Server 1909. Microsoft ended support for this version on 2021-05-11. Refer to [issue #27899](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27899) for details. +The WIP (work in progress) status for merge requests signaled to reviewers that the merge request in question wasn't ready to merge. We've renamed the WIP feature to **Draft**, a more inclusive and self-explanatory term. **Draft** clearly communicates the merge request in question isn't ready for review, and makes no assumptions about the progress being made toward it. **Draft** also reduces the cognitive load for new users, non-English speakers, and anyone unfamiliar with the WIP acronym. -### Removed Global `SAST_ANALYZER_IMAGE_TAG` in SAST CI template +### Web Application Firewall (WAF) -With the maturity of GitLab Secure scanning tools, we've needed to add more granularity to our release process. Previously, GitLab shared a major version number for [all analyzers and tools](https://docs.gitlab.com/ee/user/application_security/sast/#supported-languages-and-frameworks). This requires all tools to share a major version, and prevents the use of [semantic version numbering](https://semver.org/). In GitLab 14.0, SAST removes the `SAST_ANALYZER_IMAGE_TAG` global variable in our [managed `SAST.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/SAST.gitlab-ci.yml) CI template, in favor of the analyzer job variable setting the `major.minor` tag in the SAST vendored template. +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. -Each analyzer job now has a scoped `SAST_ANALYZER_IMAGE_TAG` variable, which will be actively managed by GitLab and set to the `major` tag for the respective analyzer. To pin to a specific version, [change the variable value to the specific version tag](https://docs.gitlab.com/ee/user/application_security/sast/#pinning-to-minor-image-version). -If you override or maintain custom versions of `SAST.gitlab-ci.yml`, update your CI templates to stop referencing the global `SAST_ANALYZER_IMAGE_TAG`, and move it to a scoped analyzer job tag. We strongly encourage [inheriting and overriding our managed CI templates](https://docs.gitlab.com/ee/user/application_security/sast/#overriding-sast-jobs) to future-proof your CI templates. This change allows you to more granularly control future analyzer updates with a pinned `major.minor` version. -This deprecation and removal changes our [previously announced plan](https://about.gitlab.com/releases/2021/02/22/gitlab-13-9-released/#pin-static-analysis-analyzers-and-tools-to-minor-versions) to pin the Static Analysis tools. +The Web Application Firewall (WAF) was deprecated in GitLab 13.6 and is removed from GitLab 14.0. The WAF had limitations inherent in the architectural design that made it difficult to meet the requirements traditionally expected of a WAF. By removing the WAF, GitLab is able to focus on improving other areas in the product where more value can be provided to users. Users who currently rely on the WAF can continue to use the free and open source [ModSecurity](https://github.com/SpiderLabs/ModSecurity) project, which is independent from GitLab. Additional details are available in the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/271276). -### Ruby version changed in `Ruby.gitlab-ci.yml` +### Windows Server 1903 image support -By default, the `Ruby.gitlab-ci.yml` file has included Ruby 2.5. +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. -To better support the latest versions of Ruby, the template is changed to use `ruby:latest`, which is currently 3.0. To better understand the changes in Ruby 3.0, please reference the [Ruby-lang.org release announcement](https://www.ruby-lang.org/en/news/2020/12/25/ruby-3-0-0-released/). +In 14.0, we have removed Windows Server 1903. Microsoft ended support for this version on 2020-08-12. Refer to [issue #27551](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27551) for details. -Relevant Issue: [Updates Ruby version 2.5 to 3.0](https://gitlab.com/gitlab-org/gitlab/-/issues/329160) +### Windows Server 1909 image support -### Segments removed from DevOps Adoption API +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. -The first release of the DevOps Adoption report had a concept of **Segments**. Segments were [quickly removed from the report](https://gitlab.com/groups/gitlab-org/-/epics/5251) because they introduced an additional layer of complexity on top of **Groups** and **Projects**. Subsequent iterations of the DevOps Adoption report focus on comparing adoption across groups rather than segments. GitLab 14.0 removes all references to **Segments** [from the GraphQL API](https://gitlab.com/gitlab-org/gitlab/-/issues/324414) and replaces them with **Enabled groups**. +In 14.0, we have removed Windows Server 1909. Microsoft ended support for this version on 2021-05-11. Refer to [issue #27899](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27899) for details. -### Service Templates removed +### `/usr/lib/gitlab-runner` symlink from package -Service Templates are [removed in GitLab 14.0](https://gitlab.com/groups/gitlab-org/-/epics/5672). They were used to apply identical settings to a large number of projects, but they only did so at the time of project creation. +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. -While they solved part of the problem, _updating_ those values later proved to be a major pain point. [Project Integration Management](https://docs.gitlab.com/ee/user/admin_area/settings/project_integration_management.html) solves this problem by enabling you to create settings at the Group or Instance level, and projects within that namespace inheriting those settings. +In GitLab Runner 13.3, a symlink was added from `/user/lib/gitlab-runner/gitlab-runner` to `/usr/bin/gitlab-runner`. In 14.0, the symlink has been removed and the runner is now installed in `/usr/bin/gitlab-runner`. Refer to [issue #26651](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/26651) for details. -### Sidekiq queue selector options no longer accept the 'experimental' prefix +### `?w=1` URL parameter to ignore whitespace changes -GitLab supports a [queue selector](https://docs.gitlab.com/ee/administration/operations/extra_sidekiq_processes.html#queue-selector) to run only a subset of background jobs for a given process. When it was introduced, this option had an 'experimental' prefix (`experimental_queue_selector` in Omnibus, `experimentalQueueSelector` in Helm charts). +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. -As announced in the [13.6 release post](https://about.gitlab.com/releases/2020/11/22/gitlab-13-6-released/#sidekiq-cluster-queue-selector-configuration-option-has-been-renamed), the 'experimental' prefix is no longer supported. Instead, `queue_selector` for Omnibus and `queueSelector` in Helm charts should be used. +To create a consistent experience for users based on their preferences, support for toggling whitespace changes via URL parameter has been removed in GitLab 14.0. -### Ubuntu 16.04 support +### `CI_PROJECT_CONFIG_PATH` variable -Ubuntu 16.04 [reached end-of-life in April 2021](https://ubuntu.com/about/release-cycle), and no longer receives maintenance updates. We strongly recommend users to upgrade to a newer release, such as 20.04. +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. -GitLab 13.12 will be the last release with Ubuntu 16.04 support. +The `CI_PROJECT_CONFIG_PATH` [predefined project variable](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html) +has been removed in favor of `CI_CONFIG_PATH`, which is functionally the same. -### Unicorn removed in favor of Puma for GitLab self-managed +If you are using `CI_PROJECT_CONFIG_PATH` in your pipeline configurations, +please update them to use `CI_CONFIG_PATH` instead. -[Support for Unicorn](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6078) has been removed in GitLab 14.0 in favor of Puma. [Puma has a multi-threaded architecture](https://docs.gitlab.com/ee/administration/operations/puma.html) which uses less memory than a multi-process application server like Unicorn. On GitLab.com, we saw a 40% reduction in memory consumption by using Puma. +### `FF_RESET_HELPER_IMAGE_ENTRYPOINT` feature flag -### Update Auto Deploy template version +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. -In GitLab 14.0, we will update the [Auto Deploy](https://docs.gitlab.com/ee/topics/autodevops/stages.html#auto-deploy) CI template to the latest version. This includes new features, bug fixes, and performance improvements with a dependency on the v2 [auto-deploy-image](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image). Auto Deploy CI tempalte v1 will is deprecated going forward. +In 14.0, we have deactivated the `FF_RESET_HELPER_IMAGE_ENTRYPOINT` feature flag. Refer to issue [#26679](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/26679) for details. -Since the v1 and v2 versions are not backward-compatible, your project might encounter an unexpected failure if you already have a deployed application. Follow the [upgrade guide](https://docs.gitlab.com/ee/topics/autodevops/upgrading_auto_deploy_dependencies.html#upgrade-guide) to upgrade your environments. You can also start using the latest template today by following the [early adoption guide](https://docs.gitlab.com/ee/topics/autodevops/upgrading_auto_deploy_dependencies.html#early-adopters). +### `FF_SHELL_EXECUTOR_USE_LEGACY_PROCESS_KILL` feature flag -### Update CI/CD templates to stop using hardcoded `master` +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. -Our CI/CD templates have been updated to no longer use hard-coded references to a `master` branch. In 14.0, they all use a variable that points to your project's configured default branch instead. If your CI/CD pipeline relies on our built-in templates, verify that this change works with your current configuration. For example, if you have a `master` branch and a different default branch, the updates to the templates may cause changes to your pipeline behavior. For more information, [read the issue](https://gitlab.com/gitlab-org/gitlab/-/issues/324131). +In [GitLab Runner 13.1](https://docs.gitlab.com/runner/executors/shell.html#gitlab-131-and-later), [issue #3376](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3376), we introduced `sigterm` and then `sigkill` to a process in the Shell executor. We also introduced a new feature flag, `FF_SHELL_EXECUTOR_USE_LEGACY_PROCESS_KILL`, so you can use the previous process termination sequence. In GitLab Runner 14.0, [issue #6413](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6413), the feature flag has been removed. -### WIP merge requests renamed 'draft merge requests' +### `FF_USE_GO_CLOUD_WITH_CACHE_ARCHIVER` feature flag -The WIP (work in progress) status for merge requests signaled to reviewers that the merge request in question wasn't ready to merge. We've renamed the WIP feature to **Draft**, a more inclusive and self-explanatory term. **Draft** clearly communicates the merge request in question isn't ready for review, and makes no assumptions about the progress being made toward it. **Draft** also reduces the cognitive load for new users, non-English speakers, and anyone unfamiliar with the WIP acronym. +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. -### Web Application Firewall (WAF) +GitLab Runner 14.0 removes the `FF_USE_GO_CLOUD_WITH_CACHE_ARCHIVER` feature flag. Refer to [issue #27175](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27175) for details. -The Web Application Firewall (WAF) was deprecated in GitLab 13.6 and is removed from GitLab 14.0. The WAF had limitations inherent in the architectural design that made it difficult to meet the requirements traditionally expected of a WAF. By removing the WAF, GitLab is able to focus on improving other areas in the product where more value can be provided to users. Users who currently rely on the WAF can continue to use the free and open source [ModSecurity](https://github.com/SpiderLabs/ModSecurity) project, which is independent from GitLab. Additional details are available in the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/271276). +### `secret_detection_default_branch` job -### `CI_PROJECT_CONFIG_PATH` removed in GitLab 14.0 +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. -GitLab 14.0 removes the `CI_PROJECT_CONFIG_PATH` pre-defined project variable in favor of `CI_CONFIG_PATH`, which is functionally the same. If you are using `CI_PROJECT_CONFIG_PATH` in your pipeline configurations, update them to use `CI_CONFIG_PATH` instead. +To ensure Secret Detection was scanning both default branches and feature branches, we introduced two separate secret detection CI jobs (`secret_detection_default_branch` and `secret_detection`) in our managed [`Secret-Detection.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/Secret-Detection.gitlab-ci.yml) template. These two CI jobs created confusion and complexity in the CI rules logic. This deprecation moves the `rule` logic into the `script` section, which then determines how the `secret_detection` job is run (historic, on a branch, commits, etc). +If you override or maintain custom versions of `SAST.gitlab-ci.yml` or `Secret-Detection.gitlab-ci.yml`, you must update your CI templates. We strongly encourage [inheriting and overriding our managed CI templates](https://docs.gitlab.com/ee/user/application_security/secret_detection/#custom-settings-example) to future-proof your CI templates. GitLab 14.0 no longer supports the old `secret_detection_default_branch` job. -### `CI_PROJECT_CONFIG_PATH` variable has been removed +### `trace` parameter in `jobs` API -The `CI_PROJECT_CONFIG_PATH` [predefined project variable](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html) -has been removed in favor of `CI_CONFIG_PATH`, which is functionally the same. +WARNING: +This feature was changed or removed in 14.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. -If you are using `CI_PROJECT_CONFIG_PATH` in your pipeline configurations, -please update them to use `CI_CONFIG_PATH` instead. +GitLab Runner was updated in GitLab 13.4 to internally stop passing the `trace` parameter to the `/api/jobs/:id` endpoint. GitLab 14.0 deprecates the `trace` parameter entirely for all other requests of this endpoint. Make sure your [GitLab Runner version matches your GitLab version](https://docs.gitlab.com/runner/#gitlab-runner-versions) to ensure consistent behavior. ## 14.1 @@ -388,3 +720,13 @@ In GitLab 14.3, we will remove the ability to filter by `name` in the [list proj The support for [`gitlab_pages['use_legacy_storage']` setting](https://docs.gitlab.com/ee/administration/pages/index.html#domain-source-configuration-before-140) in Omnibus installations has been removed. In 14.0 we removed [`domain_config_source`](https://docs.gitlab.com/ee/administration/pages/index.html#domain-source-configuration-before-140) which had been previously deprecated, and allowed users to specify disk storage. In 14.0 we added `use_legacy_storage` as a **temporary** flag to unblock upgrades, and allow us to debug issues with our users and it was deprecated and communicated for removal in 14.3. + +## 14.6 + +### Limit the number of triggered pipeline to 25K in free tier + +A large amount of triggered pipelines in a single project impacts the performance of GitLab.com. In GitLab 14.6, we are limiting the number of triggered pipelines in a single project on GitLab.com at any given moment to 25,000. This change applies to projects in the free tier only, Premium and Ultimate are not affected by this change. + +### Release CLI distributed as a generic package + +The [release-cli](https://gitlab.com/gitlab-org/release-cli) will be released as a [generic package](https://gitlab.com/gitlab-org/release-cli/-/packages) starting in GitLab 14.2. We will continue to deploy it as a binary to S3 until GitLab 14.5 and stop distributing it in S3 in GitLab 14.6. diff --git a/doc/update/upgrading_from_source.md b/doc/update/upgrading_from_source.md index 22ffcda9138..22367435ae4 100644 --- a/doc/update/upgrading_from_source.md +++ b/doc/update/upgrading_from_source.md @@ -73,7 +73,7 @@ Download Ruby and compile it: ```shell mkdir /tmp/ruby && cd /tmp/ruby -curl --remote-name --progress-bar "https://cache.ruby-lang.org/pub/ruby/2.7/ruby-2.7.4.tar.gz" +curl --remote-name --location --progress-bar "https://cache.ruby-lang.org/pub/ruby/2.7/ruby-2.7.4.tar.gz" echo '3043099089608859fc8cce7f9fdccaa1f53a462457e3838ec3b25a7d609fbc5b ruby-2.7.4.tar.gz' | sha256sum -c - && tar xzf ruby-2.7.4.tar.gz cd ruby-2.7.4 @@ -111,7 +111,7 @@ Download and install Go (for Linux, 64-bit): # Remove former Go installation folder sudo rm -rf /usr/local/go -curl --remote-name --progress-bar "https://go.dev/dl/go1.16.10.linux-amd64.tar.gz" +curl --remote-name --location --progress-bar "https://go.dev/dl/go1.16.10.linux-amd64.tar.gz" echo '414cd18ce1d193769b9e97d2401ad718755ab47816e13b2a1cde203d263b55cf go1.16.10.linux-amd64.tar.gz' | shasum -a256 -c - && \ sudo tar -C /usr/local -xzf go1.16.10.linux-amd64.tar.gz sudo ln -sf /usr/local/go/bin/{go,gofmt} /usr/local/bin/ @@ -410,6 +410,20 @@ Example: Additional instructions here. --> +### 14.5.0 + +As part of [enabling real-time issue assignees](https://gitlab.com/gitlab-org/gitlab/-/issues/330117), Action Cable is now enabled by default, and requires `config/cable.yml` to be present. +You can configure this by running: + +```shell +cd /home/git/gitlab + +sudo -u git -H cp config/cable.yml.example config/cable.yml + +# Change the Redis socket path if you are not using the default Debian / Ubuntu configuration +sudo -u git -H editor config/cable.yml +``` + ### 13.0.1 As part of [deprecating Rack Attack throttles on Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4750), the Rack Attack initializer on GitLab diff --git a/doc/update/zero_downtime.md b/doc/update/zero_downtime.md index 13658e6071b..050c5909934 100644 --- a/doc/update/zero_downtime.md +++ b/doc/update/zero_downtime.md @@ -87,6 +87,9 @@ meet the other online upgrade requirements mentioned above. ## Single-node deployment +WARNING: +You can only upgrade one minor release at a time. + Before following these instructions, note the following **important** information: - You can only upgrade one minor release at a time. So from 13.6 to 13.7, not to 13.8. @@ -158,6 +161,7 @@ you've completed these steps. ## Multi-node / HA deployment +WARNING: You can only upgrade one minor release at a time. So from 13.6 to 13.7, not to 13.8. If you attempt more than one minor release, the upgrade may fail. @@ -531,6 +535,9 @@ procedure. ## Geo deployment **(PREMIUM SELF)** +WARNING: +You can only upgrade one minor release at a time. + The order of steps is important. While following these steps, make sure you follow them in the right order, on the correct node. @@ -651,6 +658,9 @@ setting `gitlab_rails['auto_migrate'] = false` in ## Multi-node / HA deployment with Geo **(PREMIUM SELF)** +WARNING: +You can only upgrade one minor release at a time. + This section describes the steps required to upgrade a multi-node / HA deployment with Geo. Some steps must be performed on a particular node. This node is known as the “deploy node” and is noted through the following diff --git a/doc/user/admin_area/analytics/dev_ops_report.md b/doc/user/admin_area/analytics/dev_ops_report.md index df34cd03d71..2ad18d5f70e 100644 --- a/doc/user/admin_area/analytics/dev_ops_report.md +++ b/doc/user/admin_area/analytics/dev_ops_report.md @@ -39,7 +39,7 @@ feature is available. ## DevOps Adoption **(ULTIMATE SELF)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/247112) in GitLab 13.7 as a [Beta feature](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta). +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/247112) in GitLab 13.7 as a [Beta feature](../../../policy/alpha-beta-support.md#beta-features). > - The Overview tab [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/330401) in GitLab 14.1. > - DAST and SAST metrics [added](https://gitlab.com/gitlab-org/gitlab/-/issues/328033) in GitLab 14.1. > - Fuzz Testing metrics [added](https://gitlab.com/gitlab-org/gitlab/-/issues/330398) in GitLab 14.2. diff --git a/doc/user/admin_area/analytics/usage_trends.md b/doc/user/admin_area/analytics/usage_trends.md index 7901d30c3ea..a9c5adcf838 100644 --- a/doc/user/admin_area/analytics/usage_trends.md +++ b/doc/user/admin_area/analytics/usage_trends.md @@ -33,7 +33,7 @@ At the top of the page, Usage Trends shows total counts for: - Projects - Groups - Issues -- Merge Requests +- Merge requests - Pipelines These figures can be useful for understanding how much data your instance contains in total. diff --git a/doc/user/admin_area/credentials_inventory.md b/doc/user/admin_area/credentials_inventory.md index f9b5168fb08..437a72da767 100644 --- a/doc/user/admin_area/credentials_inventory.md +++ b/doc/user/admin_area/credentials_inventory.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -13,9 +13,14 @@ GitLab administrators are responsible for the overall security of their instance provides a Credentials inventory to keep track of all the credentials that can be used to access their self-managed instance. -Using Credentials inventory, you can see all the personal access tokens (PAT), SSH keys, and GPG keys -that exist in your GitLab instance. In addition, you can [revoke](#revoke-a-users-personal-access-token) -and [delete](#delete-a-users-ssh-key) and see: +Use Credentials inventory to see for your GitLab instance all: + +- Personal access tokens (PAT). +- Project access tokens (GitLab 14.8 and later). +- SSH keys. +- GPG keys. + +You can also [revoke](#revoke-a-users-personal-access-token) and [delete](#delete-a-users-ssh-key) and see: - Who they belong to. - Their access scope. @@ -28,17 +33,13 @@ To access the Credentials inventory: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Credentials**. -The following is an example of the Credentials inventory page: - - - ## Revoke a user's personal access token > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214811) in GitLab 13.4. If you see a **Revoke** button, you can revoke that user's PAT. Whether you see a **Revoke** button depends on the token state, and if an expiration date has been set. For more information, see the following table: -| Token state | [Token expiration enforced?](settings/account_and_limit_settings.md#allow-expired-personal-access-tokens-to-be-used) | Show Revoke button? | Comments | +| Token state | [Token expiration enforced?](settings/account_and_limit_settings.md#allow-expired-personal-access-tokens-to-be-used-deprecated) | Show Revoke button? | Comments | |-------------|------------------------|--------------------|----------------------------------------------------------------------------| | Active | Yes | Yes | Allows administrators to revoke the PAT, such as for a compromised account | | Active | No | Yes | Allows administrators to revoke the PAT, such as for a compromised account | @@ -49,6 +50,15 @@ If you see a **Revoke** button, you can revoke that user's PAT. Whether you see When a PAT is revoked from the credentials inventory, the instance notifies the user by email. +## Revoke a user's project access token + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/243833) in GitLab 14.8. + +The **Revoke** button next to a project access token can be selected to revoke that particular project access token. This will both: + +- Revoke the token project access token. +- Enqueue a background worker to delete the project bot user. + ## Delete a user's SSH key > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/225248) in GitLab 13.5. diff --git a/doc/user/admin_area/img/credentials_inventory_v13_10.png b/doc/user/admin_area/img/credentials_inventory_v13_10.png Binary files differdeleted file mode 100644 index 2790ca70fba..00000000000 --- a/doc/user/admin_area/img/credentials_inventory_v13_10.png +++ /dev/null diff --git a/doc/user/admin_area/index.md b/doc/user/admin_area/index.md index ba0802b3b7a..57a4a746ff0 100644 --- a/doc/user/admin_area/index.md +++ b/doc/user/admin_area/index.md @@ -32,7 +32,7 @@ The Admin Area is made up of the following sections: | **{slight-frown}** Abuse Reports | Manage [abuse reports](review_abuse_reports.md) submitted by your users. | | **{license}** License | Upload, display, and remove [licenses](license.md). | | **{cloud-gear}** Kubernetes | Create and manage instance-level [Kubernetes clusters](../instance/clusters/index.md). | -| **{push-rules}** Push rules | Configure pre-defined Git [push rules](../../push_rules/push_rules.md) for projects. Also, configure [merge requests approvers rules](merge_requests_approvals.md). | +| **{push-rules}** Push rules | Configure pre-defined Git [push rules](../project/repository/push_rules.md) for projects. Also, configure [merge requests approvers rules](merge_requests_approvals.md). | | **{location-dot}** Geo | Configure and maintain [Geo nodes](geo_nodes.md). | | **{key}** Deploy keys | Create instance-wide [SSH deploy keys](../project/deploy_keys/index.md). | | **{lock}** Credentials | View [credentials](credentials_inventory.md) that can be used to access your instance. | @@ -107,23 +107,6 @@ You can combine the filter options. For example, to list only public projects wi 1. Click the **Public** tab. 1. Enter `score` in the **Filter by name...** input box. -#### Projects pending deletion **(PREMIUM SELF)** - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37014) in GitLab 13.3. -> - [Tab renamed](https://gitlab.com/gitlab-org/gitlab/-/issues/347468) from **Deleted projects** in GitLab 14.6. - -When delayed project deletion is [enabled for a group](../group/index.md#enable-delayed-project-deletion), -projects within that group are not deleted immediately, but only after a delay. To access a list of all projects that are pending deletion: - -1. On the top bar, select **Menu > Projects > Explore projects**. -1. Select the **Pending deletion** tab (in GitLab 14.6 and later) or the **Deleted projects** tab (GitLab 14.5 and earlier). - -Listed for each project is: - -- The time the project was marked for deletion. -- The time the project is scheduled for final deletion. -- A **Restore** link to stop the project being eventually deleted. - ### Administering Users You can administer all users in the GitLab instance from the Admin Area's Users page: diff --git a/doc/user/admin_area/license.md b/doc/user/admin_area/license.md index c3f0c94db21..22133e30aa0 100644 --- a/doc/user/admin_area/license.md +++ b/doc/user/admin_area/license.md @@ -27,9 +27,8 @@ If you have questions or need assistance upgrading from GitLab CE to EE, ## Activate GitLab EE with an activation code In GitLab Enterprise Edition 14.1 and later, you need an activation code to activate -your instance. To get an activation code, [purchase a license](https://about.gitlab.com/pricing/) -or sign up for a [free trial](https://about.gitlab.com/free-trial/). The activation -code is a 24-character alphanumeric string you receive in a confirmation email. +your instance. To get an activation code you have to [purchase a license](https://about.gitlab.com/pricing/). +The activation code is a 24-character alphanumeric string you receive in a confirmation email. You can also sign in to the [Customers Portal](https://customers.gitlab.com/customers/sign_in) to copy the activation code to your clipboard. @@ -60,8 +59,10 @@ Otherwise, to upload your license: 1. On the left sidebar, select **Settings**. 1. In the **License file** area, select **Upload a license**. 1. Upload a license: - - For a file, select **Upload `.gitlab-license` file**, **Choose file**, and - select the license file from your local machine. + - For a file, either: + - Select **Upload `.gitlab-license` file**, then **Choose File** and + select the license file from your local machine. + - Drag and drop the license file to the **Drag your license file here** area. - For plain text, select **Enter license key** and paste the contents in **License key**. 1. Select the **Terms of Service** checkbox. @@ -129,6 +130,8 @@ the current date range is the active license. When you upload a future-dated license, it doesn't take effect until its applicable date. You can view all active subscriptions in the **Subscription history** table. +You can also [export](../../subscriptions/self_managed/index.md) your license usage information to a CSV file. + NOTE: In GitLab 13.6 and earlier, a banner about an expiring license may continue to display when you upload a new license. This happens when the start date of the new license diff --git a/doc/user/admin_area/moderate_users.md b/doc/user/admin_area/moderate_users.md index ee38664fa66..e8db319df77 100644 --- a/doc/user/admin_area/moderate_users.md +++ b/doc/user/admin_area/moderate_users.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -208,7 +208,13 @@ Users can also be activated using the [GitLab API](../../api/users.md#activate-u ## Ban and unban users -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/327353) in GitLab 14.2. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/327353) in GitLab 14.2 [with a flag](../../administration/feature_flags.md) named `ban_user_feature_flag`. Disabled by default. +> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/330667) in GitLab 14.8. + +FLAG: +On self-managed GitLab, by default this feature is available. +To hide the feature, ask an administrator to [disable the feature flag](../../administration/feature_flags.md) named `ban_user_feature_flag`. +On GitLab.com, this feature is available to GitLab.com administrators only. GitLab administrators can ban and unban users. Banned users are blocked, and their issues are hidden. The banned user's comments are still displayed. Hiding a banned user's comments is [tracked in this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/327356). diff --git a/doc/user/admin_area/monitoring/health_check.md b/doc/user/admin_area/monitoring/health_check.md index 75905d60c4e..213bddec325 100644 --- a/doc/user/admin_area/monitoring/health_check.md +++ b/doc/user/admin_area/monitoring/health_check.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/admin_area/reporting/spamcheck.md b/doc/user/admin_area/reporting/spamcheck.md new file mode 100644 index 00000000000..02d7cd01139 --- /dev/null +++ b/doc/user/admin_area/reporting/spamcheck.md @@ -0,0 +1,65 @@ +--- +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# Spamcheck anti-spam service **(PREMIUM SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6259) in GitLab 14.8. + +[Spamcheck](https://gitlab.com/gitlab-org/spamcheck) is an anti-spam engine +developed by GitLab originally to combat rising amount of spam in GitLab.com, +and later made public to be used in self-managed GitLab instances. + +## Enable Spamcheck + +Spamcheck is only available for package-based installations: + +1. Edit `/etc/gitlab/gitlab.rb` and enable Spamcheck: + + ```ruby + spamcheck['enable'] = true + ``` + +1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. Verify that the new services `spamcheck` and `spam-classifier` are + up and running: + + ```shell + sudo gitlab-ctl status + ``` + +## Configure GitLab to use Spamcheck + +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Settings > Reporting**. +1. Expand **Spam and Anti-bot Protection**. +1. Update the Spam Check settings: + 1. Check the "Enable Spam Check via external API endpoint" checkbox. + 1. For **URL of the external Spam Check endpoint** use `grpc://localhost:8001`. + 1. Leave **Spam Check API key** blank. +1. Select **Save changes**. + +NOTE: +In single-node instances, Spamcehck runs over `localhost`, and hence is running +in an unauthenticated mode. If on multi-node instances where GitLab runs on one +server and Spamcheck runs on another server listening over a public endpoint, it +is recommended to enforce some sort of authentication using a reverse proxy in +front of the Spamcheck service that can be used along with an API key. One +example would be to use `JWT` authentication for this and specifying a bearer +token as the API key. +[Native authentication for Spamcheck is in the works](https://gitlab.com/gitlab-com/gl-security/engineering-and-research/automation-team/spam/spamcheck/-/issues/171). + +## Running Spamcheck over TLS + +Spamcheck service on its own can not communicate directly over TLS with GitLab. +However, Spamcheck can be deployed behind a reverse proxy which performs TLS +termination. In such a scenario, GitLab can be made to communicate with +Spamcheck over TLS by specifying `tls://` scheme for the external Spamcheck URL +instead of `grpc://` in the Admin settings. diff --git a/doc/user/admin_area/review_abuse_reports.md b/doc/user/admin_area/review_abuse_reports.md index 4c5a241ab18..ec8e6f2dda4 100644 --- a/doc/user/admin_area/review_abuse_reports.md +++ b/doc/user/admin_area/review_abuse_reports.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference, howto --- diff --git a/doc/user/admin_area/settings/account_and_limit_settings.md b/doc/user/admin_area/settings/account_and_limit_settings.md index f748f575419..1d982196228 100644 --- a/doc/user/admin_area/settings/account_and_limit_settings.md +++ b/doc/user/admin_area/settings/account_and_limit_settings.md @@ -234,10 +234,14 @@ Once a lifetime for SSH keys is set, GitLab: NOTE: When a user's SSH key becomes invalid they can delete and re-add the same key again. -## Allow expired SSH keys to be used **(ULTIMATE SELF)** +## Allow expired SSH keys to be used (DEPRECATED) **(ULTIMATE SELF)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/250480) in GitLab 13.9. > - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/320970) in GitLab 14.0. +> - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/351963) in GitLab 14.8. + +WARNING: +This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/351963) in GitLab 14.8. By default, expired SSH keys **are not usable**. @@ -283,10 +287,14 @@ Once a lifetime for personal access tokens is set, GitLab: allowed lifetime. Three hours is given to allow administrators to change the allowed lifetime, or remove it, before revocation takes place. -## Allow expired Personal Access Tokens to be used **(ULTIMATE SELF)** +## Allow expired Personal Access Tokens to be used (DEPRECATED) **(ULTIMATE SELF)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214723) in GitLab 13.1. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/296881) in GitLab 13.9. +> - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/351962) in GitLab 14.8. + +WARNING: +This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/351962) in GitLab 14.8. By default, expired personal access tokens (PATs) **are not usable**. diff --git a/doc/user/admin_area/settings/continuous_integration.md b/doc/user/admin_area/settings/continuous_integration.md index e18808ffb41..18379471bcf 100644 --- a/doc/user/admin_area/settings/continuous_integration.md +++ b/doc/user/admin_area/settings/continuous_integration.md @@ -197,6 +197,12 @@ To enable or disable the banner: ## Required pipeline configuration **(PREMIUM SELF)** +WARNING: +Required pipeline configurations is in its end-of-life process for Premium users. It's +[deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/352316) for use in GitLab 14.8, +and planned to be unavailable for Premium users in GitLab 15.0. This feature is planned to continue +to be available for Ultimate users. Ultimate users are not impacted by this deprecation and removal. + NOTE: An alternative [compliance solution](../../project/settings/index.md#compliance-pipeline-configuration) is available. We recommend this alternative solution because it provides greater flexibility, diff --git a/doc/user/admin_area/settings/deprecated_api_rate_limits.md b/doc/user/admin_area/settings/deprecated_api_rate_limits.md index 9be703f3b82..d651e445a95 100644 --- a/doc/user/admin_area/settings/deprecated_api_rate_limits.md +++ b/doc/user/admin_area/settings/deprecated_api_rate_limits.md @@ -30,7 +30,7 @@ for deprecated API endpoints. No other new features are provided by this overrid Prerequisites: -- You must have the Administrator role for your instance. +- You must have administrator access for your instance. To override the general user and IP rate limits for requests to deprecated API endpoints: diff --git a/doc/user/admin_area/settings/email.md b/doc/user/admin_area/settings/email.md index 6bc9e97629c..e4fc3b6e6d4 100644 --- a/doc/user/admin_area/settings/email.md +++ b/doc/user/admin_area/settings/email.md @@ -56,7 +56,7 @@ To change the hostname used in private commit emails: NOTE: After the hostname is configured, every private commit email using the previous hostname is not -recognized by GitLab. This can directly conflict with certain [Push rules](../../../push_rules/push_rules.md) such as +recognized by GitLab. This can directly conflict with certain [Push rules](../../project/repository/push_rules.md) such as `Check whether author is a GitLab user` and `Check whether committer is the current authenticated user`. ## Custom additional text **(PREMIUM SELF)** diff --git a/doc/user/admin_area/settings/external_authorization.md b/doc/user/admin_area/settings/external_authorization.md index 4fd7c59ef24..ef980981fec 100644 --- a/doc/user/admin_area/settings/external_authorization.md +++ b/doc/user/admin_area/settings/external_authorization.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -105,7 +105,7 @@ label defined in the [global settings](#configuration) is used. The label is shown on all project pages in the upper right corner. - + <!-- ## Troubleshooting diff --git a/doc/user/admin_area/settings/files_api_rate_limits.md b/doc/user/admin_area/settings/files_api_rate_limits.md index 675561ce9cf..7305e49b0d2 100644 --- a/doc/user/admin_area/settings/files_api_rate_limits.md +++ b/doc/user/admin_area/settings/files_api_rate_limits.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Files API rate limits **(FREE SELF)** +# Rate limits on Repository files API **(FREE SELF)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68561) in GitLab 14.3. > - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75918) in GitLab 14.6. [Feature flag files_api_throttling](https://gitlab.com/gitlab-org/gitlab/-/issues/338903) removed. @@ -26,7 +26,7 @@ for the Files API. No other new features are provided by this override. Prerequisite: -- You must have the Administrator role for your instance. +- You must have administrator access for your instance. To override the general user and IP rate limits for requests to the Repository files API: diff --git a/doc/user/admin_area/settings/git_lfs_rate_limits.md b/doc/user/admin_area/settings/git_lfs_rate_limits.md index adc6cc2b11b..c10300baeef 100644 --- a/doc/user/admin_area/settings/git_lfs_rate_limits.md +++ b/doc/user/admin_area/settings/git_lfs_rate_limits.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Git LFS Rate Limits **(FREE SELF)** +# Rate limits on Git LFS **(FREE SELF)** [Git LFS (Large File Storage)](../../../topics/git/lfs/index.md) is a Git extension for handling large files. If you use Git LFS in your repository, common Git operations diff --git a/doc/user/admin_area/settings/img/classification_label_on_project_page.png b/doc/user/admin_area/settings/img/classification_label_on_project_page.png Binary files differdeleted file mode 100644 index 4aedb332cec..00000000000 --- a/doc/user/admin_area/settings/img/classification_label_on_project_page.png +++ /dev/null diff --git a/doc/user/admin_area/settings/img/classification_label_on_project_page_v14_8.png b/doc/user/admin_area/settings/img/classification_label_on_project_page_v14_8.png Binary files differnew file mode 100644 index 00000000000..4bd2e7d389b --- /dev/null +++ b/doc/user/admin_area/settings/img/classification_label_on_project_page_v14_8.png diff --git a/doc/user/admin_area/settings/index.md b/doc/user/admin_area/settings/index.md index 2820f3ae9df..a581fd4aebc 100644 --- a/doc/user/admin_area/settings/index.md +++ b/doc/user/admin_area/settings/index.md @@ -137,6 +137,7 @@ The **Network** settings contain: - [Incident Management Limits](../../../operations/incident_management/index.md) - Limit the number of inbound alerts that can be sent to a project. - [Notes creation limit](rate_limit_on_notes_creation.md) - Set a rate limit on the note creation requests. +- [Get single user limit](rate_limit_on_users_api.md) - Set a rate limit on users API endpoint to get a user by ID. ### Preferences @@ -160,7 +161,7 @@ The **Preferences** settings contain: The **Reporting** settings contain: - [Spam and Anti-bot Protection](../../../integration/recaptcha.md) - - Enable anti-spam services, like reCAPTCHA or Akismet, and set IP limits. + Enable anti-spam services, like reCAPTCHA, Akismet or [Spamcheck](../reporting/spamcheck.md), and set IP limits. - [Abuse reports](../review_abuse_reports.md) - Set notification email for abuse reports. ### Repository diff --git a/doc/user/admin_area/settings/rate_limit_on_users_api.md b/doc/user/admin_area/settings/rate_limit_on_users_api.md new file mode 100644 index 00000000000..7954055f38b --- /dev/null +++ b/doc/user/admin_area/settings/rate_limit_on_users_api.md @@ -0,0 +1,33 @@ +--- +type: reference +stage: Manage +group: Authentication & Authorization +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Rate limits on Users API **(FREE SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78364) in GitLab 14.8. + +You can configure the per user rate limit for requests to [Users API](../../../api/users.md). + +To change the rate limit: + +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Settings > Network**. +1. Expand **Users API rate limit**. +1. In the **Maximum requests per 10 minutes** text box, enter the new value. +1. Optional. In the **Users to exclude from the rate limit** box, list users allowed to exceed the limit. +1. Select **Save changes**. + +This limit is: + +- Applied independently per user. +- Not applied per IP address. + +The default value is `300`. + +Requests over the rate limit are logged into the `auth.log` file. + +For example, if you set a limit of 300, requests to the `GET /users/:id` API endpoint +exceeding a rate of 300 per 10 minutes are blocked. Access to the endpoint is allowed after ten minutes have elapsed. diff --git a/doc/user/admin_area/settings/sign_in_restrictions.md b/doc/user/admin_area/settings/sign_in_restrictions.md index 52b20d5b437..c63cd88eeb4 100644 --- a/doc/user/admin_area/settings/sign_in_restrictions.md +++ b/doc/user/admin_area/settings/sign_in_restrictions.md @@ -38,7 +38,7 @@ they do not have access to all projects, groups, or the **Admin Area** menu. To access potentially dangerous resources, an administrator can activate Admin Mode by: - Selecting the *Enable Admin Mode* button -- Trying to access any part of the UI that requires an administrator role, specifically those which call `/admin` endpoints. +- Trying to access any part of the UI that requires administrator access, specifically those which call `/admin` endpoints. The main use case allows administrators to perform their regular tasks as a regular user, based on their memberships, without having to set up a second account for diff --git a/doc/user/admin_area/settings/user_and_ip_rate_limits.md b/doc/user/admin_area/settings/user_and_ip_rate_limits.md index d713ef4b4e0..88be73c3215 100644 --- a/doc/user/admin_area/settings/user_and_ip_rate_limits.md +++ b/doc/user/admin_area/settings/user_and_ip_rate_limits.md @@ -22,6 +22,11 @@ NOTE: By default, all Git operations are first tried unauthenticated. Because of this, HTTP Git operations may trigger the rate limits configured for unauthenticated requests. +NOTE: +[In GitLab 14.8 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/344807), +the rate limits for API requests don't affect requests made by the frontend, as these are always +counted as web traffic. + ## Enable unauthenticated API request rate limit To enable the unauthenticated request rate limit: diff --git a/doc/user/admin_area/settings/visibility_and_access_controls.md b/doc/user/admin_area/settings/visibility_and_access_controls.md index 82e0d3d27d4..c38b2455a8d 100644 --- a/doc/user/admin_area/settings/visibility_and_access_controls.md +++ b/doc/user/admin_area/settings/visibility_and_access_controls.md @@ -7,12 +7,12 @@ type: reference # Control access and visibility **(FREE SELF)** -GitLab enables users with the [Administrator role](../../permissions.md) to enforce +GitLab enables users with administrator access to enforce specific controls on branches, projects, snippets, groups, and more. To access the visibility and access control options: -1. Sign in to GitLab as a user with [Administrator role](../../permissions.md). +1. Sign in to GitLab as a user with Administrator access level. 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > General**. 1. Expand the **Visibility and access controls** section. @@ -29,7 +29,7 @@ or configure [branch protection for groups](../../group/index.md#change-the-defa To change the default branch protection for the entire instance: -1. Sign in to GitLab as a user with [Administrator role](../../permissions.md). +1. Sign in to GitLab as a user with Administrator access level. 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > General**. 1. Expand the **Visibility and access controls** section. @@ -55,7 +55,7 @@ can be overridden on a per-group basis by the group's owner. In [GitLab Premium or higher](https://about.gitlab.com/pricing/), GitLab administrators can disable this privilege for group owners, enforcing the instance-level protection rule: -1. Sign in to GitLab as a user with [Administrator role](../../permissions.md). +1. Sign in to GitLab as a user with Administrator access level. 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > General**. 1. Expand the **Visibility and access controls** section. @@ -71,7 +71,7 @@ Instance-level protections for project creation define which roles can [add projects to a group](../../group/index.md#specify-who-can-add-projects-to-a-group) on the instance. To alter which roles have permission to create projects: -1. Sign in to GitLab as a user with [Administrator role](../../permissions.md). +1. Sign in to GitLab as a user with Administrator access level. 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > General**. 1. Expand the **Visibility and access controls** section. @@ -84,9 +84,9 @@ on the instance. To alter which roles have permission to create projects: ## Restrict project deletion to Administrators **(PREMIUM SELF)** Anyone with the **Owner** role, either at the project or group level, can -delete a project. To allow only users with the Administrator role to delete projects: +delete a project. To allow only users with administrator access to delete projects: -1. Sign in to GitLab as a user with [Administrator role](../../permissions.md). +1. Sign in to GitLab as a user with Administrator access level. 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > General**. 1. Expand the **Visibility and access controls** section. @@ -137,7 +137,7 @@ Alternatively, projects that are marked for removal can be deleted immediately. To set the default [visibility levels for new projects](../../../public_access/public_access.md): -1. Sign in to GitLab as a user with [Administrator role](../../permissions.md). +1. Sign in to GitLab as a user with Administrator access level. 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > General**. 1. Expand the **Visibility and access controls** section. @@ -152,7 +152,7 @@ To set the default [visibility levels for new projects](../../../public_access/p To set the default visibility levels for new [snippets](../../snippets.md): -1. Sign in to GitLab as a user with [Administrator role](../../permissions.md). +1. Sign in to GitLab as a user with Administrator access level. 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > General**. 1. Expand the **Visibility and access controls** section. @@ -166,7 +166,7 @@ For more details on snippet visibility, read To set the default visibility levels for new groups: -1. Sign in to GitLab as a user with [Administrator role](../../permissions.md). +1. Sign in to GitLab as a user with Administrator access level. 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > General**. 1. Expand the **Visibility and access controls** section. @@ -183,7 +183,7 @@ For more details on group visibility, see To restrict visibility levels for projects, snippets, and selected pages: -1. Sign in to GitLab as a user with [Administrator role](../../permissions.md). +1. Sign in to GitLab as a user with Administrator access level. 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > General**. 1. Expand the **Visibility and access controls** section. @@ -200,7 +200,7 @@ For more details on project visibility, see You can specify from which hosting sites users can [import their projects](../../project/import/index.md): -1. Sign in to GitLab as a user with [Administrator role](../../permissions.md). +1. Sign in to GitLab as a user with Administrator access level. 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > General**. 1. Expand the **Visibility and access controls** section. @@ -212,7 +212,7 @@ You can specify from which hosting sites users can [import their projects](../.. To enable the export of [projects and their data](../../../user/project/settings/import_export.md#export-a-project-and-its-data): -1. Sign in to GitLab as a user with [Administrator role](../../permissions.md). +1. Sign in to GitLab as a user with Administrator access level. 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > General**. 1. Expand the **Visibility and access controls** section. @@ -228,7 +228,7 @@ The GitLab restrictions apply at the application level. To specify the enabled Git access protocols: -1. Sign in to GitLab as a user with [Administrator role](../../permissions.md). +1. Sign in to GitLab as a user with Administrator access level. 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > General**. 1. Expand the **Visibility and access controls** section. @@ -280,7 +280,7 @@ NOTE: SSH clone URLs can be customized in `gitlab.rb` by setting `gitlab_rails['gitlab_ssh_host']` and other related settings. -## Configure defaults for RSA, DSA, ECDSA, ED25519 SSH keys +## Configure defaults for RSA, DSA, ECDSA, ED25519, ECDSA_SK, ED25519_SK SSH keys These options specify the permitted types and lengths for SSH keys. diff --git a/doc/user/analytics/code_review_analytics.md b/doc/user/analytics/code_review_analytics.md index 066b45aeb39..18a6ca20bc7 100644 --- a/doc/user/analytics/code_review_analytics.md +++ b/doc/user/analytics/code_review_analytics.md @@ -17,7 +17,7 @@ requests, and: - Take action on individual merge requests. - Reduce overall cycle time. -Code Review Analytics is available to users with at least the Reporter [role](../permissions.md), and displays a table of open merge requests that have at least one non-author comment. The review time is measured from the time the first non-author comment was submitted. +Code Review Analytics is available to users with at least the Reporter role, and displays a table of open merge requests that have at least one non-author comment. The review time is measured from the time the first non-author comment was submitted. NOTE: Initially, no data appears. Data is populated as users comment on open merge requests. diff --git a/doc/user/analytics/img/issues_created_per_month_v13_11.png b/doc/user/analytics/img/issues_created_per_month_v13_11.png Binary files differdeleted file mode 100644 index da3340bfdc2..00000000000 --- a/doc/user/analytics/img/issues_created_per_month_v13_11.png +++ /dev/null diff --git a/doc/user/analytics/img/issues_created_per_month_v14_8.png b/doc/user/analytics/img/issues_created_per_month_v14_8.png Binary files differnew file mode 100644 index 00000000000..7dcfa73ea19 --- /dev/null +++ b/doc/user/analytics/img/issues_created_per_month_v14_8.png diff --git a/doc/user/analytics/img/mr_mean_time_to_merge_metric_v13_9.png b/doc/user/analytics/img/mr_mean_time_to_merge_metric_v13_9.png Binary files differdeleted file mode 100644 index ad108dabf73..00000000000 --- a/doc/user/analytics/img/mr_mean_time_to_merge_metric_v13_9.png +++ /dev/null diff --git a/doc/user/analytics/img/mr_throughput_table_v13_3.png b/doc/user/analytics/img/mr_throughput_table_v13_3.png Binary files differdeleted file mode 100644 index bb63770dc3f..00000000000 --- a/doc/user/analytics/img/mr_throughput_table_v13_3.png +++ /dev/null diff --git a/doc/user/analytics/issue_analytics.md b/doc/user/analytics/issue_analytics.md index 6aa2f594532..62fff443073 100644 --- a/doc/user/analytics/issue_analytics.md +++ b/doc/user/analytics/issue_analytics.md @@ -34,7 +34,7 @@ You can change the total number of months displayed by setting a URL parameter. For example, `https://gitlab.com/groups/gitlab-org/-/issues_analytics?months_back=15` shows a total of 15 months for the chart in the GitLab.org group. - + ## Drill into the information diff --git a/doc/user/analytics/merge_request_analytics.md b/doc/user/analytics/merge_request_analytics.md index bb110ab495f..f9ca06c0ef9 100644 --- a/doc/user/analytics/merge_request_analytics.md +++ b/doc/user/analytics/merge_request_analytics.md @@ -1,119 +1,66 @@ --- -description: "Merge Request Analytics help you understand the efficiency of your code review process, and the productivity of your team." # Up to ~200 chars long. They will be displayed in Google Search snippets. It may help to write the page intro first, and then reuse it here. +description: "Merge request analytics help you understand the efficiency of your code review process, and the productivity of your team." # Up to ~200 chars long. They will be displayed in Google Search snippets. It may help to write the page intro first, and then reuse it here. stage: Manage group: Optimize info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Merge Request Analytics **(PREMIUM)** +# Merge request analytics **(PREMIUM)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229045) in GitLab 13.3. > - Moved to GitLab Premium in 13.9. -Merge Request Analytics helps you understand the efficiency of your code review process, and the productivity of your team. +Use merge request analytics to view: -## Overview +- The number of merge requests your organization merged per month. +- The average time between merge request creation and merge. +- Information about each merged merge request. -Merge Request Analytics displays information that will help you evaluate the efficiency and productivity of your merge request process. +You can use merge request analytics to identify: -The Throughput chart shows the number of merge requests merged, by month. Merge request throughput is -a common measure of productivity in software engineering. Although imperfect, the average throughput can -be a meaningful benchmark of your team's overall productivity. +- Low or high productivity months. +- Efficiency and productivity of your merge request process. +- Efficiency of your code review process. -To access Merge Request Analytics: +## View merge request analytics + +You must have at least the Reporter role to view merge request analytics. + +To view merge request analytics: 1. On the top bar, select **Menu > Projects** and find your project. 1. On the left sidebar, select **Analytics > Merge request**. -## Use cases +## View merge requests merged per month -This feature is designed for [development team leaders](https://about.gitlab.com/handbook/marketing/strategic-marketing/roles-personas/#delaney-development-team-lead) -and others who want to understand broad patterns in code review and productivity. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/232651) in GitLab 13.3. +> - Filtering [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229266) in GitLab 13.4 -You can use Merge Request Analytics to expose when your team is most and least productive, and -identify improvements that might substantially accelerate your development cycle. +To view the number of merge requests merged per month: -Merge Request Analytics could be used when: - -- You want to know if you were more productive this month than last month, or 12 months ago. -- You want to drill into low- or high-productivity months to understand the work that took place. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Analytics > Merge request**. +1. Optional. Filter results: + 1. Select the filter bar. + 1. Select a parameter. + 1. Select a value or enter text to refine the results. + 1. To adjust the date range: + - In the **From** field, select a start date. + - In the **To** field, select an end date. -## Visualizations and data +The **Throughput** chart shows the number of merge requests merged per month. -The following visualizations and data are available, representing all merge requests that were merged in the given date range. +The table shows up to 20 merge requests per page, and includes +information about each merge request. -### Mean time to merge +## View average time between merge request creation and merge > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229389) in GitLab 13.9. -The mean time to merge (MTTM) metric shows the average time between when a merge request is created, -and when it is merged. To view how the MTTM changes over time, compare MTTM across different date ranges. - - - -### Throughput chart - -The throughput chart shows the number of merge requests merged per month. - - - -### Throughput table - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/232651) in GitLab 13.3. - -The Throughput table displays the most recent merge requests merged in the date range. The -table displays up to 20 merge requests at a time. If there are more than 20 merge requests, -you can paginate to them. For each merge request, you can review the following data: - -- Title (as a link to the merge request itself) -- ID -- Pipeline status -- Label count -- Comment count -- Approval count (if approved) -- Date merged -- Time to merge -- Milestone -- Commit count -- Pipeline count -- Line change counts -- Assignees - - +Use the number in **Mean time to merge** to view the average time between when a merge request is +created and when it's merged. -## Filter the data +To view **Mean time to merge**: -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229266) in GitLab 13.4 - -You can filter the data that is presented on the page based on the following parameters: - -- Author -- Assignee -- Label -- Milestone -- Source branch -- Target branch - -To filter results: - -1. Select the filter bar. -1. Select a parameter to filter by. -1. Select a value from the autocompleted results, or enter search text to refine the results. -1. Press Enter. - -## Date range - -The date range is set to the past 12 months by default. You can modify the date range by changing the "From" and/or "To" values that appear alongside the filter bar. After changing either value, the data displayed on the page will update automatically. - -## Tip: Bookmark preferred settings - -You can bookmark preferred filters and date ranges. After you have applied a change to the -filter bar or the date range, you'll see that information in the URL. You can create a -bookmark for those preferred settings in your browser. - -## Permissions - -The **Merge Request Analytics** feature can be accessed only: - -- On [GitLab Premium](https://about.gitlab.com/pricing/) and above. -- By users with at least the Reporter [role](../permissions.md). +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Analytics > Merge request**. diff --git a/doc/user/analytics/productivity_analytics.md b/doc/user/analytics/productivity_analytics.md index e1ba2f5565e..be710f8cbd7 100644 --- a/doc/user/analytics/productivity_analytics.md +++ b/doc/user/analytics/productivity_analytics.md @@ -65,7 +65,7 @@ request is merged. Select the dropdown to view: The right-side histogram shows the size or complexity of a merge request. Select the dropdown to view: - + - Number of commits per merge request. - Number of lines of code (LOC) per commit. - Number of files touched. @@ -103,4 +103,4 @@ You can filter analytics based on a date range. To filter results: The **Productivity Analytics** dashboard can be accessed only: - On [GitLab Premium](https://about.gitlab.com/pricing/) and above. -- By users with at least the Reporter [role](../permissions.md). +- By users with at least the Reporter role. diff --git a/doc/user/application_security/api_fuzzing/index.md b/doc/user/application_security/api_fuzzing/index.md index be6b06a0797..4eb721f8832 100644 --- a/doc/user/application_security/api_fuzzing/index.md +++ b/doc/user/application_security/api_fuzzing/index.md @@ -114,6 +114,7 @@ To generate an API Fuzzing configuration snippet: > - Support for OpenAPI Specification v3.0 was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/228652) in GitLab 13.9. > - Support for OpenAPI Specification using YAML format was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/330583) in GitLab 14.0. > - Support for OpenAPI Specification v3.1 was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/327268) in GitLab 14.2. +> - Support to generate media type `application/xml` was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/327268) in GitLab 14.8. The [OpenAPI Specification](https://www.openapis.org/) (formerly the Swagger Specification) is an API description format for REST APIs. This section shows you how to configure API fuzzing using an OpenAPI Specification to provide information about the target API to test. @@ -125,6 +126,7 @@ the body generation is limited to these body types: - `application/x-www-form-urlencoded` - `multipart/form-data` - `application/json` +- `application/xml` #### Configure Web API fuzzing with an OpenAPI Specification @@ -459,7 +461,7 @@ Follow these steps to provide the bearer token with `FUZZAPI_OVERRIDES_ENV`: ``` 1. To validate that authentication is working, run an API fuzzing test and review the fuzzing logs - and the test API's application logs. + and the test API's application logs. See the [overrides section](#overrides) for more information about override commands. ##### Token generated at test runtime @@ -493,7 +495,7 @@ variables: FUZZAPI_PROFILE: Quick FUZZAPI_OPENAPI: test-api-specification.json FUZZAPI_TARGET_URL: http://test-deployment/ - FUZZAPI_OVERRIDES_FILE: output/api-fuzzing-overrides.json + FUZZAPI_OVERRIDES_FILE: api-fuzzing-overrides.json ``` To validate that authentication is working, run an API fuzzing test and review the fuzzing logs and @@ -535,7 +537,7 @@ variables: FUZZAPI_PROFILE: Quick-10 FUZZAPI_OPENAPI: test-api-specification.json FUZZAPI_TARGET_URL: http://test-deployment/ - FUZZAPI_OVERRIDES_FILE: output/api-fuzzing-overrides.json + FUZZAPI_OVERRIDES_FILE: api-fuzzing-overrides.json FUZZAPI_OVERRIDES_CMD: renew_token.py FUZZAPI_OVERRIDES_INTERVAL: 300 ``` @@ -575,6 +577,9 @@ profile increases as the number of tests increases. |[`FUZZAPI_OVERRIDES_FILE`](#overrides) | Path to a JSON file containing overrides. | |[`FUZZAPI_OVERRIDES_ENV`](#overrides) | JSON string containing headers to override. | |[`FUZZAPI_OVERRIDES_CMD`](#overrides) | Overrides command. | +|[`FUZZAPI_OVERRIDES_CMD_VERBOSE`](#overrides) | When set to any value. It shows overrides command output as part of the job output. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334578) in GitLab 14.8. | +|`FUZZAPI_PRE_SCRIPT` | Run user command or script before scan session starts. | +|`FUZZAPI_POST_SCRIPT` | Run user command or script after scan session has finished. | |[`FUZZAPI_OVERRIDES_INTERVAL`](#overrides) | How often to run overrides command in seconds. Defaults to `0` (once). | |[`FUZZAPI_HTTP_USERNAME`](#http-basic-authentication) | Username for HTTP authentication. | |[`FUZZAPI_HTTP_PASSWORD`](#http-basic-authentication) | Password for HTTP authentication. | @@ -754,7 +759,7 @@ variables: FUZZAPI_PROFILE: Quick FUZZAPI_OPENAPI: test-api-specification.json FUZZAPI_TARGET_URL: http://test-deployment/ - FUZZAPI_OVERRIDES_FILE: output/api-fuzzing-overrides.json + FUZZAPI_OVERRIDES_FILE: api-fuzzing-overrides.json ``` #### Using a CI/CD variable @@ -799,16 +804,181 @@ variables: If the value must be generated or regenerated on expiration, you can provide a program or script for the API fuzzer to execute on a specified interval. The provided script runs in an Alpine Linux -container that has Python 3 and Bash installed. If the Python script requires additional packages, -it must detect this and install the packages at runtime. The script creates the overrides JSON file -as defined above. +container that has Python 3 and Bash installed. + +You have to set the environment variable `FUZZAPI_OVERRIDES_CMD` to the program or script you would like +to execute. The provided command creates the overrides JSON file as defined previously. + +You might want to install other scripting runtimes like NodeJS or Ruby, or maybe you need to install a dependency for +your overrides command. In this case, we recommend setting the `FUZZAPI_PRE_SCRIPT` to the file path of a script which +provides those prerequisites. The script provided by `FUZZAPI_PRE_SCRIPT` is executed once, before the analyzer starts. + +See the [Alpine Linux package management](https://wiki.alpinelinux.org/wiki/Alpine_Linux_package_management) +page for information about installing Alpine Linux packages. You must provide three CI/CD variables, each set for correct operation: - `FUZZAPI_OVERRIDES_FILE`: File generated by the provided command. -- `FUZZAPI_OVERRIDES_CMD`: Command to generate JSON file. +- `FUZZAPI_OVERRIDES_CMD`: Overrides command in charge of generating the overrides JSON file periodically. - `FUZZAPI_OVERRIDES_INTERVAL`: Interval in seconds to run command. +Optionally: + +- `FUZZAPI_PRE_SCRIPT`: Script to install runtimes or dependencies before the analyzer starts. + +```yaml +stages: + - fuzz + +include: + - template: API-Fuzzing.gitlab-ci.yml + +variables: + FUZZAPI_PROFILE: Quick + FUZZAPI_OPENAPI: test-api-specification.json + FUZZAPI_TARGET_URL: http://test-deployment/ + FUZZAPI_OVERRIDES_FILE: api-fuzzing-overrides.json + FUZZAPI_OVERRIDES_CMD: renew_token.py + FUZZAPI_OVERRIDES_INTERVAL: 300 +``` + +#### Debugging overrides + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334578) in GitLab 14.8. + +By default the output of the overrides command is hidden. If the overrides command returns a non zero exit code, the command is displayed as part of your job output. Optionally, you can set the variable `FUZZAPI_OVERRIDES_CMD_VERBOSE` to any value in order to display overrides command output as it is generated. This is useful when testing your overrides script, but should be disabled afterwards as it slows down testing. + +It is also possible to write messages from your script to a log file that is collected when the job completes or fails. The log file must be created in a specific location and follow a naming convention. + +Adding some basic logging to your overrides script is useful in case the script fails unexpectedly during normal running of the job. The log file is automatically included as an artifact of the job, allowing you to download it after the job has finished. + +Following our example, we provided `renew_token.py` in the environmental variable `FUZZAPI_OVERRIDES_CMD`. Please notice two things in the script: + +- Log file is saved in the location indicated by the environment variable `CI_PROJECT_DIR`. +- Log file name should match `gl-*.log`. + +```python +#!/usr/bin/env python + +# Example of an overrides command + +# Override commands can update the overrides json file +# with new values to be used. This is a great way to +# update an authentication token that will expire +# during testing. + +import logging +import json +import os +import requests +import backoff + +# [1] Store log file in directory indicated by env var CI_PROJECT_DIR +working_directory = os.environ['CI_PROJECT_DIR'] + +# [2] File name should match the pattern: gl-*.log +log_file_path = os.path.join(working_directory, 'gl-user-overrides.log') + +# Set up logger +logging.basicConfig(filename=log_file_path, level=logging.DEBUG) + +# Use `backoff` decorator to retry in case of transient errors. +@backoff.on_exception(backoff.expo, + (requests.exceptions.Timeout, + requests.exceptions.ConnectionError), + max_time=30) +def get_auth_response(): + return requests.get('https://authorization.service/api/get_api_token', auth=(os.environ['AUTH_USER'], os.environ['AUTH_PWD'])) + + +# In our example, access token is retrieved from a given endpoint +try: + + # Performs a http request, response sample: + # { "Token" : "b5638ae7-6e77-4585-b035-7d9de2e3f6b3" } + response = get_auth_response() + + # Check that the request is successful. may raise `requests.exceptions.HTTPError` + response.raise_for_status() + + # Gets JSON data + response_body = response.json() + +# If needed specific exceptions can be caught +# requests.ConnectionError : A network connection error problem occurred +# requests.HTTPError : HTTP request returned an unsuccessful status code. [Response.raise_for_status()] +# requests.ConnectTimeout : The request timed out while trying to connect to the remote server +# requests.ReadTimeout : The server did not send any data in the allotted amount of time. +# requests.TooManyRedirects : The request exceeds the configured number of maximum redirections +# requests.exceptions.RequestException : All exceptions that related to Requests +except requests.exceptions.RequestException as requests_error: + # logs exceptions related to `Requests` + logging.error(f'Error, failed while performing HTTP request. Error message: {requests_error}') + raise +except requests.exceptions.JSONDecodeError as json_decode_error: + # logs errors related decoding JSON response + logging.error(f'Error, failed while decoding JSON response. Error message: {json_decode_error}') + raise +except Exception as e: + # logs any other error + logging.error(f'Error, unknown error while retrieving access token. Error message: {e}') + raise + +# computes object that holds overrides file content. +# It uses data fetched from request +overrides_data = { + "headers": { + "Authorization": f"Token {response_body['Token']}" + } +} + +# log entry informing about the file override computation +overrides_file_path = os.path.join( + working_directory, "api-fuzzing-overrides.json") +logging.info("Creating overrides file: %s" % overrides_file_path) + +# attempts to overwrite the file +try: + if os.path.exists(overrides_file_path): + os.unlink(overrides_file_path) + + # overwrites the file with our updated dictionary + with open(overrides_file_path, "wb+") as fd: + fd.write(json.dumps(overrides_data).encode('utf-8')) +except Exception as e: + # logs any other error + logging.error(f'Error, unkown error when overwritng file {overrides_file_path}. Error message: {e}') + raise + +# logs informing override has finished successfully +logging.info("Override file has been updated") + +# end +``` + +In the overrides command example, the Python script depends on the `backoff` library. To make sure the library is installed before executing the Python script, the `FUZZAPI_PRE_SCRIPT` is set to a script that will install the dependencies of your overrides command. +As for example, the following script `user-pre-scan-set-up.sh`: + +```shell +#!/bin/bash + +# user-pre-scan-set-up.sh +# Ensures python dependencies are installed + +echo "**** install python dependencies ****" + +python3 -m ensurepip +pip3 install --no-cache --upgrade \ + pip \ + backoff + +echo "**** python dependencies installed ****" + +# end +``` + +You have to update your configuration to set the `FUZZAPI_PRE_SCRIPT` to our new `user-pre-scan-set-up.sh` script. For example: + ```yaml stages: - fuzz @@ -820,11 +990,14 @@ variables: FUZZAPI_PROFILE: Quick FUZZAPI_OPENAPI: test-api-specification.json FUZZAPI_TARGET_URL: http://test-deployment/ - FUZZAPI_OVERRIDES_FILE: output/api-fuzzing-overrides.json + FUZZAPI_PRE_SCRIPT: user-pre-scan-set-up.sh + FUZZAPI_OVERRIDES_FILE: api-fuzzing-overrides.json FUZZAPI_OVERRIDES_CMD: renew_token.py FUZZAPI_OVERRIDES_INTERVAL: 300 ``` +In the previous sample, you could use the script `user-pre-scan-set-up.sh` to also install new runtimes or applications that later on you could use in your overrides command. + ### Exclude Paths > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211892) in GitLab 14.0. diff --git a/doc/user/application_security/cluster_image_scanning/index.md b/doc/user/application_security/cluster_image_scanning/index.md index 5f2dd626526..0db9af7a0d3 100644 --- a/doc/user/application_security/cluster_image_scanning/index.md +++ b/doc/user/application_security/cluster_image_scanning/index.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/) in GitLab 14.1. WARNING: -This analyzer is in [Alpha](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha) +This analyzer is in [Alpha](../../../policy/alpha-beta-support.md#alpha-features) and is unstable. The JSON report and CI/CD configuration may be subject to change or breakage across GitLab releases. @@ -301,7 +301,9 @@ the security vulnerabilities in your groups, projects, and pipelines. ## Interacting with the vulnerabilities -After a vulnerability is found, you can [address it](../vulnerabilities/index.md). +After you find a vulnerability, you can address it in the [vulnerability report](../vulnerabilities/index.md) +or the [GitLab Agent's](../../clusters/agent/install/index.md#view-vulnerabilities-in-cluster-images) +details section. ## Troubleshooting diff --git a/doc/user/application_security/configuration/index.md b/doc/user/application_security/configuration/index.md index cdcd334dba6..430f8e1a2a2 100644 --- a/doc/user/application_security/configuration/index.md +++ b/doc/user/application_security/configuration/index.md @@ -58,11 +58,11 @@ You can configure the following security controls: - [API Fuzzing](../api_fuzzing/index.md) - Select **Enable API Fuzzing** to use API Fuzzing for the current project. For more details, read [API Fuzzing](../../../user/application_security/api_fuzzing/index.md#enable-web-api-fuzzing). - [Coverage Fuzzing](../coverage_fuzzing/index.md) - - Can be configured with `.gitlab-ci.yml`. For more details, read [Coverage Fuzzing](../../../user/application_security/coverage_fuzzing/index.md#configuration). + - Can be configured with `.gitlab-ci.yml`. For more details, read [Coverage Fuzzing](../../../user/application_security/coverage_fuzzing/index.md#enable-coverage-guided-fuzz-testing). ## Compliance **(ULTIMATE)** You can configure the following security controls: - [License Compliance](../../../user/compliance/license_compliance/index.md) - - Can be configured with `.gitlab-ci.yml`. For more details, read [License Compliance](../../../user/compliance/license_compliance/index.md#configuration). + - Can be configured with `.gitlab-ci.yml`. For more details, read [License Compliance](../../../user/compliance/license_compliance/index.md#enable-license-compliance). diff --git a/doc/user/application_security/container_scanning/index.md b/doc/user/application_security/container_scanning/index.md index 5a2dd5eb54f..08a8c46cc72 100644 --- a/doc/user/application_security/container_scanning/index.md +++ b/doc/user/application_security/container_scanning/index.md @@ -222,6 +222,7 @@ You can [configure](#customizing-the-container-scanning-settings) analyzers by u | `CS_DISABLE_DEPENDENCY_LIST` | `"false"` | Disable Dependency Scanning for packages installed in the scanned image. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345434) in GitLab 14.6. | All | | `CS_DISABLE_LANGUAGE_VULNERABILITY_SCAN` | `"true"` | Disable scanning for language-specific packages installed in the scanned image. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345434) in GitLab 14.6. | All | | `CS_DOCKER_INSECURE` | `"false"` | Allow access to secure Docker registries using HTTPS without validating the certificates. | All | +| `CS_IGNORE_UNFIXED` | `"false"` | Ignore vulnerabilities that are not fixed. | All | | `CS_REGISTRY_INSECURE` | `"false"` | Allow access to insecure registries (HTTP only). Should only be set to `true` when testing the image locally. Works with all scanners, but the registry must listen on port `80/tcp` for Trivy to work. | All | | `CS_SEVERITY_THRESHOLD` | `UNKNOWN` | Severity level threshold. The scanner outputs vulnerabilities with severity level higher than or equal to this threshold. Supported levels are Unknown, Low, Medium, High, and Critical. | Trivy | | `DOCKER_IMAGE` | `$CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG` | The Docker image to be scanned. If set, this variable overrides the `$CI_APPLICATION_REPOSITORY` and `$CI_APPLICATION_TAG` variables. | All | @@ -513,7 +514,7 @@ registry.gitlab.com/security-products/container-scanning/trivy:4 The process for importing Docker images into a local offline Docker registry depends on **your network security policy**. Please consult your IT staff to find an accepted and approved process by which you can import or temporarily access external resources. These scanners -are [periodically updated](../vulnerabilities/index.md#vulnerability-scanner-maintenance), +are [periodically updated](../index.md#vulnerability-scanner-maintenance), and you may be able to make occasional updates on your own. For more information, see [the specific steps on how to update an image with a pipeline](#automating-container-scanning-vulnerability-database-updates-with-a-pipeline). @@ -728,8 +729,16 @@ the security vulnerabilities in your groups, projects and pipelines. ## Vulnerabilities database update -If you use container scanning and want more information about the vulnerabilities database update, -see the [maintenance table](../vulnerabilities/index.md#vulnerability-scanner-maintenance). +All analyzer images are [updated daily](https://gitlab.com/gitlab-org/security-products/analyzers/container-scanning/-/blob/master/README.md#image-updates). + +The images include the latest advisory database available for their respective scanner. Each +scanner includes data from multiple sources: + +- [Grype](https://github.com/anchore/grype#grypes-database). +- [Trivy](https://aquasecurity.github.io/trivy/latest/vulnerability/detection/data-source/). + +Database update information for other analyzers is available in the +[maintenance table](../index.md#vulnerability-scanner-maintenance). ## Interacting with the vulnerabilities diff --git a/doc/user/application_security/coverage_fuzzing/index.md b/doc/user/application_security/coverage_fuzzing/index.md index 89b4cdcc34d..290d4a06dcc 100644 --- a/doc/user/application_security/coverage_fuzzing/index.md +++ b/doc/user/application_security/coverage_fuzzing/index.md @@ -2,7 +2,6 @@ stage: Secure group: Dynamic Analysis info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference, howto --- # Coverage-guided fuzz testing **(ULTIMATE)** @@ -23,8 +22,14 @@ The fuzz testing process: 1. Compiles the target application. 1. Runs the instrumented application, using the `gitlab-cov-fuzz` tool. 1. Parses and analyzes the exception information output by the fuzzer. -1. Downloads the [corpus](../terminology/index.md#corpus) and crash events from previous pipelines. +1. Downloads the [corpus](../terminology/index.md#corpus) from either: + - The previous pipelines. + - If `COVFUZZ_USE_REGISTRY` is set to `true`, the [corpus registry](#corpus-registry). +1. Downloads crash events from previous pipeline. 1. Outputs the parsed crash events and data to the `gl-coverage-fuzzing-report.json` file. +1. Updates the corpus, either: + - In the job's pipeline. + - If `COVFUZZ_USE_REGISTRY` is set to `true`, in the corpus registry. The results of the coverage-guided fuzz testing are available in the CI/CD pipeline. @@ -44,9 +49,20 @@ You can use the following fuzzing engines to test the specified languages. | Python | [`pythonfuzz`](https://gitlab.com/gitlab-org/security-products/analyzers/fuzzers/pythonfuzz) | [pythonfuzz-fuzzing-example](https://gitlab.com/gitlab-org/security-products/demos/coverage-fuzzing/pythonfuzz-fuzzing-example) | | AFL (any language that works on top of AFL) | [AFL](https://lcamtuf.coredump.cx/afl/) | [afl-fuzzing-example](https://gitlab.com/gitlab-org/security-products/demos/coverage-fuzzing/afl-fuzzing-example) | -## Configuration +## Confirm status of coverage-guided fuzz testing -To enable coverage-guided fuzz testing, edit the `.gitlab-ci.yml` file: +To confirm the status of coverage-guided fuzz testing: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Security & Compliance > Configuration**. +1. In the **Coverage Fuzzing** section the status is: + - **Not configured** + - **Enabled** + - A prompt to upgrade to GitLab Ultimate. + +## Enable coverage-guided fuzz testing + +To enable coverage-guided fuzz testing, edit `.gitlab-ci.yml`: 1. Add the `fuzz` stage to the list of stages. @@ -99,10 +115,13 @@ Use the following variables to configure coverage-guided fuzz testing in your CI | CI/CD variable | Description | |---------------------------|---------------------------------------------------------------------------------| -| `COVFUZZ_ADDITIONAL_ARGS` | Arguments passed to `gitlab-cov-fuzz`. Used to customize the behavior of the underlying fuzzing engine. Read the fuzzing engine's documentation for a complete list of arguments. | +| `COVFUZZ_ADDITIONAL_ARGS` | Arguments passed to `gitlab-cov-fuzz`. Used to customize the behavior of the underlying fuzzing engine. Read the fuzzing engine's documentation for a complete list of arguments. | | `COVFUZZ_BRANCH` | The branch on which long-running fuzzing jobs are to be run. On all other branches, only fuzzing regression tests are run. Default: Repository's default branch. | | `COVFUZZ_SEED_CORPUS` | Path to a seed corpus directory. Default: empty. | | `COVFUZZ_URL_PREFIX` | Path to the `gitlab-cov-fuzz` repository cloned for use with an offline environment. You should only change this value when using an offline environment. Default: `https://gitlab.com/gitlab-org/security-products/analyzers/gitlab-cov-fuzz/-/raw`. | +| `COVFUZZ_USE_REGISTRY` | Set to `true` to have the corpus stored in the GitLab corpus registry. The variables `COVFUZZ_CORPUS_NAME` and `COVFUZZ_GITLAB_TOKEN` are required if this variable is set to `true`. Default: `false`. [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5017) in GitLab 14.8. | +| `COVFUZZ_CORPUS_NAME` | Name of the corpus to be used in the job. [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5017) in GitLab 14.8. | +| `COVFUZZ_GITLAB_TOKEN` | Environment variable configured with [Personal Access Token](../../../user/profile/personal_access_tokens.md#create-a-personal-access-token) with API read/write access. [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5017) in GitLab 14.8. | #### Seed corpus @@ -123,9 +142,93 @@ Each fuzzing step outputs these artifacts: You can download the JSON report file from the CI/CD pipelines page. For more information, see [Downloading artifacts](../../../ci/pipelines/job_artifacts.md#download-job-artifacts). -### Coverage-guided fuzz testing report +## Corpus registry + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5017) in GitLab 14.8. + +FLAG: +On self-managed GitLab, by default this feature is available. To hide the feature, ask an +administrator to [disable the feature flags](../../../administration/feature_flags.md) named +`corpus_management` and `corpus_management_ui`. On GitLab.com, this feature is available. + +The corpus registry is a library of corpuses. Corpuses in a project's registry are available to +all jobs in that project. A project-wide registry is a more efficient way to manage corpuses than +the default option of one corpus per job. + +The corpus registry uses the package registry to store the project's corpuses. Corpuses stored in +the registry are hidden to ensure data integrity. + +In the GitLab UI, with corpus management you can: + +- View details of the corpus registry. +- Download a corpus. +- Delete a corpus. +- Create a new corpus. + +When you download a corpus, the file is named `artifacts.zip`, regardless of the filename used when +the corpus was initially uploaded. This file contains only the corpus, which is different to the +artifacts files you can download from the CI/CD pipeline. + +### View details of the corpus registry + +To view details of the corpus registry: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Security & Compliance > Configuration**. +1. In the **Coverage Fuzzing** section, select **Manage corpus**. + +### Create a corpus in the corpus registry + +To create a corpus in the corpus registry, either: + +- Create a corpus in a pipeline +- Upload an existing corpus file + +#### Create a corpus in a pipeline + +To create a corpus in a pipeline: -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/220062) in GitLab 13.3 as an [Alpha feature](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha). +1. In the `.gitlab-ci.yml` file, edit the `my_fuzz_target` job. +1. Set the following variables: + - Set `COVFUZZ_USE_REGISTRY` to `true`. + - Set `COVFUZZ_CORPUS_NAME` to name the corpus. + - Set `COVFUZZ_GITLAB_TOKEN` to the value of the personal access token. + +After the `my_fuzz_target` job runs, the corpus is stored in the corpus registry, with the name +provided by the `COVFUZZ_CORPUS_NAME` variable. The corpus is updated on every pipeline run. + +#### Upload a corpus file + +To upload an existing corpus file: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Security & Compliance > Configuration**. +1. In the **Coverage Fuzzing** section, select **Manage corpus**. +1. Select **New corpus**. +1. Complete the fields. +1. Select **Upload file**. +1. Select **Add**. + +You can now reference the corpus in the `.gitlab-ci.yml` file. Ensure the value used in the +`COVFUZZ_CORPUS_NAME` variable matches exactly the name given to the uploaded corpus file. + +### Use a corpus stored in the corpus registry + +To use a corpus stored in the corpus registry, you must reference it by its name. To confirm the +name of the relevant corpus, view details of the corpus registry. + +Prerequisites: + +- [Enable coverage-guide fuzz testing](#enable-coverage-guided-fuzz-testing) in the project. + +1. Set the following variables in the `.gitlab-ci.yml` file: + - Set `COVFUZZ_USE_REGISTRY` to `true`. + - Set `COVFUZZ_CORPUS_NAME` to the name of the corpus. + - Set `COVFUZZ_GITLAB_TOKEN` to the value of the personal access token. + +## Coverage-guided fuzz testing report + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/220062) in GitLab 13.3 as an [Alpha feature](../../../policy/alpha-beta-support.md#alpha-features). For detailed information about the `gl-coverage-fuzzing-report.json` file's format, read the [schema](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/coverage-fuzzing-report-format.json). @@ -161,9 +264,9 @@ Example coverage-guided fuzzing report: ## Duration of coverage-guided fuzz testing -The available durations for coverage-guided fuzz testing are: 10 minutes (default) and 60 minutes. +The available durations for coverage-guided fuzz testing are: -- 10-minute duration: Recommended for the default branch. +- 10-minute duration (default): Recommended for the default branch. - 60-minute duration: Recommended for the development branch and merge requests. The longer duration provides greater coverage. In the `COVFUZZ_ADDITIONAL_ARGS` variable set the value `--regression=true`. @@ -256,3 +359,16 @@ vulnerability: engines listed in [Supported fuzzing engines and languages](#supported-fuzzing-engines-and-languages). <!-- vale gitlab.Acronyms = YES --> + +## Troubleshooting + +### Error "Unable to extract corpus folder from artifacts zip file" + +If you see this error message, and `COVFUZZ_USE_REGISTRY` is set to `true`, ensure that the uploaded +corpus file extracts into a folder named `corpus`. + +### Error "400 Bad request - Duplicate package is not allowed" + +If you see this error message when running the fuzzing job with `COVFUZZ_USE_REGISTRY` set to `true`, +ensure that duplicates are allowed. For more details, see +[duplicate Generic packages](../../packages/generic_packages/#do-not-allow-duplicate-generic-packages). diff --git a/doc/user/application_security/dast/browser_based.md b/doc/user/application_security/dast/browser_based.md index 5d1e57553f4..4cde1847419 100644 --- a/doc/user/application_security/dast/browser_based.md +++ b/doc/user/application_security/dast/browser_based.md @@ -10,7 +10,7 @@ type: reference, howto > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/323423) in GitLab 13.12. WARNING: -This product is in an early-access stage and is considered a [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta) feature. +This product is in an early-access stage and is considered a [beta](../../../policy/alpha-beta-support.md#beta-features) feature. GitLab DAST's new browser-based crawler is a crawl engine built by GitLab to test Single Page Applications (SPAs) and traditional web applications. Due to the reliance of modern web applications on JavaScript, handling SPAs or applications that are dependent on JavaScript is paramount to ensuring proper coverage of an application for Dynamic Application Security Testing (DAST). diff --git a/doc/user/application_security/dast/checks/1004.1.md b/doc/user/application_security/dast/checks/1004.1.md index dfbc600b05b..9626973eb36 100644 --- a/doc/user/application_security/dast/checks/1004.1.md +++ b/doc/user/application_security/dast/checks/1004.1.md @@ -4,7 +4,7 @@ group: Dynamic Analysis info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Sensitive cookie without `HttpOnly` attribute +# Sensitive cookie without HttpOnly attribute ## Description diff --git a/doc/user/application_security/dast/checks/16.2.md b/doc/user/application_security/dast/checks/16.2.md index 95461e8677d..484460b6642 100644 --- a/doc/user/application_security/dast/checks/16.2.md +++ b/doc/user/application_security/dast/checks/16.2.md @@ -38,7 +38,7 @@ the `Server` header. ## Links -- [cwe](https://cwe.mitre.org/data/definitions/16.html) +- [CWE](https://cwe.mitre.org/data/definitions/16.html) - [Apache ServerTokens](https://blog.mozilla.org/security/2016/08/26/mitigating-mime-confusion-attacks-in-firefox/) - [NGINX server_tokens](https://nginx.org/en/docs/http/ngx_http_core_module.html#server_tokens) - [IIS 10 Remove Server Header](https://docs.microsoft.com/en-us/iis/configuration/system.webserver/security/requestfiltering/#attributes) diff --git a/doc/user/application_security/dast/checks/16.3.md b/doc/user/application_security/dast/checks/16.3.md index 6f80a2a32c6..e4fc2468dae 100644 --- a/doc/user/application_security/dast/checks/16.3.md +++ b/doc/user/application_security/dast/checks/16.3.md @@ -32,4 +32,4 @@ information from the `X-Powered-By` header. ## Links - [CWE](https://cwe.mitre.org/data/definitions/16.html) -- [PHP `expose_php`](https://www.php.net/manual/en/ini.core.php#ini.expose-php) +- [PHP expose_php](https://www.php.net/manual/en/ini.core.php#ini.expose-php) diff --git a/doc/user/application_security/dast/checks/200.1.md b/doc/user/application_security/dast/checks/200.1.md new file mode 100644 index 00000000000..98a482b4a0f --- /dev/null +++ b/doc/user/application_security/dast/checks/200.1.md @@ -0,0 +1,29 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of sensitive information to an unauthorized actor (private IP address) + +## Description + +A private RFC 1918 was identified in the target application. Public facing websites should not be issuing +requests to private IP Addresses. Attackers attempting to execute subsequent attacks, such as Server-Side +Request Forgery (SSRF), may be able to use this information to identify additional internal targets. + +## Remediation + +Identify the resource that is incorrectly specifying an internal IP address and replace it with it's public +facing version, or remove the reference from the target application. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 200.1 | true | 200 | Passive | Low | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/200.html) +- [RFC](https://datatracker.ietf.org/doc/html/rfc1918) diff --git a/doc/user/application_security/dast/checks/548.1.md b/doc/user/application_security/dast/checks/548.1.md new file mode 100644 index 00000000000..94f747739c5 --- /dev/null +++ b/doc/user/application_security/dast/checks/548.1.md @@ -0,0 +1,45 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of information through directory listing + +## Description + +The target web server is configured to list the contents of directories that do not contain an index file +such as `index.html`. This could lead to accidental exposure of sensitive information, or give an attacker +details on how filenames and directories are structured and stored. + +## Remediation + +Directory indexing should be disabled. + +Apache: +For Apache based web sites, ensure all `<Directory>` definitions have `Options -Indexes` configured in the +`apache2.conf` or `httpd.conf` configuration file. + +NGINX: +For NGINX based websites, ensure all `location` definitions have the `autoindex off` directive set in the +`nginx.conf` file. + +IIS: +For IIS based websites version 7.0 and above you can use the `<directoryBrowse enabled="false" />` element +in the `applicationHost.config` or `Web.config` files. + +For all other server types, please consult your product's documentation on how to disable directory +indexing. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 548.1 | false | 548 | Passive | Low | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/598.html) +- [Apache Options](https://httpd.apache.org/docs/2.4/mod/core.html#options) +- [NGINX autoindex](https://nginx.org/en/docs/http/ngx_http_autoindex_module.html) +- [IIS directoryBrowse element](https://docs.microsoft.com/en-us/iis/configuration/system.webserver/directorybrowse) diff --git a/doc/user/application_security/dast/checks/614.1.md b/doc/user/application_security/dast/checks/614.1.md index 46f7f61b0c7..ec68ce33529 100644 --- a/doc/user/application_security/dast/checks/614.1.md +++ b/doc/user/application_security/dast/checks/614.1.md @@ -4,7 +4,7 @@ group: Dynamic Analysis info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Sensitive cookie without `Secure` attribute +# Sensitive cookie without Secure attribute ## Description @@ -35,6 +35,6 @@ Set-Cookie: {cookie_name}=<random secure value>; Secure ## Links -- [owasp](https://owasp.org/www-community/controls/SecureCookieAttribute) +- [OWASP](https://owasp.org/www-community/controls/SecureCookieAttribute) - [CWE](https://cwe.mitre.org/data/definitions/614.html) - [Mozilla MDN](https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies#restrict_access_to_cookies) diff --git a/doc/user/application_security/dast/checks/index.md b/doc/user/application_security/dast/checks/index.md index a3b89e09751..97224554723 100644 --- a/doc/user/application_security/dast/checks/index.md +++ b/doc/user/application_security/dast/checks/index.md @@ -10,12 +10,14 @@ The [DAST browser-based crawler](../browser_based.md) provides a number of vulne | ID | Check | Severity | Type | |:---|:------|:---------|:-----| -| [1004.1](1004.1.md) | Sensitive cookie without `HttpOnly` attribute | Low | Passive | +| [1004.1](1004.1.md) | Sensitive cookie without HttpOnly attribute | Low | Passive | | [16.1](16.1.md) | Missing Content-Type header | Low | Passive | | [16.2](16.2.md) | Server header exposes version information | Low | Passive | | [16.3](16.3.md) | X-Powered-By header exposes version information | Low | Passive | | [16.4](16.4.md) | X-Backend-Server header exposes server information | Info | Passive | | [16.5](16.5.md) | AspNet header exposes version information | Low | Passive | | [16.6](16.6.md) | AspNetMvc header exposes version information | Low | Passive | -| [614.1](614.1.md) | Sensitive cookie without `Secure` attribute | Low | Passive | +| [200.1](200.1.md) | Exposure of sensitive information to an unauthorized actor (private IP address) | Low | Passive | +| [548.1](548.1.md) | Exposure of information through directory listing | Low | Passive | +| [614.1](614.1.md) | Sensitive cookie without Secure attribute | Low | Passive | | [693.1](693.1.md) | Missing X-Content-Type-Options: nosniff | Low | Passive | diff --git a/doc/user/application_security/dast/index.md b/doc/user/application_security/dast/index.md index aeaa93f4a85..0865cc10691 100644 --- a/doc/user/application_security/dast/index.md +++ b/doc/user/application_security/dast/index.md @@ -53,7 +53,7 @@ results. On failure, the analyzer outputs an - [GitLab Runner](../../../ci/runners/index.md) available, with the [`docker` executor](https://docs.gitlab.com/runner/executors/docker.html). - Target application deployed. For more details, read [Deployment options](#deployment-options). -- DAST runs in the `test` stage, which is available by default. If you redefine the stages in the `.gitlab-ci.yml` file, the `test` stage is required. +- DAST runs in the `dast` stage, which must be added manually to your `.gitlab-ci.yml`. ### Deployment options @@ -588,6 +588,28 @@ Using the [`DAST_MASK_HTTP_HEADERS` CI/CD variable](#available-cicd-variables), headers whose values you want masked. For details on how to mask headers, see [Customizing the DAST settings](#customize-dast-settings). +#### Use Mutual TLS + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299596) in GitLab 14.8. + +Mutual TLS allows a target application server to verify that requests are from a known source. Browser-based scans do not support Mutual TLS. + +**Requirements** + +- Base64-encoded PKCS12 certificate +- Password of the base64-encoded PKCS12 certificate + +To enable Mutual TLS: + +1. If the PKCS12 certificate is not already base64-encoded, convert it to base64 encoding. For security reasons, we recommend encoding the certificate locally, **not** using a web-hosted conversion service. For example, to encode the certificate on either macOS or Linux: + + ```shell + base64 <path-to-pkcs12-certificate-file> + ``` + +1. Create a [masked variable](../../../ci/variables/index.md) named `DAST_PKCS12_CERTIFICATE_BASE64` and store the base64-encoded PKCS12 certificate's value in that variable. +1. Create a masked variable `DAST_PKCS12_PASSWORD` and store the PKCS12 certificate's password in that variable. + #### Available CI/CD variables These CI/CD variables are specific to DAST. They can be used to customize the behavior of DAST to your requirements. @@ -623,6 +645,8 @@ These CI/CD variables are specific to DAST. They can be used to customize the be | `DAST_PASSWORD_FIELD` <sup>1,2</sup> | string | The selector of password field at the sign-in HTML form. Example: `id:password` | | `DAST_PATHS` | string | Set to a comma-separated list of URLs for DAST to scan. For example, `/page1.html,/category1/page3.html,/page2.html`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214120) in GitLab 13.4. | | `DAST_PATHS_FILE` | string | The file path containing the paths within `DAST_WEBSITE` to scan. The file must be plain text with one path per line. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/258825) in GitLab 13.6. | +| `DAST_PKCS12_CERTIFICATE_BASE64` | string | The PKCS12 certificate used for sites that require Mutual TLS. Must be encoded as base64 text. | +| `DAST_PKCS12_PASSWORD` | string | The password of the certificate used in `DAST_PKCS12_CERTIFICATE_BASE64`. | | `DAST_REQUEST_HEADERS` <sup>1</sup> | string | Set to a comma-separated list of request header names and values. Headers are added to every request made by DAST. For example, `Cache-control: no-cache,User-Agent: DAST/1.0` | | `DAST_SKIP_TARGET_CHECK` | boolean | Set to `true` to prevent DAST from checking that the target is available before scanning. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229067) in GitLab 13.8. | | `DAST_SPIDER_MINS` <sup>1</sup> | number | The maximum duration of the spider scan in minutes. Set to `0` for unlimited. Default: One minute, or unlimited when the scan is a full scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | @@ -1172,7 +1196,7 @@ To edit an existing site profile: 1. Edit the fields then select **Save profile**. If a site profile is linked to a security policy, a user cannot edit the profile from this page. See -[Scan Execution Policies](../policies/index.md#scan-execution-policy-editor) +[Scan execution policies](../policies/scan-execution-policies.md) for more information. #### Delete a site profile @@ -1186,7 +1210,7 @@ To delete an existing site profile: 1. Select **Delete** to confirm the deletion. If a site profile is linked to a security policy, a user cannot delete the profile from this page. -See [Scan Execution Policies](../policies/index.md#scan-execution-policy-editor) +See [Scan execution policies](../policies/scan-execution-policies.md) for more information. #### Validate a site profile @@ -1329,7 +1353,7 @@ To edit a scanner profile: 1. Select **Save profile**. If a scanner profile is linked to a security policy, a user cannot edit the profile from this page. -See [Scan Execution Policies](../policies/index.md#scan-execution-policy-editor) +See [Scan execution policies](../policies/scan-execution-policies.md) for more information. #### Delete a scanner profile @@ -1343,7 +1367,7 @@ To delete a scanner profile: 1. Select **Delete**. If a scanner profile is linked to a security policy, a user cannot delete the profile from this -page. See [Scan Execution Policies](../policies/index.md#scan-execution-policy-editor) +page. See [Scan execution policies](../policies/scan-execution-policies.md) for more information. ## Auditing @@ -1365,7 +1389,7 @@ The JSON report artifacts are not a public API of DAST and their format is expec The DAST tool always emits a JSON report file called `gl-dast-report.json` and sample reports can be found in the -[DAST repository](https://gitlab.com/gitlab-org/security-products/dast/-/tree/master/test/end-to-end/expect). +[DAST repository](https://gitlab.com/gitlab-org/security-products/dast/-/tree/main/test/end-to-end/expect). ## Optimizing DAST diff --git a/doc/user/application_security/dast/run_dast_offline.md b/doc/user/application_security/dast/run_dast_offline.md index 86621d73524..63163167a6c 100644 --- a/doc/user/application_security/dast/run_dast_offline.md +++ b/doc/user/application_security/dast/run_dast_offline.md @@ -36,7 +36,7 @@ For DAST, import the following default DAST analyzer image from `registry.gitlab The process for importing Docker images into a local offline Docker registry depends on **your network security policy**. Please consult your IT staff to find an accepted and approved process by which external resources can be imported or temporarily accessed. -These scanners are [periodically updated](../vulnerabilities/index.md#vulnerability-scanner-maintenance) +These scanners are [periodically updated](../index.md#vulnerability-scanner-maintenance) with new definitions, and you may be able to make occasional updates on your own. For details on saving and transporting Docker images as a file, see Docker's documentation on diff --git a/doc/user/application_security/dast_api/index.md b/doc/user/application_security/dast_api/index.md index f1eba505589..cc20b49764f 100644 --- a/doc/user/application_security/dast_api/index.md +++ b/doc/user/application_security/dast_api/index.md @@ -69,8 +69,8 @@ starting in GitLab 14.0, GitLab will not check your repository's root for config ### OpenAPI Specification -> Support for OpenAPI Specification using YAML format was -> [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/330583) in GitLab 14.0. +> - Support for OpenAPI Specification using YAML format was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/330583) in GitLab 14.0. +> - Support to generate media type `application/xml` was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/327268) in GitLab 14.8. The [OpenAPI Specification](https://www.openapis.org/) (formerly the Swagger Specification) is an API description format for REST APIs. This section shows you how to configure API fuzzing using an OpenAPI Specification to provide information about the target API to test. @@ -82,6 +82,7 @@ the body generation is limited to these body types: - `application/x-www-form-urlencoded` - `multipart/form-data` - `application/json` +- `application/xml` Follow these steps to configure DAST API in GitLab with an OpenAPI specification: @@ -478,6 +479,9 @@ Follow these steps to provide the bearer token with `DAST_API_OVERRIDES_ENV`: `{"headers":{"Authorization":"Bearer dXNlcm5hbWU6cGFzc3dvcmQ="}}` (substitute your token). You can create CI/CD variables from the GitLab projects page at **Settings > CI/CD**, in the **Variables** section. + Due to the format of `TEST_API_BEARERAUTH` it's not possible to mask the variable. + To mask the token's value, you can create a second variable with the token value's, and define + `TEST_API_BEARERAUTH` with the value `{"headers":{"Authorization":"Bearer $MASKED_VARIABLE"}}`. 1. In your `.gitlab-ci.yml` file, set `DAST_API_OVERRIDES_ENV` to the variable you just created: @@ -529,7 +533,7 @@ variables: DAST_API_PROFILE: Quick DAST_API_OPENAPI: test-api-specification.json DAST_API_TARGET_URL: http://test-deployment/ - DAST_API_OVERRIDES_FILE: output/dast-api-overrides.json + DAST_API_OVERRIDES_FILE: dast-api-overrides.json ``` To validate that authentication is working, run an DAST API test and review the job logs and @@ -571,13 +575,12 @@ variables: DAST_API_PROFILE: Quick DAST_API_OPENAPI: test-api-specification.json DAST_API_TARGET_URL: http://test-deployment/ - DAST_API_OVERRIDES_FILE: output/dast-api-overrides.json + DAST_API_OVERRIDES_FILE: dast-api-overrides.json DAST_API_OVERRIDES_CMD: renew_token.py DAST_API_OVERRIDES_INTERVAL: 300 ``` -To validate that authentication is working, run an DAST API test and review the job logs and -the test API's application logs. +To validate that authentication is working, run an DAST API test and review the job logs and the test API's application logs. See the [overrides section](#overrides) for more information about override commands. ### Configuration files @@ -644,6 +647,9 @@ can be added, removed, and modified by creating a custom configuration. |[`DAST_API_OVERRIDES_FILE`](#overrides) | Path to a JSON file containing overrides. | |[`DAST_API_OVERRIDES_ENV`](#overrides) | JSON string containing headers to override. | |[`DAST_API_OVERRIDES_CMD`](#overrides) | Overrides command. | +|[`DAST_API_OVERRIDES_CMD_VERBOSE`](#overrides) | When set to any value. It shows overrides command output as part of the job output. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334578) in GitLab 14.6. | +|`DAST_API_PRE_SCRIPT` | Run user command or script before scan session starts. | +|`DAST_API_POST_SCRIPT` | Run user command or script after scan session has finished. | |[`DAST_API_OVERRIDES_INTERVAL`](#overrides) | How often to run overrides command in seconds. Defaults to `0` (once). | |[`DAST_API_HTTP_USERNAME`](#http-basic-authentication) | Username for HTTP authentication. | |[`DAST_API_HTTP_PASSWORD`](#http-basic-authentication) | Password for HTTP authentication. | @@ -825,7 +831,7 @@ variables: DAST_API_PROFILE: Quick DAST_API_OPENAPI: test-api-specification.json DAST_API_TARGET_URL: http://test-deployment/ - DAST_API_OVERRIDES_FILE: output/dast-api-overrides.json + DAST_API_OVERRIDES_FILE: dast-api-overrides.json ``` #### Using a CI/CD variable @@ -869,17 +875,29 @@ variables: #### Using a command If the value must be generated or regenerated on expiration, you can provide a program or script for -the DAST API scanner to execute on a specified interval. The provided script runs in an Alpine Linux -container that has Python 3 and Bash installed. If the Python script requires additional packages, -it must detect this and install the packages at runtime. The script creates the overrides JSON file -as defined above. +the DAST API scanner to execute on a specified interval. The provided command runs in an Alpine Linux +container that has Python 3 and Bash installed. + +You have to set the environment variable `DAST_API_OVERRIDES_CMD` to the program or script you would like +to execute. The provided command creates the overrides JSON file as defined previously. + +You might want to install other scripting runtimes like NodeJS or Ruby, or maybe you need to install a dependency for +your overrides command. In this case, we recommend setting the `DAST_API_PRE_SCRIPT` to the file path of a script which +provides those prerequisites. The script provided by `DAST_API_PRE_SCRIPT` is executed once, before the analyzer starts. + +See the [Alpine Linux package management](https://wiki.alpinelinux.org/wiki/Alpine_Linux_package_management) +page for information about installing Alpine Linux packages. You must provide three CI/CD variables, each set for correct operation: - `DAST_API_OVERRIDES_FILE`: File generated by the provided command. -- `DAST_API_OVERRIDES_CMD`: Command to generate JSON file. +- `DAST_API_OVERRIDES_CMD`: Overrides command in charge of generating the overrides JSON file periodically. - `DAST_API_OVERRIDES_INTERVAL`: Interval in seconds to run command. +Optionally: + +- `DAST_API_PRE_SCRIPT`: Script to install runtimes or dependencies before the scan starts. + ```yaml stages: - dast @@ -891,11 +909,167 @@ variables: DAST_API_PROFILE: Quick DAST_API_OPENAPI: test-api-specification.json DAST_API_TARGET_URL: http://test-deployment/ - DAST_API_OVERRIDES_FILE: output/dast-api-overrides.json + DAST_API_OVERRIDES_FILE: dast-api-overrides.json DAST_API_OVERRIDES_CMD: renew_token.py DAST_API_OVERRIDES_INTERVAL: 300 ``` +#### Debugging overrides + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334578) in GitLab 14.8. + +By default the output of the overrides command is hidden. If the overrides command returns a non zero exit code, the command is displayed as part of your job output. Optionally, you can set the variable `DAST_API_OVERRIDES_CMD_VERBOSE` to any value in order to display overrides command output as it is generated. This is useful when testing your overrides script, but should be disabled afterwards as it slows down testing. + +It is also possible to write messages from your script to a log file that is collected when the job completes or fails. The log file must be created in a specific location and following a naming convention. + +Adding some basic logging to your overrides script is useful in case the script fails unexpectedly during normal running of the job. The log file is automatically included as an artifact of the job, allowing you to download it after the job has finished. + +Following our example, we provided `renew_token.py` in the environment variable `DAST_API_OVERRIDES_CMD`. Please notice two things in the script: + +- Log file is saved in the location indicated by the environmental variable `CI_PROJECT_DIR`. +- Log file name should match `gl-*.log`. + +```python +#!/usr/bin/env python + +# Example of an overrides command + +# Override commands can update the overrides json file +# with new values to be used. This is a great way to +# update an authentication token that will expire +# during testing. + +import logging +import json +import os +import requests +import backoff + +# [1] Store log file in directory indicated by env var CI_PROJECT_DIR +working_directory = os.environ['CI_PROJECT_DIR'] + +# [2] File name should match the pattern: gl-*.log +log_file_path = os.path.join(working_directory, 'gl-user-overrides.log') + +# Set up logger +logging.basicConfig(filename=log_file_path, level=logging.DEBUG) + +# Use `backoff` decorator to retry in case of transient errors. +@backoff.on_exception(backoff.expo, + (requests.exceptions.Timeout, + requests.exceptions.ConnectionError), + max_time=30) +def get_auth_response(): + return requests.get('https://authorization.service/api/get_api_token', auth=(os.environ['AUTH_USER'], os.environ['AUTH_PWD'])) + +# In our example, access token is retrieved from a given endpoint +try: + + # Performs a http request, response sample: + # { "Token" : "b5638ae7-6e77-4585-b035-7d9de2e3f6b3" } + response = get_auth_response() + + # Check that the request is successful. may raise `requests.exceptions.HTTPError` + response.raise_for_status() + + # Gets JSON data + response_body = response.json() + +# If needed specific exceptions can be caught +# requests.ConnectionError : A network connection error problem occurred +# requests.HTTPError : HTTP request returned an unsuccessful status code. [Response.raise_for_status()] +# requests.ConnectTimeout : The request timed out while trying to connect to the remote server +# requests.ReadTimeout : The server did not send any data in the allotted amount of time. +# requests.TooManyRedirects : The request exceeds the configured number of maximum redirections +# requests.exceptions.RequestException : All exceptions that related to Requests +except requests.exceptions.RequestException as requests_error: + # logs exceptions related to `Requests` + logging.error(f'Error, failed while performing HTTP request. Error message: {requests_error}') + raise +except requests.exceptions.JSONDecodeError as json_decode_error: + # logs errors related decoding JSON response + logging.error(f'Error, failed while decoding JSON response. Error message: {json_decode_error}') + raise +except Exception as e: + # logs any other error + logging.error(f'Error, unknown error while retrieving access token. Error message: {e}') + raise + +# computes object that holds overrides file content. +# It uses data fetched from request +overrides_data = { + "headers": { + "Authorization": f"Token {response_body['Token']}" + } +} + +# log entry informing about the file override computation +# the location of the overrides json file is also CI_PROJECT_DIR +overrides_file_path = os.path.join( + working_directory, "dast-api-overrides.json") +logging.info("Creating overrides file: %s" % overrides_file_path) + +# attempts to overwrite the file +try: + if os.path.exists(overrides_file_path): + os.unlink(overrides_file_path) + + # overwrites the file with our updated dictionary + with open(overrides_file_path, "wb+") as fd: + fd.write(json.dumps(overrides_data).encode('utf-8')) +except Exception as e: + # logs any other error + logging.error(f'Error, unkown error when overwritng file {overrides_file_path}. Error message: {e}') + raise + +# logs informing override has finished successfully +logging.info("Override file has been updated") + +# end +``` + +In the overrides command example, the Python script depends on the `backoff` library. To make sure the library is installed before executing the Python script, the `DAST_API_PRE_SCRIPT` is set to a script that will install the dependencies of your overrides command. +As for example, the following script `user-pre-scan-set-up.sh` + +```shell +#!/bin/bash + +# user-pre-scan-set-up.sh +# Ensures python dependencies are installed + +echo "**** install python dependencies ****" + +python3 -m ensurepip +pip3 install --no-cache --upgrade \ + pip \ + backoff + +echo "**** python dependencies installed ****" + +# end +``` + +You have to update your configuration to set the `DAST_API_PRE_SCRIPT` to our new `user-pre-scan-set-up.sh` script. For example: + +```yaml +stages: + - dast + +include: + - template: DAST-API.gitlab-ci.yml + +variables: + DAST_API_PROFILE: Quick + DAST_API_OPENAPI: test-api-specification.json + DAST_API_TARGET_URL: http://test-deployment/ + DAST_API_PRE_SCRIPT: user-pre-scan-set-up.sh + DAST_API_OVERRIDES_FILE: dast-api-overrides.json + DAST_API_OVERRIDES_CMD: renew_token.py + DAST_API_OVERRIDES_INTERVAL: 300 +``` + +In the previous sample, you could use the script `user-pre-scan-set-up.sh` to also install new runtimes or applications that later on you could use in our overrides command. + ### Exclude Paths > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211892) in GitLab 14.0. diff --git a/doc/user/application_security/dependency_scanning/analyzers.md b/doc/user/application_security/dependency_scanning/analyzers.md index 1b502b306bb..551488c0dc0 100644 --- a/doc/user/application_security/dependency_scanning/analyzers.md +++ b/doc/user/application_security/dependency_scanning/analyzers.md @@ -29,11 +29,16 @@ Dependency Scanning supports the following official analyzers: The analyzers are published as Docker images, which Dependency Scanning uses to launch dedicated containers for each analysis. +The Dependency Scanning analyzers' current major version number is 2. + Dependency Scanning is pre-configured with a set of **default images** that are maintained by GitLab, but users can also integrate their own **custom images**. WARNING: -The `bundler-audit` analyzer is deprecated and will be removed in GitLab 15.0 since it duplicates the functionality of the `gemnasium` analyzer. For more information, read the [deprecation announcement](../../../update/deprecations.md#deprecation-of-bundler-audit-dependency-scanning-tool). +The `bundler-audit` analyzer is deprecated and will be removed in GitLab 15.0 since it duplicates the functionality of the `gemnasium` analyzer. For more information, read the [deprecation announcement](../../../update/deprecations.md#bundler-audit-dependency-scanning-tool). + +WARNING: +The `retire.js` analyzer is deprecated and will be removed in GitLab 15.0 since it duplicates the functionality of the `gemnasium` analyzer. For more information, read the [deprecation announcement](../../../update/deprecations.md#retire-js-dependency-scanning-tool). ## Official default analyzers diff --git a/doc/user/application_security/dependency_scanning/index.md b/doc/user/application_security/dependency_scanning/index.md index a8cc33d5545..a169b78a193 100644 --- a/doc/user/application_security/dependency_scanning/index.md +++ b/doc/user/application_security/dependency_scanning/index.md @@ -338,12 +338,12 @@ To support the following package managers, the GitLab analyzers proceed in two s | Package Manager | Preinstalled Versions | Tested Versions | | ------ | ------ | ------ | | Bundler | [2.1.4](https://gitlab.com/gitlab-org/security-products/analyzers/bundler-audit/-/blob/v2.11.3/Dockerfile#L15)<sup><b><a href="#exported-dependency-information-notes-1">1</a></b></sup> | [1.17.3](https://gitlab.com/gitlab-org/security-products/tests/ruby-bundler/-/blob/master/Gemfile.lock#L118), [2.1.4](https://gitlab.com/gitlab-org/security-products/tests/ruby-bundler/-/blob/bundler2-FREEZE/Gemfile.lock#L118) | -| sbt | [1.3.8](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.23.0/config/.tool-versions#L4) | [1.0.4](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/master/.gitlab-ci.yml#L263), [1.1.4](https://gitlab.com/gitlab-org/security-products/tests/scala-sbt-multiproject/-/blob/main/project/build.properties#L1), [1.1.6](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/master/.gitlab-ci.yml#L272), [1.2.8](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/master/.gitlab-ci.yml#L281), [1.3.12](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/master/.gitlab-ci.yml#L290), [1.4.6](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/master/.gitlab-ci.yml#L299) | +| sbt | [1.6.1](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.24.6/config/.tool-versions#L4) | [1.0.4](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.24.6/.gitlab-ci.yml#L330), [1.1.4](https://gitlab.com/gitlab-org/security-products/tests/scala-sbt-multiproject/-/blob/main/project/build.properties#L1), [1.1.6](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.24.6/.gitlab-ci.yml#L339), [1.2.8](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.24.6/.gitlab-ci.yml#L348), [1.3.12](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.24.6/.gitlab-ci.yml#L357), [1.4.6](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.24.6/.gitlab-ci.yml#L366), [1.6.1](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.24.6/.gitlab-ci.yml#L384) | | Maven | [3.6.3](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.23.0/config/.tool-versions#L3) | [3.6.3](https://gitlab.com/gitlab-org/security-products/tests/java-maven/-/blob/master/pom.xml#L3) | -| Gradle | [6.7.1](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.23.0/config/.tool-versions#L5) | [5.6.4](https://gitlab.com/gitlab-org/security-products/tests/java-gradle/-/blob/master/gradle/wrapper/gradle-wrapper.properties#L3), [6.5](https://gitlab.com/gitlab-org/security-products/tests/java-gradle/-/blob/java-14/gradle/wrapper/gradle-wrapper.properties#L3), [6.7-rc-1](https://gitlab.com/gitlab-org/security-products/tests/java-gradle/-/blob/java-15/gradle/wrapper/gradle-wrapper.properties#L3), [6.9](https://gitlab.com/gitlab-org/security-products/tests/java-gradle/-/blob/java-14-gradle-6-9/gradle/wrapper/gradle-wrapper.properties#L3), [7.0-rc-2](https://gitlab.com/gitlab-org/security-products/tests/java-gradle/-/blob/java-16/gradle/wrapper/gradle-wrapper.properties#L3) | -| setuptools | [50.3.2](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/v2.29.9/Dockerfile#L27) | [57.5.0](https://gitlab.com/gitlab-org/security-products/tests/python-setuptools/-/blob/main/setup.py) | +| Gradle | [6.7.1](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.23.0/config/.tool-versions#L5) | [5.6.4](https://gitlab.com/gitlab-org/security-products/tests/java-gradle/-/blob/master/gradle/wrapper/gradle-wrapper.properties#L3), [6.5](https://gitlab.com/gitlab-org/security-products/tests/java-gradle/-/blob/java-14/gradle/wrapper/gradle-wrapper.properties#L3), [6.7-rc-1](https://gitlab.com/gitlab-org/security-products/tests/java-gradle/-/blob/java-15/gradle/wrapper/gradle-wrapper.properties#L3), [6.9](https://gitlab.com/gitlab-org/security-products/tests/java-gradle/-/blob/java-14-gradle-6-9/gradle/wrapper/gradle-wrapper.properties#L3), [7.0-rc-2](https://gitlab.com/gitlab-org/security-products/tests/java-gradle/-/blob/java-16/gradle/wrapper/gradle-wrapper.properties#L3) | +| setuptools | [50.3.2](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/v2.29.9/Dockerfile#L27) | [57.5.0](https://gitlab.com/gitlab-org/security-products/tests/python-setuptools/-/blob/main/setup.py) | | pip | [20.2.4](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/v2.29.9/Dockerfile#L26) | [20.x](https://gitlab.com/gitlab-org/security-products/tests/python-pip/-/blob/master/requirements.txt) | -| Pipenv | [2018.11.26](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python/-/blob/v2.18.4/requirements.txt#L13) | [2018.11.26](https://gitlab.com/gitlab-org/security-products/tests/python-pipenv/-/blob/pipfile-lock-FREEZE/Pipfile.lock#L6)<sup><b><a href="#exported-dependency-information-notes-2">2</a></b></sup>, [2018.11.26](https://gitlab.com/gitlab-org/security-products/tests/python-pipenv/-/blob/master/Pipfile) | +| Pipenv | [2018.11.26](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python/-/blob/v2.18.4/requirements.txt#L13) | [2018.11.26](https://gitlab.com/gitlab-org/security-products/tests/python-pipenv/-/blob/pipfile-lock-FREEZE/Pipfile.lock#L6)<sup><b><a href="#exported-dependency-information-notes-2">2</a></b></sup>, [2018.11.26](https://gitlab.com/gitlab-org/security-products/tests/python-pipenv/-/blob/master/Pipfile) | <!-- markdownlint-disable MD044 --> <ol> @@ -389,7 +389,8 @@ The following analyzers are executed, each of which have different behavior when Does not support multiple lockfiles. When multiple lockfiles exist, `bundler-audit` analyzes the first lockfile discovered while traversing the directory tree in alphabetical order. -We execute both analyzers because they use different sources of vulnerability data. The result is more comprehensive analysis than if only one was executed. +WARNING: +The `bundler-audit` analyzer is deprecated and will be removed in GitLab 15.0 since it duplicates the functionality of the `gemnasium` analyzer. For more information, read the [deprecation announcement](../../../update/deprecations.md#bundler-audit-dependency-scanning-tool). #### Python @@ -418,6 +419,11 @@ The following analyzers are executed, each of which have different behavior when Does not support multiple lockfiles. When multiple lockfiles exist, `Retire.js` analyzes the first lockfile discovered while traversing the directory tree in alphabetical order. +From GitLab 14.8 the `Gemnasium` analyzer scans supported JavaScript projects for vendored libraries +(that is, those checked into the project but not managed by the package manager). + +WARNING: +The `retire.js` analyzer is deprecated and will be removed in GitLab 15.0 since it duplicates the functionality of the `gemnasium` analyzer. For more information, read the [deprecation announcement](../../../update/deprecations.md#retire-js-dependency-scanning-tool). We execute both analyzers because they use different sources of vulnerability data. The result is more comprehensive analysis than if only one was executed. #### PHP, Go, C, C++, .NET, C# @@ -547,7 +553,7 @@ The following variables allow configuration of global dependency scanning settin The following variables are used for configuring specific analyzers (used for a specific language/framework). | CI/CD variable | Analyzer | Default | Description | -| ------------------------------------ | ------------------ | ---------------------------- |------------ | +|--------------------------------------| ------------------ | ---------------------------- |------------ | | `BUNDLER_AUDIT_UPDATE_DISABLED` | `bundler-audit` | `"false"` | Disable automatic updates for the `bundler-audit` analyzer. Use if you're running dependency scanning in an offline, air-gapped environment.| | `BUNDLER_AUDIT_ADVISORY_DB_URL` | `bundler-audit` | `https://github.com/rubysec/ruby-advisory-db` | URL of the advisory database used by bundler-audit. | | `BUNDLER_AUDIT_ADVISORY_DB_REF_NAME` | `bundler-audit` | `master` | Git ref for the advisory database specified by `BUNDLER_AUDIT_ADVISORY_DB_URL`. | @@ -556,6 +562,7 @@ The following variables are used for configuring specific analyzers (used for a | `GEMNASIUM_DB_REMOTE_URL` | `gemnasium` | `https://gitlab.com/gitlab-org/security-products/gemnasium-db.git` | Repository URL for fetching the Gemnasium database. | | `GEMNASIUM_DB_REF_NAME` | `gemnasium` | `master` | Branch name for remote repository database. `GEMNASIUM_DB_REMOTE_URL` is required. | | `DS_REMEDIATE` | `gemnasium` | `"true"` | Enable automatic remediation of vulnerable dependencies. | +| `GEMNASIUM_LIBRARY_SCAN_ENABLED` | `gemnasium` | `"true"` | Enable detecting vulnerabilities in vendored JavaScript libraries. For now, `gemnasium` leverages [`Retire.js`](https://github.com/RetireJS/retire.js) to do this job. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/350512) in GitLab 14.8. | | `DS_JAVA_VERSION` | `gemnasium-maven` | `11` | Version of Java. Available versions: `8`, `11`, `13`, `14`, `15`, `16`. | | `MAVEN_CLI_OPTS` | `gemnasium-maven` | `"-DskipTests --batch-mode"` | List of command line arguments that are passed to `maven` by the analyzer. See an example for [using private repositories](../index.md#using-private-maven-repositories). | | `GRADLE_CLI_OPTS` | `gemnasium-maven` | | List of command line arguments that are passed to `gradle` by the analyzer. | @@ -565,11 +572,36 @@ The following variables are used for configuring specific analyzers (used for a | `PIP_REQUIREMENTS_FILE` | `gemnasium-python` | | Pip requirements file to be scanned. | | `DS_PIP_VERSION` | `gemnasium-python` | | Force the install of a specific pip version (example: `"19.3"`), otherwise the pip installed in the Docker image is used. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12811) in GitLab 12.7) | | `DS_PIP_DEPENDENCY_PATH` | `gemnasium-python` | | Path to load Python pip dependencies from. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12412) in GitLab 12.2) | -| `DS_PYTHON_VERSION` | `retire.js` | | Version of Python. If set to 2, dependencies are installed using Python 2.7 instead of Python 3.6. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12296) in GitLab 12.1, [removed](https://www.python.org/doc/sunset-python-2/) in GitLab 13.7). | | `RETIREJS_JS_ADVISORY_DB` | `retire.js` | `https://raw.githubusercontent.com/RetireJS/retire.js/master/repository/jsrepository.json` | Path or URL to `retire.js` JS vulnerability data file. Note that if the URL hosting the data file uses a custom SSL certificate, for example in an offline installation, you can pass the certificate in the `ADDITIONAL_CA_CERT_BUNDLE` variable. | | `RETIREJS_NODE_ADVISORY_DB` | `retire.js` | `https://raw.githubusercontent.com/RetireJS/retire.js/master/repository/npmrepository.json` | Path or URL to `retire.js` node vulnerability data file. Note that if the URL hosting the data file uses a custom SSL certificate, for example in an offline installation, you can pass the certificate in the `ADDITIONAL_CA_CERT_BUNDLE` variable. | | `RETIREJS_ADVISORY_DB_INSECURE` | `retire.js` | `false` | Enable fetching remote JS and Node vulnerability data files (defined by the `RETIREJS_JS_ADVISORY_DB` and `RETIREJS_NODE_ADVISORY_DB` variables) from hosts using an insecure or self-signed SSL (TLS) certificate. | +#### Other variables + +The previous tables are not an exhaustive list of all variables that can be used. They contain all specific GitLab and analyzer variables we support and test. There are many variables, such as environment variables, that you can pass in and they will work. This is a large list, many of which we may be unaware of, and as such is not documented. + +For example, to pass the non-GitLab environment variable `HTTPS_PROXY` to all Dependency Scanning jobs, +set it as a [custom CI/CD variable in your `.gitlab-ci.yml`](../../../ci/variables/#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file) +file like this: + +```yaml +variables: + HTTPS_PROXY: "https://squid-proxy:3128" +``` + +Alternatively we may use it in specific jobs, like Dependency Scanning: + +```yaml +dependency_scanning: + variables: + HTTPS_PROXY: $HTTPS_PROXY +``` + +As we have not tested all variables you may find some will work and others will not. +If one does not work and you need it we suggest +[submitting a feature request](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Feature%20proposal%20-%20detailed&issue[title]=Docs%20feedback%20-%20feature%20proposal:%20Write%20your%20title) +or [contributing to the code](../../../development/index.md) to enable it to be used. + ### Using a custom SSL CA certificate authority You can use the `ADDITIONAL_CA_CERT_BUNDLE` CI/CD variable to configure a custom SSL CA certificate authority. The `ADDITIONAL_CA_CERT_BUNDLE` value should contain the [text representation of the X.509 PEM public-key certificate](https://tools.ietf.org/html/rfc7468#section-5.1). For example, to configure this value in the `.gitlab-ci.yml` file, use the following: @@ -613,7 +645,7 @@ vulnerabilities in your groups, projects and pipelines. Read more about the ## Vulnerabilities database update For more information about the vulnerabilities database update, see the -[maintenance table](../vulnerabilities/index.md#vulnerability-scanner-maintenance). +[maintenance table](../index.md#vulnerability-scanner-maintenance). ## Dependency List @@ -735,6 +767,87 @@ Here's an example dependency scanning report: } ``` +### CycloneDX reports + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/350509) in GitLab 14.8 in [Beta](../../../policy/alpha-beta-support.md#beta-features). + +In addition to the [JSON report file](#reports-json-format), the [Gemnasium](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium) +Dependency Scanning tool outputs a [CycloneDX](https://cyclonedx.org/) report for +each supported lock or build file it detects. These CycloneDX reports are named +`cyclonedx-<package-type>-<package-manager>.json`, and are saved in the same directory +as the detected lock or build files. + +For example, if your project has the following structure: + +```plaintext +. +├── ruby-project/ +│ └── Gemfile.lock +├── ruby-project-2/ +│ └── Gemfile.lock +├── php-project/ +│ └── composer.lock +└── go-project/ + └── go.sum +``` + +Then the Gemnasium scanner generates the following CycloneDX reports: + +```plaintext +. +├── ruby-project/ +│ ├── Gemfile.lock +│ └── cyclonedx-gem-bundler.json +├── ruby-project-2/ +│ ├── Gemfile.lock +│ └── cyclonedx-gem-bundler.json +├── php-project/ +│ ├── composer.lock +│ └── cyclonedx-packagist-composer.json +└── go-project/ + ├── go.sum + └── cyclonedx-go-go.json +``` + +The CycloneDX reports can be downloaded [the same way as other job artifacts](../../../ci/pipelines/job_artifacts.md#download-job-artifacts). + +### Merging multiple CycloneDX Reports + +You can use a CI/CD job to merge multiple CycloneDX Reports into a single report. +For example: + +```yaml +stages: + - test + - merge-cyclonedx-reports + +include: + - template: Security/Dependency-Scanning.gitlab-ci.yml + +merge cyclonedx reports: + stage: merge-cyclonedx-reports + image: alpine:latest + script: + - wget https://github.com/CycloneDX/cyclonedx-cli/releases/download/v0.22.0/cyclonedx-linux-musl-x64 -O /usr/local/bin/cyclonedx-cli + - chmod 755 /usr/local/bin/cyclonedx-cli + - apk --update add --no-cache icu-dev libstdc++ + - find * -name "cyclonedx-*.json" -exec cyclonedx-cli merge --input-files {} --output-file cyclonedx-all.json + + artifacts: + paths: + - cyclonedx-all.json +``` + +GitLab uses [CycloneDX Properties](https://cyclonedx.org/use-cases/#properties--name-value-store) +to store implementation-specific details in the metadata of each CycloneDX report, +such as the location of build and lock files. If multiple CycloneDX reports are merged together, +this information is removed from the resulting merged file. + +NOTE: +CycloneDX reports are a [Beta](../../../policy/alpha-beta-support.md#beta-features) feature, +and the reports are subject to change during the beta period. Do not build integrations +that rely on the format of these reports staying consistent, as the format might change +before the feature is made generally available. + ## Versioning and release process Please check the [Release Process documentation](https://gitlab.com/gitlab-org/security-products/release/blob/master/docs/release_process.md). @@ -789,7 +902,7 @@ registry.gitlab.com/gitlab-org/security-products/analyzers/bundler-audit:2 The process for importing Docker images into a local offline Docker registry depends on **your network security policy**. Please consult your IT staff to find an accepted and approved process by which external resources can be imported or temporarily accessed. -These scanners are [periodically updated](../vulnerabilities/index.md#vulnerability-scanner-maintenance) +These scanners are [periodically updated](../index.md#vulnerability-scanner-maintenance) with new definitions, and you may be able to make occasional updates on your own. For details on saving and transporting Docker images as a file, see Docker's documentation on diff --git a/doc/user/application_security/iac_scanning/index.md b/doc/user/application_security/iac_scanning/index.md index 4d5c1da3c47..89d3531bccd 100644 --- a/doc/user/application_security/iac_scanning/index.md +++ b/doc/user/application_security/iac_scanning/index.md @@ -36,9 +36,15 @@ GitLab IaC scanning supports a variety of IaC configuration files. Our IaC secur |------------------------------------------|----------------------------------|-------------------------------| | Ansible | [KICS](https://kics.io/) | 14.5 | | AWS CloudFormation | [KICS](https://kics.io/) | 14.5 | +| Azure Resource Manager <sup>1</sup> | [KICS](https://kics.io/) | 14.5 | +| Dockerfile | [KICS](https://kics.io/) | 14.5 | +| Google Deployment Manager | [KICS](https://kics.io/) | 14.5 | | Kubernetes | [KICS](https://kics.io/) | 14.5 | +| OpenAPI | [KICS](https://kics.io/) | 14.5 | | Terraform | [KICS](https://kics.io/) | 14.5 | +1. IaC scanning can analyze Azure Resource Manager templates in JSON format. If you write templates in the [Bicep](https://docs.microsoft.com/en-us/azure/azure-resource-manager/bicep/overview) language, you must use [the bicep CLI](https://docs.microsoft.com/en-us/azure/azure-resource-manager/bicep/bicep-cli) to convert your Bicep files into JSON before GitLab IaC scanning can analyze them. + ### Making IaC analyzers available to all GitLab tiers All open source (OSS) analyzers are available with the GitLab Free tier. Future proprietary analyzers may be restricted to higher tiers. diff --git a/doc/user/application_security/index.md b/doc/user/application_security/index.md index f5c727a1418..6a0b81335fd 100644 --- a/doc/user/application_security/index.md +++ b/doc/user/application_security/index.md @@ -2,7 +2,6 @@ stage: Secure group: Static Analysis info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference, howto --- # Secure your application **(ULTIMATE)** @@ -46,6 +45,25 @@ GitLab uses the following tools to scan and report known vulnerabilities found i | [Coverage fuzzing](coverage_fuzzing/index.md) | Find unknown bugs and vulnerabilities with coverage-guided fuzzing. | | [Cluster Image Scanning](cluster_image_scanning/index.md) | Scan Kubernetes clusters for known vulnerabilities. | +## Vulnerability scanner maintenance + +The following vulnerability scanners and their databases are regularly updated: + +| Secure scanning tool | Vulnerabilities database updates | +|:----------------------------------------------------------------|:---------------------------------| +| [Container Scanning](container_scanning/index.md) | A job runs on a daily basis to build new images with the latest vulnerability database updates from the upstream scanner. For more details, see [Vulnerabilities database update](container_scanning/index.md#vulnerabilities-database-update). | +| [Dependency Scanning](dependency_scanning/index.md) | Relies on `bundler-audit` (for Ruby gems), `retire.js` (for npm packages), and `gemnasium` (the GitLab tool for all libraries). Both `bundler-audit` and `retire.js` fetch their vulnerabilities data from GitHub repositories, so vulnerabilities added to `ruby-advisory-db` and `retire.js` are immediately available. The tools themselves are updated once per month if there's a new version. The [Gemnasium DB](https://gitlab.com/gitlab-org/security-products/gemnasium-db) is updated on a daily basis using [data from NVD, the `ruby-advisory-db` and the GitHub Security Advisory Database as data sources](https://gitlab.com/gitlab-org/security-products/gemnasium-db/-/blob/master/SOURCES.md). See our [current measurement of time from CVE being issued to our product being updated](https://about.gitlab.com/handbook/engineering/development/performance-indicators/#cve-issue-to-update). | +| [Dynamic Application Security Testing (DAST)](dast/index.md) | The scanning engine is updated on a periodic basis. See the [version of the underlying tool `zaproxy`](https://gitlab.com/gitlab-org/security-products/dast/blob/main/Dockerfile#L1). The scanning rules are downloaded at scan runtime. | +| [Static Application Security Testing (SAST)](sast/index.md) | Relies exclusively on [the tools GitLab wraps](sast/index.md#supported-languages-and-frameworks). The underlying analyzers are updated at least once per month if a relevant update is available. The vulnerabilities database is updated by the upstream tools. | + +In versions of GitLab that use the same major version of the analyzer, you do not have to update +GitLab to benefit from the latest vulnerabilities definitions. The security tools are released as +Docker images. The vendored job definitions that enable them use major release tags according to +[semantic versioning](https://semver.org/). Each new release of the tools overrides these tags. +Although in a major analyzer version you automatically get the latest versions of the scanning +tools, there are some [known issues](https://gitlab.com/gitlab-org/gitlab/-/issues/9725) with this +approach. + ## Security scanning with Auto DevOps To enable all GitLab Security scanning tools, with default settings, enable @@ -98,10 +116,10 @@ base address for Docker images. You can override this globally by setting the CI the container-scanning analyzer which uses `registry.gitlab.com/security-products/container-scanning` as its registry. -### Use security scanning tools with Pipelines for Merge Requests +### Use security scanning tools with merge request pipelines By default, the application security jobs are configured to run for branch pipelines only. -To use them with [pipelines for merge requests](../../ci/pipelines/merge_request_pipelines.md), +To use them with [merge request pipelines](../../ci/pipelines/merge_request_pipelines.md), you may need to override the default `rules:` configuration to add: ```yaml @@ -149,8 +167,8 @@ The artifact generated by the secure analyzer contains all findings it discovers ### All tiers Merge requests which have run security scans let you know that the generated -reports are available to download. To download a report, click on the -**Download results** dropdown, and select the desired report. +reports are available to download. To download a report, select +**Download results**, and select the desired report.  @@ -200,12 +218,17 @@ security issues: ### Vulnerability-Check rule +WARNING: +This feature is in its end-of-life process. It is [deprecated](../../update/deprecations.md#vulnerability-check) +for use in GitLab 14.8, and is planned for removal in GitLab 15.0. Users should migrate to the new +[Security Approval Policies](policies/#scan-result-policy-editor). + To prevent a merge request introducing a security vulnerability in a project, enable the Vulnerability-Check rule. While this rule is enabled, additional merge request approval by [eligible approvers](../project/merge_requests/approvals/rules.md#eligible-approvers) is required when the latest security report in a merge request: -- Contains vulnerabilities with states (for example, `previously detected`, `dismissed`) matching the rule's vulnerability states. Only `newly detected` will be considered if the target branch differs from the project default branch. +- Contains vulnerabilities with states (for example, `previously detected`, `dismissed`) matching the rule's vulnerability states. Only `newly detected` are considered if the target branch differs from the project default branch. - Contains vulnerabilities with severity levels (for example, `high`, `critical`, or `unknown`) matching the rule's severity levels. - Contains a vulnerability count higher than the rule allows. @@ -218,7 +241,7 @@ An approval is optional when the security report: the rule's severity levels. - Contains a vulnerability count equal to or less than what the rule allows. -Project members assigned [at least the Maintainer role](../permissions.md#project-members-permissions) can enable or edit +Project members with at least the Maintainer role can enable or edit the Vulnerability-Check rule. #### Enable the Vulnerability-Check rule @@ -451,28 +474,24 @@ GitLab provides two methods of accomplishing this, each with advantages and disa - [Compliance framework pipelines](../project/settings/#compliance-pipeline-configuration) are recommended when: - - Scan execution enforcement is required for SAST IaC, Container Scanning, Dependency Scanning, + - Scan execution enforcement is required for SAST IaC, Dependency Scanning, License Compliance, API Fuzzing, or Coverage-guided Fuzzing. - - Scan execution enforcement of SAST or Secret Detection when customization of the default scan - variables is required. - Scan execution enforcement is required for scanners external to GitLab. - Enforced execution is required for custom jobs other than security scans. -- [Scan execution policies](policies/#scan-execution-policies) +- [Scan execution policies](policies/scan-execution-policies.md) are recommended when: - - Scan execution enforcement is required for DAST. - - Scan execution enforcement is required for SAST or Secret Detection with the default scan - variables. + - Scan execution enforcement is required for DAST, SAST, Secret Detection, or Container Scanning. - Scans are required to run on a regular, scheduled cadence. Additional details about the differences between the two solutions are outlined below: | | Compliance Framework Pipelines | Scan Execution Policies | | ------ | ------ | ------ | -| **Flexibility** | Supports anything that can be done in a CI file. | Limited to only the items for which GitLab has explicitly added support. DAST, SAST, and Secret Detection scans are supported. | +| **Flexibility** | Supports anything that can be done in a CI file. | Limited to only the items for which GitLab has explicitly added support. DAST, SAST, Secret Detection, and Container Scanning scans are supported. | | **Usability** | Requires knowledge of CI YAML. | Follows a `rules` and `actions`-based YAML structure. | -| **Inclusion in CI pipeline** | The compliance pipeline is executed instead of the project's `gitlab-ci.yml` file. To include the project's `gitlab-ci.yml` file, use an `include` statement. Defined variables aren't allowed to be overwritten by the included project's YAML file. | Forced inclusion of a new job into the CI pipeline. DAST jobs that must be customized on a per-project basis can have project-level Site Profiles and Scan Profiles defined. To ensure separation of duties, these profiles are immutable when referenced in a scan execution policy. | +| **Inclusion in CI pipeline** | The compliance pipeline is executed instead of the project's `gitlab-ci.yml` file. To include the project's `gitlab-ci.yml` file, use an `include` statement. Defined variables aren't allowed to be overwritten by the included project's YAML file. | Forced inclusion of a new job into the CI pipeline. DAST jobs that must be customized on a per-project basis can have project-level Site Profiles and Scan Profiles defined. To ensure separation of duties, these profiles are immutable when referenced in a scan execution policy. All jobs can be customized as part of the security policy itself with the same variables that are normally available to the CI job. | | **Schedulable** | Can be scheduled through a scheduled pipeline on the group. | Can be scheduled natively through the policy configuration itself. | | **Separation of Duties** | Only group owners can create compliance framework labels. Only project owners can apply compliance framework labels to projects. The ability to make or approve changes to the compliance pipeline definition is limited to individuals who are explicitly given access to the project that contains the compliance pipeline. | Only project owners can define a linked security policy project. The ability to make or approve changes to security policies is limited to individuals who are explicitly given access to the security policy project. | | **Ability to apply one standard to multiple projects** | The same compliance framework label can be applied to multiple projects inside a group. | The same security policy project can be used for multiple projects across GitLab with no requirement of being located in the same group. | diff --git a/doc/user/application_security/policies/img/scan_result_policy_yaml_mode_v14_6.png b/doc/user/application_security/policies/img/scan_result_policy_yaml_mode_v14_6.png Binary files differnew file mode 100644 index 00000000000..57649c58d8b --- /dev/null +++ b/doc/user/application_security/policies/img/scan_result_policy_yaml_mode_v14_6.png diff --git a/doc/user/application_security/policies/index.md b/doc/user/application_security/policies/index.md index 11f2b91177a..803f3983b96 100644 --- a/doc/user/application_security/policies/index.md +++ b/doc/user/application_security/policies/index.md @@ -17,8 +17,9 @@ can access these by navigating to your project's **Security & Compliance > Polic GitLab supports the following security policies: -- [Container Network Policy](#container-network-policy) -- [Scan Execution Policy](#scan-execution-policy-schema) +- [Scan Execution Policy](scan-execution-policies.md) +- [Scan Result Policy](scan-result-policies.md) +- [Container Network Policy](#container-network-policy) (DEPRECATED) ## Policy management @@ -77,9 +78,62 @@ by the Rule mode, Rule mode is automatically disabled. If the YAML is incorrect, you must use YAML mode to fix your policy before Rule mode is available again. +## Security Policies project + +NOTE: +We recommend using the [Security Policies project](#security-policies-project) +exclusively for managing policies for the project. Do not add your application's source code to such +projects. + +The Security Policies feature is a repository to store policies. All security policies are stored in +the `.gitlab/security-policies/policy.yml` YAML file. The format for this YAML is specific to the type of policy that is being stored there. Examples and schema information are available for the following policy types: + +- [Scan execution policy](scan-execution-policies.md#example-security-policies-project) +- [Scan result policy](scan-result-policies.md#example-security-scan-result-policies-project) + +Policies created in this project are applied through a background job that runs once every 10 +minutes. Allow up to 10 minutes for any policy changes committed to this project to take effect. + +## Security Policy project selection + +NOTE: +Only project Owners have the [permissions](../../permissions.md#project-members-permissions) +to select Security Policy Project. + +When the Security Policy project is created and policies are created within that repository, you +must create an association between that project and the project you want to apply policies to: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Security & Compliance > Policies**. +1. Select **Edit Policy Project**, and search for and select the + project you would like to link from the dropdown menu. +1. Select **Save**. + +  + +### Unlink Security Policy projects + +Project owners can unlink Security Policy projects from development projects. To do this, follow +the steps described in [Security Policy project selection](#security-policy-project-selection), +but select the trash can icon in the modal. + +## Scan execution policies + +See [Scan execution policies](scan-execution-policies.md). + +## Scan result policy editor + +See [Scan result policies](scan-result-policies.md). + ## Container Network Policy -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32365) in GitLab 12.9. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32365) in GitLab 12.9. +> - [Deprecated](https://gitlab.com/groups/gitlab-org/-/epics/7476) in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) in GitLab 15.0. + +WARNING: +Container Network Policy is in its end-of-life process. It's [deprecated](https://gitlab.com/groups/gitlab-org/-/epics/7476) +for use in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) +in GitLab 15.0. The **Container Network Policy** section provides packet flow metrics for your application's Kubernetes namespace. This section has the following @@ -159,7 +213,7 @@ at the bottom of the editor. You can use policy alerts to track your policy's impact. Alerts are only available if you've [installed](../../clusters/agent/repository.md) -and [configured](../../clusters/agent/install/index.md#register-an-agent-with-gitlab) +and [configured](../../clusters/agent/install/index.md#register-the-agent-with-gitlab) an agent for this project. There are two ways to create policy alerts: @@ -176,276 +230,7 @@ There are two ways to create policy alerts: Once added, the UI updates and displays a warning about the dangers of too many alerts. -## Security Policies project - -NOTE: -We recommend using the [Security Policies project](#security-policies-project) -exclusively for managing policies for the project. Do not add your application's source code to such -projects. - -The Security Policies feature is a repository to store policies. All security policies are stored as -the `.gitlab/security-policies/policy.yml` YAML file with this format: - -```yaml ---- -scan_execution_policy: -- name: Enforce DAST in every pipeline - description: This policy enforces pipeline configuration to have a job with DAST scan - enabled: true - rules: - - type: pipeline - branches: - - master - actions: - - scan: dast - scanner_profile: Scanner Profile A - site_profile: Site Profile B -- name: Enforce DAST in every pipeline in main branch - description: This policy enforces pipeline configuration to have a job with DAST scan for main branch - enabled: true - rules: - - type: pipeline - branches: - - main - actions: - - scan: dast - scanner_profile: Scanner Profile C - site_profile: Site Profile D -``` - -## Security Policy project selection - -NOTE: -Only project Owners have the [permissions](../../permissions.md#project-members-permissions) -to select Security Policy Project. - -When the Security Policy project is created and policies are created within that repository, you -must create an association between that project and the project you want to apply policies to: - -1. On the top bar, select **Menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > Policies**. -1. Select **Edit Policy Project**, and search for and select the - project you would like to link from the dropdown menu. -1. Select **Save**. - -  - -### Unlink Security Policy projects - -Project owners can unlink Security Policy projects from development projects. To do this, follow -the steps described in [Security Policy project selection](#security-policy-project-selection), -but select the trash can icon in the modal. - -## Scan execution policies - -Project owners can use scan execution policies to require that security scans run on a specified -schedule or with the project pipeline. Required scans are injected into the CI pipeline as new jobs -with a long, random job name. In the unlikely event of a job name collision, the security policy job -overwrites any pre-existing job in the pipeline. - -This feature has some overlap with [compliance framework pipelines](../../project/settings/#compliance-pipeline-configuration), -as we have not [unified the user experience for these two features](https://gitlab.com/groups/gitlab-org/-/epics/7312). -For details on the similarities and differences between these features, see -[Enforce scan execution](../#enforce-scan-execution). - -### Scan Execution Policy editor - -NOTE: -Only project Owners have the [permissions](../../permissions.md#project-members-permissions) -to select Security Policy Project. - -Once your policy is complete, save it by selecting **Create via merge request** -at the bottom of the editor. You are redirected to the merge request on the project's -configured security policy project. If one does not link to your project, a security -policy project is automatically created. Existing policies can also be -removed from the editor interface by selecting **Delete policy** -at the bottom of the editor. - - - -The policy editor currently only supports the YAML mode. The Rule mode is tracked in the [Allow Users to Edit Rule-mode Scan Execution Policies in the Policy UI](https://gitlab.com/groups/gitlab-org/-/epics/5363) epic. - -### Scan Execution Policies Schema - -The YAML file with Scan Execution Policies consists of an array of objects matching Scan Execution Policy Schema nested under the `scan_execution_policy` key. You can configure a maximum of 5 policies under the `scan_execution_policy` key. - -When you save a new policy, GitLab validates its contents against [this JSON schema](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/validators/json_schemas/security_orchestration_policy.json). -If you're not familiar with how to read [JSON schemas](https://json-schema.org/), -the following sections and tables provide an alternative. - -| Field | Type | Possible values | Description | -|-------|------|-----------------|-------------| -| `scan_execution_policy` | `array` of Scan Execution Policy | | List of scan execution policies (maximum 5) | - -### Scan Execution Policy Schema - -| Field | Type | Possible values | Description | -|-------|------|-----------------|-------------| -| `name` | `string` | | Name of the policy. | -| `description` (optional) | `string` | | Description of the policy. | -| `enabled` | `boolean` | `true`, `false` | Flag to enable (`true`) or disable (`false`) the policy. | -| `rules` | `array` of rules | | List of rules that the policy applies. | -| `actions` | `array` of actions | | List of actions that the policy enforces. | - -### `pipeline` rule type - -This rule enforces the defined actions whenever the pipeline runs for a selected branch. - -| Field | Type | Possible values | Description | -|-------|------|-----------------|-------------| -| `type` | `string` | `pipeline` | The rule's type. | -| `branches` | `array` of `string` | `*` or the branch's name | The branch the given policy applies to (supports wildcard). | - -### `schedule` rule type - -This rule enforces the defined actions and schedules a scan on the provided date/time. - -| Field | Type | Possible values | Description | -|------------|------|-----------------|-------------| -| `type` | `string` | `schedule` | The rule's type. | -| `branches` | `array` of `string` | `*` or the branch's name | The branch the given policy applies to (supports wildcard). | -| `cadence` | `string` | CRON expression (for example, `0 0 * * *`) | A whitespace-separated string containing five fields that represents the scheduled time. | -| `clusters` | `object` | | The cluster where the given policy enforces running selected scans (only for `container_scanning`/`cluster_image_scanning` scans). The key of the object is the name of the Kubernetes cluster configured for your project in GitLab. In the optionally provided value of the object, you can precisely select Kubernetes resources that are scanned. | - -#### `cluster` schema - -Use this schema to define `clusters` objects in the [`schedule` rule type](#schedule-rule-type). - -| Field | Type | Possible values | Description | -|--------------|---------------------|--------------------------|-------------| -| `containers` | `array` of `string` | | The container name that is scanned (only the first value is currently supported). | -| `resources` | `array` of `string` | | The resource name that is scanned (only the first value is currently supported). | -| `namespaces` | `array` of `string` | | The namespace that is scanned (only the first value is currently supported). | -| `kinds` | `array` of `string` | `deployment`/`daemonset` | The resource kind that should be scanned (only the first value is currently supported). | - -### `scan` action type - -This action executes the selected `scan` with additional parameters when conditions for at least one -rule in the defined policy are met. - -| Field | Type | Possible values | Description | -|-------|------|-----------------|-------------| -| `scan` | `string` | `dast`, `secret_detection`, `sast`, `container_scanning`, `cluster_image_scanning` | The action's type. | -| `site_profile` | `string` | Name of the selected [DAST site profile](../dast/index.md#site-profile). | The DAST site profile to execute the DAST scan. This field should only be set if `scan` type is `dast`. | -| `scanner_profile` | `string` or `null` | Name of the selected [DAST scanner profile](../dast/index.md#scanner-profile). | The DAST scanner profile to execute the DAST scan. This field should only be set if `scan` type is `dast`.| -| `variables` | `object` | | Set of variables applied and enforced for the selected scan. The object's key is the variable name with a value provided as a string. | - -Note the following: - -- You must create the [site profile](../dast/index.md#site-profile) and [scanner profile](../dast/index.md#scanner-profile) - with selected names for each project that is assigned to the selected Security Policy Project. - Otherwise, the policy is not applied and a job with an error message is created instead. -- Once you associate the site profile and scanner profile by name in the policy, it is not possible - to modify or delete them. If you want to modify them, you must first disable the policy by setting - the `active` flag to `false`. -- When configuring policies with a scheduled DAST scan, the author of the commit in the security - policy project's repository must have access to the scanner and site profiles. Otherwise, the scan - is not scheduled successfully. -- For a secret detection scan, only rules with the default ruleset are supported. [Custom rulesets](../secret_detection/index.md#custom-rulesets) - are not supported. -- A secret detection scan runs in `normal` mode when executed as part of a pipeline, and in - [`historic`](../secret_detection/index.md#full-history-secret-detection) - mode when executed as part of a scheduled scan. -- A container scanning and cluster image scanning scans configured for the `pipeline` rule type ignores the cluster defined in the `clusters` object. - They use predefined CI/CD variables defined for your project. Cluster selection with the `clusters` object is supported for the `schedule` rule type. - Cluster with name provided in `clusters` object must be created and configured for the project. To be able to successfully perform the `container_scanning`/`cluster_image_scanning` scans for the cluster you must follow instructions for the [Cluster Image Scanning feature](../cluster_image_scanning/index.md#prerequisites). -- The SAST scan uses the default template and runs in a [child pipeline](../../../ci/pipelines/parent_child_pipelines.md). - -### Example security policies project - -You can use this example in a `.gitlab/security-policies/policy.yml`, as described in -[Security policies project](#security-policies-project). - -```yaml ---- -scan_execution_policy: -- name: Enforce DAST in every release pipeline - description: This policy enforces pipeline configuration to have a job with DAST scan for release branches - enabled: true - rules: - - type: pipeline - branches: - - release/* - actions: - - scan: dast - scanner_profile: Scanner Profile A - site_profile: Site Profile B -- name: Enforce DAST and secret detection scans every 10 minutes - description: This policy enforces DAST and secret detection scans to run every 10 minutes - enabled: true - rules: - - type: schedule - branches: - - main - cadence: "*/10 * * * *" - actions: - - scan: dast - scanner_profile: Scanner Profile C - site_profile: Site Profile D - - scan: secret_detection -- name: Enforce Secret Detection and Container Scanning in every default branch pipeline - description: This policy enforces pipeline configuration to have a job with Secret Detection and Container Scanning scans for the default branch - enabled: true - rules: - - type: pipeline - branches: - - main - actions: - - scan: secret_detection - - scan: sast - variables: - SAST_EXCLUDED_ANALYZERS: brakeman - - scan: container_scanning -- name: Enforce Cluster Image Scanning on production-cluster every 24h - description: This policy enforces Cluster Image Scanning scan to run every 24 hours - enabled: true - rules: - - type: schedule - cadence: "15 3 * * *" - clusters: - production-cluster: - containers: - - database - resources: - - production-application - namespaces: - - production-namespace - kinds: - - deployment - actions: - - scan: cluster_image_scanning -``` - -In this example: - -- For every pipeline executed on branches that match the `release/*` wildcard (for example, branch - `release/v1.2.1`), DAST scans run with `Scanner Profile A` and `Site Profile B`. -- DAST and secret detection scans run every 10 minutes. The DAST scan runs with `Scanner Profile C` - and `Site Profile D`. -- Secret detection, container scanning, and SAST scans run for every pipeline executed on the `main` - branch. The SAST scan runs with the `SAST_EXCLUDED_ANALYZER` variable set to `"brakeman"`. -- Cluster Image Scanning scan runs every 24h. The scan runs on the `production-cluster` cluster and fetches vulnerabilities - from the container with the name `database` configured for deployment with the name `production-application` in the `production-namespace` namespace. - -### Example for scan execution policy editor - -You can use this example in the YAML mode of the [Scan Execution Policy editor](#scan-execution-policy-editor). -It corresponds to a single object from the previous example. - -```yaml -name: Enforce Secret Detection and Container Scanning in every default branch pipeline -description: This policy enforces pipeline configuration to have a job with Secret Detection and Container Scanning scans for the default branch -enabled: true -rules: - - type: pipeline - branches: - - main -actions: - - scan: secret_detection - - scan: container_scanning -``` - ## Roadmap -See the [Category Direction page](https://about.gitlab.com/direction/protect/container_network_security/) -for more information on the product direction of Container Network Security. +See the [Category Direction page](https://about.gitlab.com/direction/protect/security_orchestration/) +for more information on the product direction of security policies within GitLab. diff --git a/doc/user/application_security/policies/scan-execution-policies.md b/doc/user/application_security/policies/scan-execution-policies.md new file mode 100644 index 00000000000..4e44162d5c5 --- /dev/null +++ b/doc/user/application_security/policies/scan-execution-policies.md @@ -0,0 +1,219 @@ +--- +stage: Protect +group: Container Security +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Scan execution policies **(ULTIMATE)** + +Project owners can use scan execution policies to require that security scans run on a specified +schedule or with the project pipeline. Required scans are injected into the CI pipeline as new jobs +with a long, random job name. In the unlikely event of a job name collision, the security policy job +overwrites any pre-existing job in the pipeline. + +This feature has some overlap with [compliance framework pipelines](../../project/settings/#compliance-pipeline-configuration), +as we have not [unified the user experience for these two features](https://gitlab.com/groups/gitlab-org/-/epics/7312). +For details on the similarities and differences between these features, see +[Enforce scan execution](../#enforce-scan-execution). + +## Scan execution policy editor + +NOTE: +Only project Owners have the [permissions](../../permissions.md#project-members-permissions) +to select Security Policy Project. + +Once your policy is complete, save it by selecting **Create via merge request** +at the bottom of the editor. You are redirected to the merge request on the project's +configured security policy project. If one does not link to your project, a security +policy project is automatically created. Existing policies can also be +removed from the editor interface by selecting **Delete policy** +at the bottom of the editor. + +All scan execution policy changes are applied through a background job that runs once every 10 +minutes. Allow up to 10 minutes for any policy changes committed to this project to take effect. + + + +The policy editor currently only supports the YAML mode. The Rule mode is tracked in the [Allow Users to Edit Rule-mode Scan Execution Policies in the Policy UI](https://gitlab.com/groups/gitlab-org/-/epics/5363) epic. + +## Scan execution policies schema + +The YAML file with scan execution policies consists of an array of objects matching scan execution +policy schema nested under the `scan_execution_policy` key. You can configure a maximum of 5 +policies under the `scan_execution_policy` key. + +When you save a new policy, GitLab validates its contents against [this JSON schema](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/validators/json_schemas/security_orchestration_policy.json). +If you're not familiar with how to read [JSON schemas](https://json-schema.org/), +the following sections and tables provide an alternative. + +| Field | Type | Possible values | Description | +|-------|------|-----------------|-------------| +| `scan_execution_policy` | `array` of scan execution policy | | List of scan execution policies (maximum 5) | + +## Scan execution policy schema + +| Field | Type | Possible values | Description | +|-------|------|-----------------|-------------| +| `name` | `string` | | Name of the policy. | +| `description` (optional) | `string` | | Description of the policy. | +| `enabled` | `boolean` | `true`, `false` | Flag to enable (`true`) or disable (`false`) the policy. | +| `rules` | `array` of rules | | List of rules that the policy applies. | +| `actions` | `array` of actions | | List of actions that the policy enforces. | + +## `pipeline` rule type + +This rule enforces the defined actions whenever the pipeline runs for a selected branch. + +| Field | Type | Possible values | Description | +|-------|------|-----------------|-------------| +| `type` | `string` | `pipeline` | The rule's type. | +| `branches` | `array` of `string` | `*` or the branch's name | The branch the given policy applies to (supports wildcard). | + +## `schedule` rule type + +This rule enforces the defined actions and schedules a scan on the provided date/time. + +| Field | Type | Possible values | Description | +|------------|------|-----------------|-------------| +| `type` | `string` | `schedule` | The rule's type. | +| `branches` | `array` of `string` | `*` or the branch's name | The branch the given policy applies to (supports wildcard). | +| `cadence` | `string` | CRON expression (for example, `0 0 * * *`) | A whitespace-separated string containing five fields that represents the scheduled time. | +| `clusters` | `object` | | The cluster where the given policy enforces running selected scans (only for `container_scanning`/`cluster_image_scanning` scans). The key of the object is the name of the Kubernetes cluster configured for your project in GitLab. In the optionally provided value of the object, you can precisely select Kubernetes resources that are scanned. | + +### `cluster` schema + +Use this schema to define `clusters` objects in the [`schedule` rule type](#schedule-rule-type). + +| Field | Type | Possible values | Description | +|--------------|---------------------|--------------------------|-------------| +| `containers` | `array` of `string` | | The container name that is scanned (only the first value is currently supported). | +| `resources` | `array` of `string` | | The resource name that is scanned (only the first value is currently supported). | +| `namespaces` | `array` of `string` | | The namespace that is scanned (only the first value is currently supported). | +| `kinds` | `array` of `string` | `deployment`/`daemonset` | The resource kind that should be scanned (only the first value is currently supported). | + +## `scan` action type + +This action executes the selected `scan` with additional parameters when conditions for at least one +rule in the defined policy are met. + +| Field | Type | Possible values | Description | +|-------|------|-----------------|-------------| +| `scan` | `string` | `dast`, `secret_detection`, `sast`, `container_scanning`, `cluster_image_scanning` | The action's type. | +| `site_profile` | `string` | Name of the selected [DAST site profile](../dast/index.md#site-profile). | The DAST site profile to execute the DAST scan. This field should only be set if `scan` type is `dast`. | +| `scanner_profile` | `string` or `null` | Name of the selected [DAST scanner profile](../dast/index.md#scanner-profile). | The DAST scanner profile to execute the DAST scan. This field should only be set if `scan` type is `dast`.| +| `variables` | `object` | | A set of CI variables, supplied as an array of `key: value` pairs, to apply and enforce for the selected scan. The `key` is the variable name, with its `value` provided as a string. This parameter supports any variable that the GitLab CI job supports for the specified scan. | + +Note the following: + +- You must create the [site profile](../dast/index.md#site-profile) and [scanner profile](../dast/index.md#scanner-profile) + with selected names for each project that is assigned to the selected Security Policy Project. + Otherwise, the policy is not applied and a job with an error message is created instead. +- Once you associate the site profile and scanner profile by name in the policy, it is not possible + to modify or delete them. If you want to modify them, you must first disable the policy by setting + the `active` flag to `false`. +- When configuring policies with a scheduled DAST scan, the author of the commit in the security + policy project's repository must have access to the scanner and site profiles. Otherwise, the scan + is not scheduled successfully. +- For a secret detection scan, only rules with the default ruleset are supported. [Custom rulesets](../secret_detection/index.md#custom-rulesets) + are not supported. +- A secret detection scan runs in `normal` mode when executed as part of a pipeline, and in + [`historic`](../secret_detection/index.md#full-history-secret-detection) + mode when executed as part of a scheduled scan. +- A container scanning and cluster image scanning scans configured for the `pipeline` rule type ignores the cluster defined in the `clusters` object. + They use predefined CI/CD variables defined for your project. Cluster selection with the `clusters` object is supported for the `schedule` rule type. + Cluster with name provided in `clusters` object must be created and configured for the project. To be able to successfully perform the `container_scanning`/`cluster_image_scanning` scans for the cluster you must follow instructions for the [Cluster Image Scanning feature](../cluster_image_scanning/index.md#prerequisites). +- The SAST scan uses the default template and runs in a [child pipeline](../../../ci/pipelines/parent_child_pipelines.md). + +## Example security policies project + +You can use this example in a `.gitlab/security-policies/policy.yml`, as described in +[Security policies project](index.md#security-policies-project). + +```yaml +--- +scan_execution_policy: +- name: Enforce DAST in every release pipeline + description: This policy enforces pipeline configuration to have a job with DAST scan for release branches + enabled: true + rules: + - type: pipeline + branches: + - release/* + actions: + - scan: dast + scanner_profile: Scanner Profile A + site_profile: Site Profile B +- name: Enforce DAST and secret detection scans every 10 minutes + description: This policy enforces DAST and secret detection scans to run every 10 minutes + enabled: true + rules: + - type: schedule + branches: + - main + cadence: "*/10 * * * *" + actions: + - scan: dast + scanner_profile: Scanner Profile C + site_profile: Site Profile D + - scan: secret_detection +- name: Enforce Secret Detection and Container Scanning in every default branch pipeline + description: This policy enforces pipeline configuration to have a job with Secret Detection and Container Scanning scans for the default branch + enabled: true + rules: + - type: pipeline + branches: + - main + actions: + - scan: secret_detection + - scan: sast + variables: + SAST_EXCLUDED_ANALYZERS: brakeman + - scan: container_scanning +- name: Enforce Cluster Image Scanning on production-cluster every 24h + description: This policy enforces Cluster Image Scanning scan to run every 24 hours + enabled: true + rules: + - type: schedule + cadence: "15 3 * * *" + clusters: + production-cluster: + containers: + - database + resources: + - production-application + namespaces: + - production-namespace + kinds: + - deployment + actions: + - scan: cluster_image_scanning +``` + +In this example: + +- For every pipeline executed on branches that match the `release/*` wildcard (for example, branch + `release/v1.2.1`), DAST scans run with `Scanner Profile A` and `Site Profile B`. +- DAST and secret detection scans run every 10 minutes. The DAST scan runs with `Scanner Profile C` + and `Site Profile D`. +- Secret detection, container scanning, and SAST scans run for every pipeline executed on the `main` + branch. The SAST scan runs with the `SAST_EXCLUDED_ANALYZER` variable set to `"brakeman"`. +- Cluster Image Scanning scan runs every 24h. The scan runs on the `production-cluster` cluster and fetches vulnerabilities + from the container with the name `database` configured for deployment with the name `production-application` in the `production-namespace` namespace. + +## Example for scan execution policy editor + +You can use this example in the YAML mode of the [scan execution policy editor](#scan-execution-policy-editor). +It corresponds to a single object from the previous example. + +```yaml +name: Enforce Secret Detection and Container Scanning in every default branch pipeline +description: This policy enforces pipeline configuration to have a job with Secret Detection and Container Scanning scans for the default branch +enabled: true +rules: + - type: pipeline + branches: + - main +actions: + - scan: secret_detection + - scan: container_scanning +``` diff --git a/doc/user/application_security/policies/scan-result-policies.md b/doc/user/application_security/policies/scan-result-policies.md new file mode 100644 index 00000000000..7857de8c780 --- /dev/null +++ b/doc/user/application_security/policies/scan-result-policies.md @@ -0,0 +1,175 @@ +--- +stage: Protect +group: Container Security +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Scan result policies **(ULTIMATE)** + +You can use scan result policies to take action based on scan results. For example, one type of scan +result policy is a security approval policy that allows approval to be required based on the +findings of one or more security scan jobs. Scan result policies are evaluated after a CI scanning +job is fully executed. + +## Scan result policy editor + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/77814) in GitLab 14.8 with a flag named `scan_result_policy`. Disabled by default. + +NOTE: +Only project Owners have the [permissions](../../permissions.md#project-members-permissions) +to select Security Policy Project. + +Once your policy is complete, save it by selecting **Create merge request** at the bottom of the +editor. This redirects you to the merge request on the project's configured security policy project. +If a security policy project doesn't link to your project, GitLab creates such a project for you. +Existing policies can also be removed from the editor interface by selecting **Delete policy** at +the bottom of the editor. + +All scan result policy changes are applied through a background job that runs once every 10 minutes. +Allow up to 10 minutes for any policy changes committed to this project to take effect. + +The policy editor only supports YAML mode. To follow work on Rule mode, see the epic +[Allow Users to Edit Rule-mode scan result policies in the Policy UI](https://gitlab.com/groups/gitlab-org/-/epics/5363). + + + +## Scan result policies schema + +The YAML file with scan result policies consists of an array of objects matching the scan result +policy schema nested under the `scan_result_policy` key. You can configure a maximum of five +policies under the `scan_result_policy` key. + +When you save a new policy, GitLab validates its contents against [this JSON schema](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/validators/json_schemas/security_orchestration_policy.json). +If you're not familiar with how to read [JSON schemas](https://json-schema.org/), +the following sections and tables provide an alternative. + +| Field | Type | Possible values | Description | +|-------|------|-----------------|-------------| +| `scan_result_policy` | `array` of Scan Result Policy | | List of scan result policies (maximum 5). | + +## Scan result policy schema + +| Field | Type | Possible values | Description | +|-------|------|-----------------|-------------| +| `name` | `string` | | Name of the policy. | +| `description` (optional) | `string` | | Description of the policy. | +| `enabled` | `boolean` | `true`, `false` | Flag to enable (`true`) or disable (`false`) the policy. | +| `rules` | `array` of rules | | List of rules that the policy applies. | +| `actions` | `array` of actions | | List of actions that the policy enforces. | + +## `scan_finding` rule type + +This rule enforces the defined actions based on the information provided. + +| Field | Type | Possible values | Description | +|------------|------|-----------------|-------------| +| `type` | `string` | `scan_finding` | The rule's type. | +| `branches` | `array` of `string` | `*` or the branch's name | The branch the given policy applies to (supports wildcard). | +| `scanners` | `array` of `string` | `sast`, `secret_detection`, `dependency_scanning`, `container_scanning`, `dast`, `coverage_fuzzing`, `api_fuzzing` | The security scanners for this rule to consider. | +| `vulnerabilities_allowed` | `integer` | Greater than or equal to zero | Number of vulnerabilities allowed before this rule is considered. | +| `severity_levels` | `array` of `string` | `info`, `unknown`, `low`, `medium`, `high`, `critical`| The severity levels for this rule to consider. | +| `vulnerability_states` | `array` of `string` | `newly_detected`, `detected`, `confirmed`, `resolved`, `dismissed` | The vulnerability states for this rule to consider when the target branch is set to the default branch. | + +## `require_approval` action type + +This action sets an approval rule to be required when conditions are met for at least one rule in +the defined policy. + +| Field | Type | Possible values | Description | +|-------|------|-----------------|-------------| +| `type` | `string` | `require_approval` | The action's type. | +| `approvals_required` | `integer` | Greater than or equal to zero | The number of MR approvals required. | +| `user_approvers` | `array` of `string` | Username of one of more users | The users to consider as approvers. | +| `user_approvers_ids` | `array` of `integer` | ID of one of more users | The IDs of users to consider as approvers. | +| `group_approvers` | `array` of `string` | Path of one of more groups | The groups to consider as approvers. | +| `group_approvers_ids` | `array` of `integer` | ID of one of more groups | The IDs of groups to consider as approvers. | + +Requirements and limitations: + +- You must add the respective [security scanning tools](../index.md#security-scanning-tools). + Otherwise, scan result policies won't have any effect. +- The maximum number of policies is five. +- Each policy can have a maximum of five rules. + +## Example security scan result policies project + +You can use this example in a `.gitlab/security-policies/policy.yml`, as described in +[Security policies project](index.md#security-policies-project): + +```yaml +--- +scan_result_policy: +- name: critical vulnerability CS approvals + description: critical severity level only for container scanning + enabled: true + rules: + - type: scan_finding + branches: + - main + scanners: + - container_scanning + vulnerabilities_allowed: 0 + severity_levels: + - critical + vulnerability_states: + - newly_detected + actions: + - type: require_approval + approvals_required: 1 + user_approvers: + - adalberto.dare +- name: secondary CS approvals + description: secondary only for container scanning + enabled: true + rules: + - type: scan_finding + branches: + - main + scanners: + - container_scanning + vulnerabilities_allowed: 1 + severity_levels: + - low + - unknown + vulnerability_states: + - newly_detected + actions: + - type: require_approval + approvals_required: 1 + user_approvers: + - sam.white +``` + +In this example: + +- Every MR that contains new `critical` vulnerabilities identified by container scanning requires + one approval from `alberto.dare`. +- Every MR that contains more than one new `low` or `unknown` vulnerability identified by container + scanning requires one approval from `sam.white`. + +## Example for Scan Result Policy editor + +You can use this example in the YAML mode of the [Scan Result Policy editor](#scan-result-policy-editor). +It corresponds to a single object from the previous example: + +```yaml +- name: critical vulnerability CS approvals + description: critical severity level only for container scanning + enabled: true + rules: + - type: scan_finding + branches: + - main + scanners: + - container_scanning + vulnerabilities_allowed: 1 + severity_levels: + - critical + vulnerability_states: + - newly_detected + actions: + - type: require_approval + approvals_required: 1 + user_approvers: + - adalberto.dare +``` diff --git a/doc/user/application_security/sast/index.md b/doc/user/application_security/sast/index.md index de2f9af82c7..3c0a2caf114 100644 --- a/doc/user/application_security/sast/index.md +++ b/doc/user/application_security/sast/index.md @@ -288,12 +288,14 @@ brakeman-sast: > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/235382) in GitLab 13.5. > - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/339614) support for > passthrough chains. Expanded to include additional passthrough types of `file`, `git`, and `url` in GitLab 14.6. +> - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/235359) support for overriding rules in GitLab 14.8. You can customize the default scanning rules provided by our SAST analyzers. -Ruleset customization supports two capabilities that can be used +Ruleset customization supports the following that can be used simultaneously: - [Disabling predefined rules](index.md#disable-predefined-analyzer-rules). Available for all analyzers. +- [Overriding predefined rules](index.md#override-predefined-analyzer-rules). Available for all analyzers. - Modifying the default behavior of a given analyzer by [synthesizing and passing a custom configuration](index.md#synthesize-a-custom-configuration). Available for only `nodejs-scan`, `gosec`, and `semgrep`. To customize the default scanning rules, create a file containing custom rules. These rules @@ -343,6 +345,50 @@ and `sobelow` by matching the `type` and `value` of identifiers: value = "sql_injection" ``` +#### Override predefined analyzer rules + +To override analyzer rules: + +1. In one or more `ruleset.identifier` subsections, list the rules that you want to override. Every `ruleset.identifier` section has: + + - a `type` field, to name the predefined rule identifier that the targeted analyzer uses. + - a `value` field, to name the rule to be overridden. + +1. In the `ruleset.override` context of a `ruleset` section, + provide the keys to override. Any combination of keys can be + overridden. Valid keys are: + + - description + - message + - name + - severity (valid options are: Critical, High, Medium, Low, Unknown, Info) + +##### Example: Override predefined rules of SAST analyzers + +In the following example, rules from `eslint` +and `gosec` are matched by the `type` and `value` of identifiers and +then overridden: + +```toml +[eslint] + [[eslint.ruleset]] + [eslint.ruleset.identifier] + type = "eslint_rule_id" + value = "security/detect-object-injection" + [eslint.ruleset.override] + description = "OVERRIDDEN description" + message = "OVERRIDDEN message" + name = "OVERRIDDEN name" + severity = "Critical" +[gosec] + [[gosec.ruleset]] + [gosec.ruleset.identifier] + type = "CWE" + value = "CWE-79" + [gosec.ruleset.override] + severity = "Critical" +``` + #### Synthesize a custom configuration To create a custom configuration, you can use passthrough chains. @@ -639,15 +685,33 @@ variables: ### Pre-compilation -If your project requires custom build configurations, it can be preferable to avoid -compilation during your SAST execution and instead pass all job artifacts from an -earlier stage in the pipeline. This is the current strategy when requiring -a `before_script` execution to prepare your scan job. +Most GitLab SAST analyzers directly scan your source code without compiling it first. +However, for technical reasons, some analyzers can only scan compiled code. + +By default, these analyzers automatically attempt to fetch dependencies and compile your code so it can be scanned. +Automatic compilation can fail if: + +- your project requires custom build configurations. +- you use language versions that aren't built into the analyzer. + +To resolve these issues, you can skip the analyzer's compilation step and directly provide artifacts from an earlier stage in your pipeline instead. +This strategy is called _pre-compilation_. + +Pre-compilation is available for the analyzers that support the `COMPILE` CI/CD variable. +See [Analyzer settings](#analyzer-settings) for the current list. + +To use pre-compilation: + +1. Output your project's dependencies to a directory in the project's working directory, then save that directory as an artifact by [setting the `artifacts: paths` configuration](../../../ci/yaml/index.md#artifactspaths). +1. Provide the `COMPILE: "false"` CI/CD variable to the analyzer to disable automatic compilation. +1. Add your compilation stage as a dependency for the analyzer job. + +To allow the analyzer to recognize the compiled artifacts, you must explicitly specify the path to +the vendored directory. +This configuration can vary per analyzer. For Maven projects, you can use `MAVEN_REPO_PATH`. +See [Analyzer settings](#analyzer-settings) for the complete list of available options. -To pass your project's dependencies as artifacts, the dependencies must be included -in the project's working directory and specified using the `artifacts:path` configuration. -If all dependencies are present, the `COMPILE=false` CI/CD variable can be provided to the -analyzer and compilation is skipped: +The following example pre-compiles a Maven project and provides it to the SpotBugs SAST analyzer: ```yaml stages: @@ -678,11 +742,6 @@ spotbugs-sast: sast: gl-sast-report.json ``` -To allow the analyzer to recognize the compiled artifacts, you must explicitly specify the path to -the vendored directory. This configuration can vary per analyzer but in the case of Java above, you -can use `MAVEN_REPO_PATH`. See -[Analyzer settings](#analyzer-settings) for the complete list of available options. - ### Available CI/CD variables SAST can be configured using the [`variables`](../../../ci/yaml/index.md#variables) parameter in @@ -779,7 +838,7 @@ Some analyzers can be customized with CI/CD variables. | `SAST_GOSEC_CONFIG` | Gosec | **{warning}** **[Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/328301)** in GitLab 14.0 - use custom rulesets instead. Path to configuration for Gosec (optional). | | `PHPCS_SECURITY_AUDIT_PHP_EXTENSIONS` | phpcs-security-audit | Comma separated list of additional PHP Extensions. | | `SAST_DISABLE_BABEL` | NodeJsScan | **{warning}** **[Removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/64025)** in GitLab 13.5 | -| `SAST_SEMGREP_METRICS` | Semgrep | Set to `"false"` to disable sending anonymized scan metrics to [r2c](https://r2c.dev/). Default: `true`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/330565) in GitLab 14.0. | +| `SAST_SEMGREP_METRICS` | Semgrep | Set to `"false"` to disable sending anonymized scan metrics to [r2c](https://r2c.dev/). Default: `true`. Introduced in GitLab 14.0 from the [confidential issue](../../project/issues/confidential_issues.md) `https://gitlab.com/gitlab-org/gitlab/-/issues/330565`. | #### Custom CI/CD variables @@ -819,86 +878,18 @@ variables: ## Reports JSON format -The SAST tool emits a JSON report file. For more information, see the -[schema for this report](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/sast-report-format.json). - -The JSON report file can be downloaded from the CI pipelines page, or the -pipelines tab on merge requests by [setting `artifacts: paths`](../../../ci/yaml/index.md#artifactspaths) to `gl-sast-report.json`. For more information see [Downloading artifacts](../../../ci/pipelines/job_artifacts.md). - -Here's an example SAST report: - -```json-doc -{ - "version": "2.0", - "vulnerabilities": [ - { - "id": "9e96e0ab-23da-4d7d-a09e-0acbaa5e83ca", - "category": "sast", - "name": "Predictable pseudorandom number generator", - "message": "Predictable pseudorandom number generator", - "description": "The use of java.util.Random is predictable", - "severity": "Medium", - "confidence": "Medium", - "scanner": { - "id": "find_sec_bugs", - "name": "Find Security Bugs" - }, - "location": { - "file": "groovy/src/main/groovy/com/gitlab/security_products/tests/App.groovy", - "start_line": 47, - "end_line": 47, - "class": "com.gitlab.security_products.tests.App", - "method": "generateSecretToken2", - "dependency": { - "package": {} - } - }, - "identifiers": [ - { - "type": "find_sec_bugs_type", - "name": "Find Security Bugs-PREDICTABLE_RANDOM", - "value": "PREDICTABLE_RANDOM", - "url": "https://find-sec-bugs.github.io/bugs.htm#PREDICTABLE_RANDOM" - }, - { - "type": "cwe", - "name": "CWE-330", - "value": "330", - "url": "https://cwe.mitre.org/data/definitions/330.html" - } - ] - }, - { - "id": "e6dbf91f-4c07-46f7-a365-0169489c27d1", - "category": "sast", - "message": "Probable insecure usage of temp file/directory.", - "severity": "Medium", - "confidence": "Medium", - "scanner": { - "id": "bandit", - "name": "Bandit" - }, - "location": { - "file": "python/hardcoded/hardcoded-tmp.py", - "start_line": 10, - "end_line": 10, - "dependency": { - "package": {} - } - }, - "identifiers": [ - { - "type": "bandit_test_id", - "name": "Bandit Test ID B108", - "value": "B108", - "url": "https://docs.openstack.org/bandit/latest/plugins/b108_hardcoded_tmp_directory.html" - } - ] - }, - ], - "remediations": [] -} -``` +SAST outputs a report file in JSON format. The report file contains details of all found vulnerabilities. +To download the report file, you can either: + +- Download the file from the CI/CD pipelines page. +- In the pipelines tab on merge requests, set [`artifacts: paths`](../../../ci/yaml/index.md#artifactspaths) to `gl-sast-report.json`. + +For information, see [Download job artifacts](../../../ci/pipelines/job_artifacts.md#download-job-artifacts). + +For details of the report file's schema, see +[SAST report file schema](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/sast-report-format.json). + +For an example SAST report file, see [`gl-secret-detection-report.json`](https://gitlab.com/gitlab-org/security-products/analyzers/secrets/-/blob/master/qa/expect/secrets/gl-secret-detection-report.json) example. ## Running SAST in an offline environment @@ -945,7 +936,7 @@ registry.gitlab.com/security-products/sast/spotbugs:2 The process for importing Docker images into a local offline Docker registry depends on **your network security policy**. Please consult your IT staff to find an accepted and approved -process by which external resources can be imported or temporarily accessed. These scanners are [periodically updated](../vulnerabilities/index.md#vulnerability-scanner-maintenance) +process by which external resources can be imported or temporarily accessed. These scanners are [periodically updated](../index.md#vulnerability-scanner-maintenance) with new definitions, and you may be able to make occasional updates on your own. For details on saving and transporting Docker images as a file, see Docker's documentation on diff --git a/doc/user/application_security/secret_detection/index.md b/doc/user/application_security/secret_detection/index.md index c5761a5743f..2ce2d59898f 100644 --- a/doc/user/application_security/secret_detection/index.md +++ b/doc/user/application_security/secret_detection/index.md @@ -63,7 +63,7 @@ as shown in the following table: | [Configure Secret Detection Scanners](#configuration) | **{check-circle}** | **{check-circle}** | | [Customize Secret Detection Settings](#customizing-settings) | **{check-circle}** | **{check-circle}** | | View [JSON Report](../sast/index.md#reports-json-format) | **{check-circle}** | **{check-circle}** | -| Presentation of JSON Report in Merge Request | **{dotted-circle}** | **{check-circle}** | +| Presentation of JSON Report in merge request | **{dotted-circle}** | **{check-circle}** | | View identified secrets in the pipelines' **Security** tab | **{dotted-circle}** | **{check-circle}** | | [Interaction with Vulnerabilities](../vulnerabilities/index.md) | **{dotted-circle}** | **{check-circle}** | | [Access to Security Dashboard](../security_dashboard/index.md) | **{dotted-circle}** | **{check-circle}** | @@ -182,14 +182,89 @@ Secret Detection can be customized by defining available CI/CD variables: > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211387) in GitLab 13.5. > - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/339614) support for > passthrough chains. Expanded to include additional passthrough types of `file`, `git`, and `url` in GitLab 14.6. +> - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/235359) support for overriding rules in GitLab 14.8. You can customize the default secret detection rules provided with GitLab. +Ruleset customization supports the following capabilities that can be used +simultaneously: + +- [Disabling predefined rules](index.md#disable-predefined-analyzer-rules). +- [Overriding predefined rules](index.md#override-predefined-analyzer-rules). +- Modifying the default behavior of the Secret Detection analyzer by [synthesizing and passing a custom configuration](index.md#synthesize-a-custom-configuration). Available for only `nodejs-scan`, `gosec`, and `semgrep`. + Customization allows replacing the default secret detection rules with rules that you define. To create a custom ruleset: 1. Create a `.gitlab` directory at the root of your project, if one doesn't already exist. 1. Create a custom ruleset file named `secret-detection-ruleset.toml` in the `.gitlab` directory. + +#### Disable predefined analyzer rules + +To disable analyzer rules: + +1. Set the `disabled` flag to `true` in the context of a `ruleset` section. + +1. In one or more `ruleset.identifier` subsections, list the rules that you want disabled. Every `ruleset.identifier` section has: + + - a `type` field, to name the predefined rule identifier. + - a `value` field, to name the rule to be disabled. + +##### Example: Disable predefined rules of Secret Detection analyzer + +In the following example, the disabled rules is assigned to `secrets` +by matching the `type` and `value` of identifiers: + +```toml +[secrets] + [[secrets.ruleset]] + disable = true + [secrets.ruleset.identifier] + type = "gitleaks_rule_id" + value = "RSA private key" +``` + +#### Override predefined analyzer rules + +To override rules: + +1. In one or more `ruleset.identifier` subsections, list the rules that you want to override. Every `ruleset.identifier` section has: + + - a `type` field, to name the predefined rule identifier that the Secret Detection analyzer uses. + - a `value` field, to name the rule to be overridden. + +1. In the `ruleset.override` context of a `ruleset` section, + provide the keys to override. Any combination of keys can be + overridden. Valid keys are: + + - description + - message + - name + - severity (valid options are: Critical, High, Medium, Low, Unknown, Info) + +##### Example: Override predefined rules of Secret Detection analyzer + +In the following example, rules +are matched by the `type` and `value` of identifiers and +then overridden: + +```toml +[secrets] + [[secrets.ruleset]] + [secrets.ruleset.identifier] + type = "gitleaks_rule_id" + value = "RSA private key" + [secrets.ruleset.override] + description = "OVERRIDDEN description" + message = "OVERRIDDEN message" + name = "OVERRIDDEN name" + severity = "Info" +``` + +#### Synthesize a custom configuration + +To create a custom configuration, you can use passthrough chains. + 1. In the `secret-detection-ruleset.toml` file, do one of the following: - Define a custom ruleset: @@ -239,31 +314,8 @@ From highest to lowest severity, the logging levels are: ## Post-processing and revocation -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4639) in GitLab 13.6. - -Upon detection of a secret, GitLab supports post-processing hooks. These can be used to take actions like notifying the cloud service who issued the secret. The cloud provider can confirm the credentials and take remediation actions like revoking or reissuing a new secret and notifying the creator of the secret. Post-processing workflows vary by supported cloud providers. - -GitLab currently supports post-processing for following service providers: - -- Amazon Web Services (AWS) - -Third party cloud and SaaS providers can [express integration interest by filling out this form](https://forms.gle/wWpvrtLRK21Q2WJL9). Learn more about the [technical details of post-processing secrets](https://gitlab.com/groups/gitlab-org/-/epics/4639). - -NOTE: -Post-processing is currently limited to a project's default branch, see the above epic for future efforts to support additional branches. - -```mermaid -sequenceDiagram - autonumber - Rails->>+Sidekiq: gl-secret-detection-report.json - Sidekiq-->+Sidekiq: Ci::BuildFinishedWorker - Sidekiq-->+RevocationAPI: GET revocable keys types - RevocationAPI-->>-Sidekiq: OK - Sidekiq->>+RevocationAPI: POST revoke revocable keys - RevocationAPI-->>-Sidekiq: ACCEPTED - RevocationAPI-->>+Cloud Vendor: revoke revocable keys - Cloud Vendor-->>+RevocationAPI: ACCEPTED -``` +Upon detection of a secret, GitLab SaaS supports post-processing hooks. +For more information, see [Post-processing and revocation](post_processing.md). ## Full History Secret Detection @@ -316,7 +368,7 @@ registry.gitlab.com/security-products/secret-detection:3 The process for importing Docker images into a local offline Docker registry depends on **your network security policy**. Please consult your IT staff to find an accepted and approved -process by which external resources can be imported or temporarily accessed. These scanners are [periodically updated](../vulnerabilities/index.md#vulnerability-scanner-maintenance) +process by which external resources can be imported or temporarily accessed. These scanners are [periodically updated](../index.md#vulnerability-scanner-maintenance) with new definitions, and you may be able to make occasional updates on your own. For details on saving and transporting Docker images as a file, see Docker's documentation on @@ -370,7 +422,7 @@ For information on this, see the [general Application Security troubleshooting s ### Error: `Couldn't run the gitleaks command: exit status 2` -If a pipeline is triggered from a Merge Request containing 60 commits while the `GIT_DEPTH` variable's +If a pipeline is triggered from a merge request containing 60 commits while the `GIT_DEPTH` variable's value is less than that, the Secret Detection job fails as the clone is not deep enough to contain all of the relevant commits. For information on the current default value, see the [pipeline configuration documentation](../../../ci/pipelines/settings.md#limit-the-number-of-changes-fetched-during-clone). @@ -395,3 +447,12 @@ secret_detection: variables: GIT_DEPTH: 100 ``` + +### `secret-detection` job fails with `ERR fatal: ambiguous argument` message + +Your `secret-detection` job can fail with `ERR fatal: ambiguous argument` error if your +repository's default branch is unrelated to the branch the job was triggered for. +See issue [!352014](https://gitlab.com/gitlab-org/gitlab/-/issues/352014) for more details. + +To resolve the issue, make sure to correctly [set your default branch](../../project/repository/branches/default.md#change-the-default-branch-name-for-a-project) on your repository. You should set it to a branch +that has related history with the branch you run the `secret-detection` job on. diff --git a/doc/user/application_security/secret_detection/post_processing.md b/doc/user/application_security/secret_detection/post_processing.md new file mode 100644 index 00000000000..972558c3b95 --- /dev/null +++ b/doc/user/application_security/secret_detection/post_processing.md @@ -0,0 +1,179 @@ +--- +stage: Secure +group: Static Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Secret Detection post-processing and revocation **(FREE SAAS)** + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4639) in GitLab 13.6. + +GitLab supports running post-processing hooks after detecting a secret. These +hooks can perform actions, like notifying the cloud service that issued the secret. +The cloud provider can then confirm the credentials and take remediation actions, like: + +- Revoking a secret. +- Reissuing a secret. +- Notifying the creator of the secret. + +GitLab SaaS supports post-processing for Amazon Web Services (AWS). +Post-processing workflows vary by supported cloud providers. + +Post-processing is limited to a project's default branch. The epic +[Post-processing of leaked secrets](https://gitlab.com/groups/gitlab-org/-/epics/4639). +contains: + +- Technical details of post-processing secrets. +- Discussions of efforts to support additional branches. + +NOTE: +Post-processing is currently limited to a project's default branch + +## High-level architecture + +This diagram describes how a post-processing hook revokes a secret within the GitLab application: + +```mermaid +sequenceDiagram + autonumber + GitLab Rails->>+Sidekiq: gl-secret-detection-report.json + Sidekiq-->+Sidekiq: StoreSecurityReportsWorker + Sidekiq-->+RevocationAPI: GET revocable keys types + RevocationAPI-->>-Sidekiq: OK + Sidekiq->>+RevocationAPI: POST revoke revocable keys + RevocationAPI-->>-Sidekiq: ACCEPTED + RevocationAPI-->>+Cloud Vendor: revoke revocable keys + Cloud Vendor-->>+RevocationAPI: ACCEPTED +``` + +## Integrate your cloud provider service with GitLab Saas + +Third party cloud and SaaS providers can [express integration interest by filling out this form](https://forms.gle/wWpvrtLRK21Q2WJL9). + +### Implement a vendor revocation receiver service + +A vendor revocation receiver service integrates with a GitLab instance to receive +a web notification and respond to leaked token requests. + +To implement a receiver service to revoke leaked tokens: + +1. Create a publicly accessible HTTP service matching the corresponding API contract + below. Your service should be idempotent and rate-limited. +1. When a pipeline corresponding to its revocable token type (in the example, `my_api_token`) + completes, GitLab sends a request to your receiver service. +1. The included URL should be publicly accessible, and contain the commit where the + leaked token can be found. For example: + + ```plaintext + POST / HTTP/2 + Accept: */* + Content-Type: application/json + X-Gitlab-Token: MYSECRETTOKEN + + [ + {"type": "my_api_token", "token":"XXXXXXXXXXXXXXXX","url": "https://example.com/some-repo/blob/abcdefghijklmnop/compromisedfile1.java"} + ] + ``` + +## Implement a revocation service for self-managed + +Self-managed instances interested in using the revocation capabilities must: + +- Deploy the [RevocationAPI](#high-level-architecture). +- Configure the GitLab instance to use the RevocationAPI. + +A RevocationAPI must: + +- Match a minimal API specification. +- Provide two endpoints: + - Fetching revocable token types. + - Revoking leaked tokens. +- Be rate-limited and idempotent. + +Requests to the documented endpoints are authenticated via API tokens passed in +the `Authorization` header. Request and response bodies, if present, are +expected to have the content type `application/json`. + +All endpoints may return these responses: + +- `401 Unauthorized` +- `405 Method Not Allowed` +- `500 Internal Server Error` + +### `GET /v1/revocable_token_types` + +Returns the valid `type` values for use in the `revoke_tokens` endpoint. + +NOTE: +These values match the concatenation of [the `secrets` analyzer's](index.md) +[primary identifier](../../../development/integrations/secure.md#identifiers) by means +of concatenating the `primary_identifier.type` and `primary_identifier.value`. +In the case below, a finding identifier matches: + +```json +{"type": "gitleaks_rule_id", "name": "Gitleaks rule ID GitLab Personal Access Token", "value": "GitLab Personal Access Token"} +``` + +| Status Code | Description | +| ----- | --- | +| `200` | The response body contains the valid token `type` values. | + +Example response body: + +```json +{ + "types": ["gitleaks_rule_id_gitlab_personal_access_token"] +} +``` + +### `POST /v1/revoke_tokens` + +Accepts a list of tokens to be revoked by the appropriate provider. + +| Status Code | Description | +| ----- | --- | +| `204` | All submitted tokens have been accepted for eventual revocation. | +| `400` | The request body is invalid or one of the submitted token types is not supported. The request should not be retried. | +| `429` | The provider has received too many requests. The request should be retried later. | + +Example request body: + +```json +[{ + "type": "gitleaks_rule_id_gitlab_personal_access_token", + "token": "glpat--8GMtG8Mf4EnMJzmAWDU", + "location": "https://example.com/some-repo/blob/abcdefghijklmnop/compromisedfile1.java" +}, +{ + "type": "gitleaks_rule_id_gitlab_personal_access_token", + "token": "glpat--tG84EGK33nMLLDE70zU", + "location": "https://example.com/some-repo/blob/abcdefghijklmnop/compromisedfile2.java" +}] +``` + +### Configure GitLab to interface with RevocationAPI + +You must configure the following database settings in the GitLab instance: + +- `secret_detection_token_revocation_enabled` +- `secret_detection_token_revocation_url` +- `secret_detection_token_revocation_token` +- `secret_detection_revocation_token_types_url` + +For example, to configure these values in the +[Rails console](../../../administration/operations/rails_console.md#starting-a-rails-console-session): + +```ruby +::Gitlab::CurrentSettings.update!(secret_detection_token_revocation_token: 'MYSECRETTOKEN') +::Gitlab::CurrentSettings.update!(secret_detection_token_revocation_url: 'https://example.gitlab.com/revocation_service/v1/revoke_tokens') +::Gitlab::CurrentSettings.update!(secret_detection_revocation_token_types_url: 'https://example.gitlab.com/revocation_service/v1/revocable_token_types') +::Gitlab::CurrentSettings.update!(secret_detection_token_revocation_enabled: true) +``` + +After you configure these values, completing a pipeline performs these actions: + +1. The revocation service is triggered once. +1. A request is made to `secret_detection_revocation_token_types_url` to fetch a + list of revocable tokens. +1. Any Secret Detection findings matching the results of the `token_types` request + are included in the subsequent revocation request. diff --git a/doc/user/application_security/security_dashboard/index.md b/doc/user/application_security/security_dashboard/index.md index 937ca33c3a1..a0cfd6d9d77 100644 --- a/doc/user/application_security/security_dashboard/index.md +++ b/doc/user/application_security/security_dashboard/index.md @@ -75,7 +75,7 @@ To view the total number of vulnerabilities per scan: Depending on the type of security scanner, you can download: -- A JSON artifact that contains the security scanner [report]('https://docs.gitlab.com/ee/development/integrations/secure.html#report'). +- A JSON artifact that contains the security scanner [report](../../../development/integrations/secure.md#report). - A CSV file that contains URLs and endpoints scanned by the security scanner. To download a security scan output: diff --git a/doc/user/application_security/threat_monitoring/index.md b/doc/user/application_security/threat_monitoring/index.md index ae5f6ba0fe1..390882d4326 100644 --- a/doc/user/application_security/threat_monitoring/index.md +++ b/doc/user/application_security/threat_monitoring/index.md @@ -7,7 +7,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Threat Monitoring **(ULTIMATE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14707) in GitLab 12.9. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14707) in GitLab 12.9. +> - [Deprecated](https://gitlab.com/groups/gitlab-org/-/epics/7476) in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) in GitLab 15.0. + +WARNING: +Threat Monitoring is in its end-of-life process. It's [deprecated](https://gitlab.com/groups/gitlab-org/-/epics/7476) +for use in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) +in GitLab 15.0. The **Threat Monitoring** page provides alerts and metrics for the GitLab application runtime security features. You can access diff --git a/doc/user/application_security/vulnerabilities/index.md b/doc/user/application_security/vulnerabilities/index.md index 9aa8a0cd3cd..7b39002bac3 100644 --- a/doc/user/application_security/vulnerabilities/index.md +++ b/doc/user/application_security/vulnerabilities/index.md @@ -1,5 +1,4 @@ --- -type: reference, howto stage: Secure group: Threat Insights info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments @@ -160,23 +159,3 @@ To manually apply the patch that GitLab generated for a vulnerability: 1. Ensure your local project has the same commit checked out that was used to generate the patch. 1. Run `git apply remediation.patch`. 1. Verify and commit the changes to your branch. - -## Vulnerability scanner maintenance - -The following vulnerability scanners and their databases are regularly updated: - -| Secure scanning tool | Vulnerabilities database updates | -|:----------------------------------------------------------------|----------------------------------| -| [Container Scanning](../container_scanning/index.md) | A job runs on a daily basis to build new images with the latest vulnerability database updates from the upstream scanner. | -| [Dependency Scanning](../dependency_scanning/index.md) | Relies on `bundler-audit` (for Ruby gems), `retire.js` (for npm packages), and `gemnasium` (the GitLab tool for all libraries). Both `bundler-audit` and `retire.js` fetch their vulnerabilities data from GitHub repositories, so vulnerabilities added to `ruby-advisory-db` and `retire.js` are immediately available. The tools themselves are updated once per month if there's a new version. The [Gemnasium DB](https://gitlab.com/gitlab-org/security-products/gemnasium-db) is updated on a daily basis using [data from NVD, the `ruby-advisory-db` and the GitHub Security Advisory Database as data sources](https://gitlab.com/gitlab-org/security-products/gemnasium-db/-/blob/master/SOURCES.md). See our [current measurement of time from CVE being issued to our product being updated](https://about.gitlab.com/handbook/engineering/development/performance-indicators/#cve-issue-to-update). | -| [Dynamic Application Security Testing (DAST)](../dast/index.md) | The scanning engine is updated on a periodic basis. See the [version of the underlying tool `zaproxy`](https://gitlab.com/gitlab-org/security-products/dast/blob/main/Dockerfile#L1). The scanning rules are downloaded at scan runtime. | -| [Static Application Security Testing (SAST)](../sast/index.md) | Relies exclusively on [the tools GitLab wraps](../sast/index.md#supported-languages-and-frameworks). The underlying analyzers are updated at least once per month if a relevant update is available. The vulnerabilities database is updated by the upstream tools. | - -You do not have to update GitLab to benefit from the latest vulnerabilities definitions. -The security tools are released as Docker images. The vendored job definitions that enable them use -major release tags according to [semantic versioning](https://semver.org/). Each new release of the -tools overrides these tags. -The Docker images are updated to match the previous GitLab releases. Although -you automatically get the latest versions of the scanning tools, -there are some [known issues](https://gitlab.com/gitlab-org/gitlab/-/issues/9725) -with this approach. diff --git a/doc/user/application_security/vulnerabilities/severities.md b/doc/user/application_security/vulnerabilities/severities.md index f3e8e98bce3..89464064ea3 100644 --- a/doc/user/application_security/vulnerabilities/severities.md +++ b/doc/user/application_security/vulnerabilities/severities.md @@ -35,10 +35,10 @@ the following tables: ## SAST -| GitLab analyzer | Outputs severity levels? | Native severity level type | Native severity level example | -|--------------------------------------------------------------------------------------------------------|--------------------------|----------------------------|------------------------------------| -| [`security-code-scan`](https://gitlab.com/gitlab-org/security-products/analyzers/security-code-scan) | **{dotted-circle}** No | N/A | N/A | -| [`brakeman`](https://gitlab.com/gitlab-org/security-products/analyzers/brakeman) | **{dotted-circle}** No | N/A | N/A | +| GitLab analyzer | Outputs severity levels? | Native severity level type | Native severity level example | +|----------------------------------------------------------------------------------------------------------|--------------------------|----------------------------|------------------------------------| +| [`security-code-scan`](https://gitlab.com/gitlab-org/security-products/analyzers/security-code-scan) | **{check-circle}** Yes | String | `CRITICAL`, `HIGH`, `MEDIUM` in [analyzer version 3.2.0 and later](https://gitlab.com/gitlab-org/security-products/analyzers/security-code-scan/-/blob/master/CHANGELOG.md#v320). In earlier versions, hardcoded to `Unknown`. | +| [`brakeman`](https://gitlab.com/gitlab-org/security-products/analyzers/brakeman) | **{check-circle}** Yes | String | `HIGH`, `MEDIUM`, `LOW` | | [`sobelow`](https://gitlab.com/gitlab-org/security-products/analyzers/sobelow) | **{check-circle}** Yes | N/A | Hardcodes all severity levels to `Unknown` | | [`nodejs-scan`](https://gitlab.com/gitlab-org/security-products/analyzers/nodejs-scan) | **{check-circle}** Yes | String | `INFO`, `WARNING`, `ERROR` | | [`flawfinder`](https://gitlab.com/gitlab-org/security-products/analyzers/flawfinder) | **{check-circle}** Yes | Integer | `0`, `1`, `2`, `3`, `4`, `5` | diff --git a/doc/user/application_security/vulnerability_report/index.md b/doc/user/application_security/vulnerability_report/index.md index 05ff5c511f9..ba1455ab70a 100644 --- a/doc/user/application_security/vulnerability_report/index.md +++ b/doc/user/application_security/vulnerability_report/index.md @@ -7,7 +7,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Vulnerability Report **(ULTIMATE)** -The Vulnerability Report provides information about vulnerabilities from scans of the default branch. It is available for projects, groups, and the Security Center. +The Vulnerability Report provides information about vulnerabilities from scans of the default branch. It contains cumulative results of all successful jobs, regardless of whether the pipeline was successful. + +The scan results from a pipeline are only ingested after all the jobs in the pipeline complete. Partial results for a pipeline with jobs in progress can be seen in the pipeline security tab. + +The report is available for projects, groups, and the Security Center. At all levels, the Vulnerability Report contains: diff --git a/doc/user/asciidoc.md b/doc/user/asciidoc.md index da75c008ed1..80c85358fcb 100644 --- a/doc/user/asciidoc.md +++ b/doc/user/asciidoc.md @@ -232,6 +232,11 @@ v1.0, 2019-01-01 #### Includes +NOTE: +[Wiki pages](project/wiki/index.md#create-a-new-wiki-page) created with the AsciiDoc +format are saved with the file extension `.asciidoc`. When working with AsciiDoc wiki +pages, change the file name from `.adoc` to `.asciidoc`. + ```plaintext include::basics.adoc[] diff --git a/doc/user/clusters/agent/ci_cd_tunnel.md b/doc/user/clusters/agent/ci_cd_tunnel.md index f1c49b87383..5fe772d9686 100644 --- a/doc/user/clusters/agent/ci_cd_tunnel.md +++ b/doc/user/clusters/agent/ci_cd_tunnel.md @@ -12,53 +12,52 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Moved](https://gitlab.com/groups/gitlab-org/-/epics/6290) to GitLab Free in 14.5. > - Support for Omnibus installations was [introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5686) in GitLab 14.5. -The CI/CD Tunnel enables users to access Kubernetes clusters from GitLab CI/CD jobs even if there is no network -connectivity between GitLab Runner and a cluster. GitLab Runner does not have to be running in the same cluster. +To use GitLab CI/CD to safely deploy your application to a cluster, you can use the CI/CD Tunnel. -Only CI/CD jobs set in the configuration project can access one of the configured agents. +You can authorize multiple projects to access the same cluster, so you +can keep your application's codebase in one repository and configure +your cluster in another. This method is scalable and can save you resources. -## Prerequisites - -- An existing Kubernetes cluster. -- An agent [installed on your cluster](install/index.md#install-the-agent-into-the-cluster). +To ensure access to your cluster is safe, only the projects you +authorize can access your Agent through the CI/CD Tunnel. -## Use the CI/CD Tunnel to run Kubernetes commands from GitLab CI/CD - -If your project has access to one or more Agent records available, its CI/CD -jobs provide a `KUBECONFIG` variable compatible with `kubectl`. +## Prerequisites -Also, each Agent has a separate context (`kubecontext`). By default, -there isn't any context selected. -Contexts are named in the following format: `<agent-configuration-project-path>:<agent-name>`. -To get the list of available contexts, run `kubectl config get-contexts`. +To use the CI/CD Tunnel, you need an existing Kubernetes cluster connected to GitLab through the +[GitLab Agent](install/index.md#install-the-agent-onto-the-cluster). -## Share the CI/CD Tunnel provided by an Agent with other projects and groups +To run your CI/CD jobs using the CI/CD Tunnel, you do not need to have a runner in the same cluster. -The Agent can be configured to enable access to the CI/CD Tunnel to other projects or all the projects under a given group. This way you can have a single agent serving all the requests for several projects saving on resources and maintenance. +## How the CI/CD Tunnel works -You can read more on how to [authorize access in the Agent configuration reference](repository.md#authorize-projects-and-groups-to-use-an-agent). +When you authorize a project to use an Agent, the Tunnel automatically +injects a `KUBECONFIG` variable into its CI/CD jobs. This way, you can +run `kubectl` commands from GitLab CI/CD scripts that belong to the +authorized project. -## Restrict access of authorized projects and groups **(PREMIUM)** +When you authorize a group, all the projects that belong to that group +become authorized to access the selected Agent. -You can [configure various impersonations](repository.md#use-impersonation-to-restrict-project-and-group-access) to restrict the permissions of a shared CI/CD Tunnel. +An Agent can only authorize projects or groups in the same group +hierarchy as the Agent's configuration project. You can authorize +up to 100 projects and 100 groups per Agent. -## Example for a `kubectl` command using the CI/CD Tunnel +Also, each Agent has a separate context (`kubecontext`). +The Tunnel uses this information to safely allow access to the cluster from +jobs running in the projects you authorized. -The following example shows a CI/CD job that runs a `kubectl` command using the CI/CD Tunnel. -You can run any Kubernetes-specific commands similarly, such as `kubectl`, `helm`, -`kpt`, and so on. To do so: +### `~/.kube/cache` permissions + +`kubectl` and other tools based on the same libraries (such as Helm, `kpt`, and `kustomize`) cache information about +the cluster in `~/.kube/cache`. If this directory is not writable, the tool fetches information on each invocation, +making interactions slower and creating unnecessary load on the cluster. Make sure that this directory in the container image +you use is writable for the best experience. -1. Set your Agent's context in the first command with the format `<agent-configuration-project-path>:<agent-name>`. -1. Run Kubernetes commands. +## Configure the CI/CD Tunnel -For example: +The CI/CD Tunnel is configured directly through the +Agent's configuration file ([`config.yaml`](repository.md)) to: -```yaml - deploy: - image: - name: bitnami/kubectl:latest - entrypoint: [""] - script: - - kubectl config use-context path/to/agent-configuration-project:your-agent-name - - kubectl get pods -``` +- Authorize [projects](repository.md#authorize-projects-to-use-an-agent) and [groups](repository.md#authorize-groups-to-use-an-agent) to use the same Agent. +- [Run `kubectl` commands using the CI/CD Tunnel](repository.md#run-kubectl-commands-using-the-cicd-tunnel). +- [Restrict access of authorized projects and groups through impersonation strategies](repository.md#use-impersonation-to-restrict-project-and-group-access). diff --git a/doc/user/clusters/agent/index.md b/doc/user/clusters/agent/index.md index a235c0ef6f8..3fb141e402f 100644 --- a/doc/user/clusters/agent/index.md +++ b/doc/user/clusters/agent/index.md @@ -4,60 +4,71 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab Agent for Kubernetes **(FREE)** +# Connecting a Kubernetes cluster with GitLab > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/223061) in GitLab 13.4. > - Support for `grpcs` [introduced](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/issues/7) in GitLab 13.6. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/300960) in GitLab 13.10, KAS became available on GitLab.com under `wss://kas.gitlab.com` through an Early Adopter Program. -> - Introduced in GitLab 13.11, the GitLab Agent became available to every project on GitLab.com. +> - Agent Server [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/300960) on GitLab.com under `wss://kas.gitlab.com` through an Early Adopter Program in GitLab 13.10. +> - The agent became available to every project on GitLab.com in GitLab 13.11. > - [Moved](https://gitlab.com/groups/gitlab-org/-/epics/6290) from GitLab Premium to GitLab Free in 14.5. -> - [Renamed](https://gitlab.com/groups/gitlab-org/-/epics/7167) from "GitLab Kubernetes Agent" to "GitLab Agent for Kubernetes" in GitLab 14.6. +> - [Renamed](https://gitlab.com/groups/gitlab-org/-/epics/7167) from "GitLab Kubernetes Agent" to "GitLab agent for Kubernetes" in GitLab 14.6. -The [GitLab Agent for Kubernetes](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent) ("Agent", for short) -is an active in-cluster component for connecting Kubernetes clusters to GitLab safely to support cloud-native deployment, management, and monitoring. +You can connect your Kubernetes cluster with GitLab to deploy, manage, +and monitor your cloud-native solutions. You can choose from two primary workflows. -The Agent is installed into the cluster through code, providing you with a fast, safe, stable, and scalable solution. +In a **GitOps workflow**, you keep your Kubernetes manifests in GitLab. You install a GitLab agent in your cluster, and +any time you update your manifests, the agent updates the cluster. This workflow is fully driven with Git and is considered pull-based, +because the cluster is pulling updates from your GitLab repository. + +In a **CI/CD** workflow, you use GitLab CI/CD to query and update your cluster by using the Kubernetes API. +This workflow is considered push-based, because GitLab is pushing requests from GitLab CI/CD to your cluster. + +Both of these workflows require you to [install an agent in your cluster](install/index.md). + +## Supported cluster versions + +GitLab supports the following Kubernetes versions. You can upgrade your +Kubernetes version to a supported version at any time: + +- 1.20 (support ends on July 22, 2022) +- 1.19 (support ends on February 22, 2022) +- 1.18 (support ends on November 22, 2021) +- 1.17 (support ends on September 22, 2021) + +GitLab supports at least two production-ready Kubernetes minor +versions at any given time. GitLab regularly reviews the supported versions and +provides a three-month deprecation period before removing support for a specific +version. The list of supported versions is based on: + +- The versions supported by major managed Kubernetes providers. +- The versions [supported by the Kubernetes community](https://kubernetes.io/releases/version-skew-policy/#supported-versions). + +[This epic](https://gitlab.com/groups/gitlab-org/-/epics/4827) tracks support for other Kubernetes versions. + +Some GitLab features might work on versions not listed here. + +## Using Kubernetes with GitOps **(PREMIUM)** With GitOps, you can manage containerized clusters and applications from a Git repository that: - Is the single source of truth of your system. - Is the single place where you operate your system. -- Is a single resource to monitor your system. -By combining GitLab, Kubernetes, and GitOps, it results in a robust infrastructure: +By combining GitLab, Kubernetes, and GitOps, you can have: - GitLab as the GitOps operator. - Kubernetes as the automation and convergence system. -- GitLab CI/CD as the Continuous Integration and Continuous Deployment engine. +- GitLab CI/CD for Continuous Integration and the agent for Continuous Deployment. Beyond that, you can use all the features offered by GitLab as the all-in-one DevOps platform for your product and your team. -## Agent's features +### GitOps workflow **(PREMIUM)** -By using the Agent, you can: - -- Connect GitLab with a Kubernetes cluster behind a firewall or a -Network Address Translation (NAT). -- Have real-time access to API endpoints in your cluster from GitLab CI/CD. -- Use GitOps to configure your cluster through the [Agent's repository](repository.md). -- Perform pull-based or push-based GitOps deployments. -- Configure [Network Security Alerts](#kubernetes-network-security-alerts) -based on [Container Network Policies](../../application_security/policies/index.md#container-network-policy). -- Track objects applied to your cluster through [inventory objects](../../infrastructure/clusters/deploy/inventory_object.md). -- Use the [CI/CD Tunnel](ci_cd_tunnel.md) to access Kubernetes clusters -from GitLab CI/CD jobs while keeping the cluster's APIs safe and unexposed -to the internet. -- [Deploy the GitLab Runner in a Kubernetes cluster](https://docs.gitlab.com/runner/install/kubernetes-agent.html). - -See the [Agent roadmap](https://gitlab.com/groups/gitlab-org/-/epics/3329) to track its development. - -To contribute to the Agent, see the [Agent's development documentation](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/tree/master/doc). - -## Agent's GitOps workflow **(PREMIUM)** - -The Agent uses multiple GitLab projects to provide a flexible workflow +The agent uses multiple GitLab projects to provide a flexible workflow that can suit various needs. This diagram shows these repositories and the main +The agent uses multiple GitLab projects to provide a flexible workflow. +This diagram shows these repositories and the main actors involved in a deployment: ```mermaid @@ -65,7 +76,7 @@ sequenceDiagram participant D as Developer participant A as Application code repository participant M as Manifest repository - participant K as GitLab Agent + participant K as GitLab agent participant C as Agent configuration repository loop Regularly K-->>C: Grab the configuration @@ -78,61 +89,28 @@ sequenceDiagram end ``` -For more details, refer to our [architecture documentation](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/architecture.md#high-level-architecture) in the Agent project. - -## Install the Agent in your cluster - -See how to [install the Agent in your cluster](install/index.md). - -## GitOps deployments **(PREMIUM)** - -To perform GitOps deployments with the Agent, you need: +For details, view the [architecture documentation](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/architecture.md#high-level-architecture). -- A properly-configured Kubernetes cluster where the Agent is running. -- A [configuration repository](repository.md) that contains a -`config.yaml` file, which tells the Agent the repositories to synchronize -with the cluster. -- A manifest repository that contains manifest files. Any changes to manifest files are applied to the cluster. +To perform GitOps deployments, you need: -You can use a single GitLab project or different projects for the Agent -configuration and manifest files, as follows: +- A properly-configured Kubernetes cluster where the GitLab agent is running. +- A project that contains the agent's configuration file (`config.yaml`) in the repository. + This file tells the agent which repositories to synchronize with the cluster. +- A project that contains Kubernetes manifests. Any changes to manifests are applied to the cluster. -- Single GitLab project (recommended): When you use a single repository to hold - both the manifest and the configuration files, these projects can be either - private or public. -- Two GitLab projects: When you use two different GitLab projects (one for - manifest files and another for configuration files), the manifests project must - be public, while the configuration project can be either private or public. +You can keep the agent's configuration file and Kubernetes manifests in one project, or you can use multiple. -Support for separated private manifest and configuration repositories is tracked in this [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/220912). +- One GitLab project (recommended): When you use one project for both the Kubernetes manifests + and the agent's configuration file, the projects can be either private or public. +- Two GitLab projects: When you use two different GitLab projects (one for Kubernetes + manifests and another for the agent's configuration file), the project with Kubernetes manifests must + be public. The project with the agent's configuration file can be either private or public. -## Kubernetes Network Security Alerts **(ULTIMATE)** - -The GitLab Agent also provides an integration with Cilium. This integration provides a simple way to -generate network policy-related alerts and to surface those alerts in GitLab. - -There are several components that work in concert for the Agent to generate the alerts: - -- A working Kubernetes cluster. -- Cilium integration through either of these options: - - Installation through [cluster management template](../../project/clusters/protect/container_network_security/quick_start_guide.md#use-the-cluster-management-template-to-install-cilium). - - Enablement of [hubble-relay](https://docs.cilium.io/en/v1.8/concepts/overview/#hubble) on an - existing installation. -- One or more network policies through any of these options: - - Use the [Container Network Policy editor](../../application_security/policies/index.md#container-network-policy-editor) to create and manage policies. - - Use an [AutoDevOps](../../application_security/policies/index.md#container-network-policy) configuration. - - Add the required labels and annotations to existing network policies. -- A configuration repository with [Cilium configured in `config.yaml`](repository.md#surface-network-security-alerts-from-cluster-to-gitlab) - -The setup process follows the same [Agent's installation steps](install/index.md), -with the following differences: - -- When you define a configuration repository, you must do so with [Cilium settings](repository.md#surface-network-security-alerts-from-cluster-to-gitlab). -- You do not need to specify the `gitops` configuration section. +Support for separate private projects is tracked in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/283885). ## Remove an agent -You can remove an agent using the [GitLab UI](#remove-an-agent-through-the-gitlab-ui) or through the [GraphQL API](#remove-an-agent-with-the-gitlab-graphql-api). +You can remove an agent by using the [GitLab UI](#remove-an-agent-through-the-gitlab-ui) or the [GraphQL API](#remove-an-agent-with-the-gitlab-graphql-api). ### Remove an agent through the GitLab UI @@ -140,16 +118,16 @@ You can remove an agent using the [GitLab UI](#remove-an-agent-through-the-gitla To remove an agent from the UI: -1. Go to your agent's configuration repository. -1. From your project's sidebar, select **Infrastructure > Kubernetes clusters**. -1. Select your agent from the table, and then in the **Options** column, click the vertical ellipsis -(**{ellipsis_v}**) button and select **Delete agent**. +1. On the top bar, select **Menu > Projects** and find the project that contains the agent's configuration file. +1. From the left sidebar, select **Infrastructure > Kubernetes clusters**. +1. In the table, in the row for your agent, in the **Options** column, select the vertical ellipsis (**{ellipsis_v}**). +1. Select **Delete agent**. ### Remove an agent with the GitLab GraphQL API 1. Get the `<cluster-agent-token-id>` from a query in the interactive GraphQL explorer. -For GitLab.com, go to <https://gitlab.com/-/graphql-explorer> to open GraphQL Explorer. -For self-managed GitLab instances, go to `https://gitlab.example.com/-/graphql-explorer`, replacing `gitlab.example.com` with your own instance's URL. + - For GitLab.com, go to <https://gitlab.com/-/graphql-explorer> to open GraphQL Explorer. + - For self-managed GitLab, go to `https://gitlab.example.com/-/graphql-explorer`, replacing `gitlab.example.com` with your instance's URL. ```graphql query{ @@ -195,185 +173,48 @@ For self-managed GitLab instances, go to `https://gitlab.example.com/-/graphql-e } ``` -1. Delete the Agent in your cluster: +1. Delete the agent in your cluster: ```shell kubectl delete -n gitlab-kubernetes-agent -f ./resources.yml ``` -## Migrating to the GitLab Agent from the legacy certificate-based integration - -Find out how to [migrate to the GitLab Agent for Kubernetes](../../infrastructure/clusters/migrate_to_gitlab_agent.md) from the certificate-based integration depending on the features you use. - -## Troubleshooting - -If you face any issues while using the Agent, read the -service logs with the following command: - -```shell -kubectl logs -f -l=app=gitlab-kubernetes-agent -n gitlab-kubernetes-agent -``` - -GitLab administrators can additionally view the [GitLab Agent Server logs](../../../administration/clusters/kas.md#troubleshooting). - -### Agent logs - -#### Transport: Error while dialing failed to WebSocket dial +## Migrating to the agent from the legacy certificate-based integration -```json -{ - "level": "warn", - "time": "2020-11-04T10:14:39.368Z", - "msg": "GetConfiguration failed", - "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing failed to WebSocket dial: failed to send handshake request: Get \\\"https://gitlab-kas:443/-/kubernetes-agent\\\": dial tcp: lookup gitlab-kas on 10.60.0.10:53: no such host\"" -} -``` - -This error is shown if there are some connectivity issues between the address -specified as `kas-address`, and your Agent pod. To fix it, make sure that you -specified the `kas-address` correctly. - -```json -{ - "level": "error", - "time": "2021-06-25T21:15:45.335Z", - "msg": "Reverse tunnel", - "mod_name": "reverse_tunnel", - "error": "Connect(): rpc error: code = Unavailable desc = connection error: desc= \"transport: Error while dialing failed to WebSocket dial: expected handshake response status code 101 but got 301\"" -} -``` - -This error occurs if the `kas-address` doesn't include a trailing slash. To fix it, make sure that the -`wss` or `ws` URL ends with a trailing slash, such as `wss://GitLab.host.tld:443/-/kubernetes-agent/` -or `ws://GitLab.host.tld:80/-/kubernetes-agent/`. - -#### ValidationError(Deployment.metadata) - -```json -{ - "level": "info", - "time": "2020-10-30T08:56:54.329Z", - "msg": "Synced", - "project_id": "root/kas-manifest001", - "resource_key": "apps/Deployment/kas-test001/nginx-deployment", - "sync_result": "error validating data: [ValidationError(Deployment.metadata): unknown field \"replicas\" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta, ValidationError(Deployment.metadata): unknown field \"selector\" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta, ValidationError(Deployment.metadata): unknown field \"template\" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta]" -} -``` +Find out how to [migrate to the agent for Kubernetes](../../infrastructure/clusters/migrate_to_gitlab_agent.md) from the certificate-based integration. -This error is shown if a manifest file is malformed, and Kubernetes can't -create specified objects. Make sure that your manifest files are valid. You -may try using them to create objects in Kubernetes directly for more troubleshooting. +## Kubernetes network security alerts **(ULTIMATE)** -#### Error while dialing failed to WebSocket dial: failed to send handshake request - -```json -{ - "level": "warn", - "time": "2020-10-30T09:50:51.173Z", - "msg": "GetConfiguration failed", - "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing failed to WebSocket dial: failed to send handshake request: Get \\\"https://GitLabhost.tld:443/-/kubernetes-agent\\\": net/http: HTTP/1.x transport connection broken: malformed HTTP response \\\"\\\\x00\\\\x00\\\\x06\\\\x04\\\\x00\\\\x00\\\\x00\\\\x00\\\\x00\\\\x00\\\\x05\\\\x00\\\\x00@\\\\x00\\\"\"" -} -``` - -This error is shown if you configured `wss` as `kas-address` on the agent side, -but KAS on the server side is not available via `wss`. To fix it, make sure the -same schemes are configured on both sides. - -It's not possible to set the `grpc` scheme due to the issue -[It is not possible to configure KAS to work with `grpc` without directly editing GitLab KAS deployment](https://gitlab.com/gitlab-org/gitlab/-/issues/276888). To use `grpc` while the -issue is in progress, directly edit the deployment with the -`kubectl edit deployment gitlab-kas` command, and change `--listen-websocket=true` to `--listen-websocket=false`. After running that command, you should be able to use -`grpc://gitlab-kas.<YOUR-NAMESPACE>:8150`. - -#### Decompressor is not installed for grpc-encoding - -```json -{ - "level": "warn", - "time": "2020-11-05T05:25:46.916Z", - "msg": "GetConfiguration.Recv failed", - "error": "rpc error: code = Unimplemented desc = grpc: Decompressor is not installed for grpc-encoding \"gzip\"" -} -``` +> [Deprecated](https://gitlab.com/groups/gitlab-org/-/epics/7476) in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) in GitLab 15.0. -This error is shown if the version of the agent is newer that the version of KAS. -To fix it, make sure that both `agentk` and KAS use the same versions. +WARNING: +Cilium integration is in its end-of-life process. It's [deprecated](https://gitlab.com/groups/gitlab-org/-/epics/7476) +for use in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) +in GitLab 15.0. -#### Certificate signed by unknown authority +The agent for Kubernetes also provides an integration with Cilium. This integration provides a simple way to +generate network policy-related alerts and to surface those alerts in GitLab. -```json -{ - "level": "error", - "time": "2021-02-25T07:22:37.158Z", - "msg": "Reverse tunnel", - "mod_name": "reverse_tunnel", - "error": "Connect(): rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing failed to WebSocket dial: failed to send handshake request: Get \\\"https://GitLabhost.tld:443/-/kubernetes-agent/\\\": x509: certificate signed by unknown authority\"" -} -``` +Several components work in concert for the agent to generate the alerts: -This error is shown if your GitLab instance is using a certificate signed by an internal CA that -is unknown to the agent. One approach to fixing it is to present the CA certificate file to the agent -via a Kubernetes `configmap` and mount the file in the agent `/etc/ssl/certs` directory from where it -will be picked up automatically. +- A working Kubernetes cluster. +- Cilium integration through either of these options: + - Installation through [cluster management template](../../project/clusters/protect/container_network_security/quick_start_guide.md#use-the-cluster-management-template-to-install-cilium). + - Enablement of [hubble-relay](https://docs.cilium.io/en/v1.8/concepts/overview/#hubble) on an + existing installation. +- One or more network policies through any of these options: + - Use the [Container Network Policy editor](../../application_security/policies/index.md#container-network-policy-editor) to create and manage policies. + - Use an [AutoDevOps](../../application_security/policies/index.md#container-network-policy) configuration. + - Add the required labels and annotations to existing network policies. +- A configuration repository with [Cilium configured in `config.yaml`](repository.md#surface-network-security-alerts-from-cluster-to-gitlab) -For example, if your internal CA certificate is `myCA.pem`: +The setup process follows the same [agent's installation steps](install/index.md), +with the following differences: -```plaintext -kubectl -n gitlab-kubernetes-agent create configmap ca-pemstore --from-file=myCA.pem -``` +- When you define a configuration repository, you must do so with [Cilium settings](repository.md#surface-network-security-alerts-from-cluster-to-gitlab). +- You do not need to specify the `gitops` configuration section. -Then in `resources.yml`: - -```yaml - spec: - serviceAccountName: gitlab-kubernetes-agent - containers: - - name: agent - image: "registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:<version>" - args: - - --token-file=/config/token - - --kas-address - - wss://kas.host.tld:443 # replace this line with the line below if using Omnibus GitLab or GitLab.com. - # - wss://gitlab.host.tld:443/-/kubernetes-agent/ - # - wss://kas.gitlab.com # for GitLab.com users, use this KAS. - # - grpc://host.docker.internal:8150 # use this attribute when connecting from Docker. - volumeMounts: - - name: token-volume - mountPath: /config - - name: ca-pemstore-volume - mountPath: /etc/ssl/certs/myCA.pem - subPath: myCA.pem - volumes: - - name: token-volume - secret: - secretName: gitlab-kubernetes-agent-token - - name: ca-pemstore-volume - configMap: - name: ca-pemstore - items: - - key: myCA.pem - path: myCA.pem -``` +## Related topics -Alternatively, you can mount the certificate file at a different location and include it using the -`--ca-cert-file` agent parameter: - -```yaml - containers: - - name: agent - image: "registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:<version>" - args: - - --ca-cert-file=/tmp/myCA.pem - - --token-file=/config/token - - --kas-address - - wss://kas.host.tld:443 # replace this line with the line below if using Omnibus GitLab or GitLab.com. - # - wss://gitlab.host.tld:443/-/kubernetes-agent/ - # - wss://kas.gitlab.com # for GitLab.com users, use this KAS. - # - grpc://host.docker.internal:8150 # use this attribute when connecting from Docker. - volumeMounts: - - name: token-volume - mountPath: /config - - name: ca-pemstore-volume - mountPath: /tmp/myCA.pem - subPath: myCA.pem -``` +- [Troubleshooting](troubleshooting.md) +- [Contribute to the GitLab agent's development](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/tree/master/doc) diff --git a/doc/user/clusters/agent/install/index.md b/doc/user/clusters/agent/install/index.md index b2372789284..4d196e57f8f 100644 --- a/doc/user/clusters/agent/install/index.md +++ b/doc/user/clusters/agent/install/index.md @@ -6,112 +6,140 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Install the GitLab Agent **(FREE)** -> [Moved](https://gitlab.com/groups/gitlab-org/-/epics/6290) from GitLab Premium to GitLab Free in 14.5. +> - [Moved](https://gitlab.com/groups/gitlab-org/-/epics/6290) from GitLab Premium to GitLab Free in 14.5. +> - [Introduced](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/merge_requests/594) multi-arch images in GitLab 14.8. The first multi-arch release is `v14.8.1`. It supports AMD64 and ARM64 architectures. -To get started with the Agent, install it in your cluster. +To connect a cluster to GitLab, you need to install the GitLab Agent +onto your cluster. -## Prerequisites **(SELF)** +## Prerequisites -- An existing Kubernetes cluster. +- An existing Kubernetes cluster. If you don't have a cluster yet, you can create a new cluster on cloud providers, such as: + - [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine/docs/quickstart) + - [Amazon Elastic Kubernetes Service (EKS)](https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html) + - [Digital Ocean](https://docs.digitalocean.com/products/kubernetes/quickstart/) - On self-managed GitLab instances, a GitLab administrator needs to set up the [GitLab Agent Server (KAS)](../../../../administration/clusters/kas.md). ## Installation steps -To install the [Agent](../index.md) in your cluster: +To install the GitLab Agent on your cluster: 1. [Define a configuration repository](#define-a-configuration-repository). -1. [Register an agent with GitLab](#register-an-agent-with-gitlab). -1. [Install the agent into the cluster](#install-the-agent-into-the-cluster). +1. [Register the Agent with GitLab](#register-the-agent-with-gitlab). +1. [Install the Agent onto the cluster](#install-the-agent-onto-the-cluster). <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch a GitLab 14.2 [walking-through video](https://www.youtube.com/watch?v=XuBpKtsgGkE) with this process. +When you complete the installation process, you can +[view your Agent's status and activity information](#view-your-agents). +You can also [configure](#configure-the-agent) it to your needs. + ### Define a configuration repository > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/259669) in GitLab 13.7, the Agent manifest configuration can be added to multiple directories (or subdirectories) of its repository. > - Group authorization was [introduced](https://gitlab.com/groups/gitlab-org/-/epics/5784) in GitLab 14.3. -To create an agent, you need a GitLab repository to hold the configuration file. +To create an Agent, you need a GitLab repository to hold its +configuration file. If you already have a repository holding your +cluster's manifest files, you can use it to store your +Agent's configuration file and sync them with no further steps. -After installed, when you update the configuration file, GitLab transmits the -information to the cluster automatically without downtime. +#### Create the Agent's configuration file -In your repository, add the Agent configuration file under: +To create an Agent, go to the repository where you want to store +it and add the Agent's configuration file under: ```plaintext .gitlab/agents/<agent-name>/config.yaml ``` -Make sure that `<agent-name>` conforms to the [Agent's naming format](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/identity_and_auth.md#agent-identity-and-name). - -WARNING: -The agent is only recognized if you use `.yaml` extension for the `config.yaml` file. The extension `.yml` is **not** recognized. - -You **don't have to add any content** to this file when you create it. The fact that the file exists -tells GitLab that this is an agent configuration file and enables the [CI/CD tunnel](../ci_cd_tunnel.md#example-for-a-kubectl-command-using-the-cicd-tunnel). Later on, you can use this -file to [configure the agent](../repository.md) by setting up parameters such as: +You **don't have to add any content** to this file at the moment you +create it. The fact that the file exists tells GitLab that this is +an Agent. You can edit it later to [configure the Agent](#configure-the-agent). -- Groups and projects that can access the agent via the [CI/CD Tunnel](../ci_cd_tunnel.md). -- [Manifest projects to synchronize](../repository.md#synchronize-manifest-projects). -- The address of the `hubble-relay` for the [Network Security policy integrations](../../../project/clusters/protect/index.md). +When creating this file, pay special attention to: -To see all the settings available, read the [Agent configuration repository documentation](../repository.md). +- The [Agent's naming format](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/identity_and_auth.md#agent-identity-and-name). +- The file extension: use the `.yaml` extension (`config.yaml`). The `.yml` extension is **not** recognized. -### Register an agent with GitLab +### Register the Agent with GitLab > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5786) in GitLab 14.1, you can create a new Agent record directly from the GitLab UI. -Next, create a GitLab Rails Agent record to associate it with -the configuration repository project. Creating this record also creates a Secret needed to configure -the Agent in subsequent steps. +Now that you've created your Agent's configuration file, register it +with GitLab. +When you register the Agent, GitLab generates a token that you need for +installing the Agent onto your cluster. -In GitLab: +In GitLab, go to the project where you added your Agent's configuration +file and: 1. Ensure that [GitLab CI/CD is enabled in your project](../../../../ci/enable_or_disable_ci.md#enable-cicd-in-a-project). 1. From your project's sidebar, select **Infrastructure > Kubernetes clusters**. 1. Select **Actions**. -1. From the **Select an agent** dropdown, select the agent you want to connect and select **Register an agent** to access the installation form. -1. The form reveals your registration token. Securely store this secret token as you cannot view it again. -1. Copy the command under **Recommended installation method**. +1. From the **Select an Agent** dropdown list, select the Agent you want to register and select **Register an Agent**. +1. GitLab generates a registration token for this Agent. Securely store this secret token, as you need it to install the Agent onto your cluster and to [update the Agent](#update-the-agent-version) to another version. +1. Copy the command under **Recommended installation method**. You need it to install the Agent onto your cluster through the one-liner installation method. + +### Install the Agent onto the cluster -### Install the agent into the cluster +To connect your cluster to GitLab, install the registered Agent +onto your cluster. To install it, you can use either: -In your computer: +- [The one-liner installation method](#one-liner-installation). +- [The advanced installation method](#advanced-installation). -1. Open your local terminal and connect to your cluster. +You can use the one-liner installation for trying to use the Agent for the first time, to do internal setups with +high trust, and to quickly get started. For long-term production usage, you may want to use the advanced installation +method to benefit from more configuration options. + +#### One-liner installation + +The one-liner installation is the simplest process, but you need +Docker installed locally. If you don't have it, you can either install +it or opt to the [advanced installation method](#advanced-installation). + +To install the Agent on your cluster using the one-liner installation: + +1. In your computer, open the terminal and connect to your cluster. 1. Run the command you copied when registering your cluster in the previous step. -See the following sections to learn about customizing the installation. +Optionally, you can [customize the one-liner installation command](#customize-the-one-liner-installation). -## Simple installation method +##### Customize the one-liner installation -The command provided by GitLab does the following things: +The one-liner command generated by GitLab: - Creates a namespace for the deployment (`gitlab-kubernetes-agent`). -- Sets up a service account with `cluster-admin` rights. Read more on [how you can restrict this service account](#customize-the-permissions-for-the-agentk-service-account). -- Creates a `Secret` resource for the agent registration token. +- Sets up a service account with `cluster-admin` rights (see [how to restrict this service account](#customize-the-permissions-for-the-agentk-service-account)). +- Creates a `Secret` resource for the Agent's registration token. - Creates a `Deployment` resource for the `agentk` pod. -The one-liner installer can be customized at the command line. To find out the various options the above Docker container supports, run: +You can edit these parameters according to your needs to customize the +one-liner installation command at the command line. To find all available +options, run in your terminal: ```shell docker run --pull=always --rm registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/cli:stable generate --help ``` WARNING: -`--agent-version stable` can be used to refer to the latest stable release at the time when the command runs. It's fine for -testing purposes but for production please make sure to specify a matching version explicitly. - -## Advanced installation method +`--agent-version stable` can be used to refer to the latest stable +release at the time when the command runs. It's fine for testing +purposes but for production please make sure to specify a matching +version explicitly. -For more advanced configurations, we recommend to use [the `kpt` based installation method](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/tree/master/build/deployment/gitlab-agent). +#### Advanced installation -Otherwise, follow the manual installation steps described below. +For advanced installation options, use [the `kpt` installation method](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/tree/master/build/deployment/gitlab-agent). -### Customize the permissions for the `agentk` service account +##### Customize the permissions for the `agentk` service account -The GitLab Agent for Kubernetes allows you to fully own your cluster and requires only the permissions you give. Still, for easy getting started, by default the generated manifests provide `cluster-admin` rights to the agent. +The GitLab Agent allows you to fully own your cluster and grant GitLab +the permissions you want. Still, to facilitate the process, by default the +generated manifests provide `cluster-admin` rights to the Agent. -As part of the advanced installation method, you can restrict the agent access rights using Kustomize overlays. [An example is commented out](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/build/deployment/gitlab-agent/cluster/kustomization.yaml) in the `kpt` package you retrieved as part of the installation. +You can restrict the Agent's access rights using Kustomize overlays. [An example is commented out](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/build/deployment/gitlab-agent/cluster/kustomization.yaml) in the `kpt` package you retrieved as part of the installation. To create restricted permissions: @@ -121,50 +149,115 @@ To create restricted permissions: The above setup allows you to regularly update from the upstream package using `kpt pkg update gitlab-agent --strategy resource-merge` and maintain your customizations at the same time. -## Example projects +## Configure the Agent -The following example projects can help you get started with the Agent. +When successfully installed, you can [configure the Agent](../repository.md) +by editing its configuration file. +When you update the configuration file, GitLab transmits the +information to the cluster automatically without downtime. -- [Configuration repository](https://gitlab.com/gitlab-org/configure/examples/kubernetes-agent) -- This basic GitOps example deploys NGINX: [Manifest repository](https://gitlab.com/gitlab-org/configure/examples/gitops-project) +## View your Agents -## View installed Agents +> The version of installed `agentk` shown on the Agent tab [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340882) in GitLab 14.8. +If you have at least the Developer role, you can access the Agent's +configuration repository and view the Agent's list: -Users with at least the [Developer](../../../permissions.md) can access the user interface -for the Agent at **Infrastructure > Kubernetes clusters**, under the -**Agent** tab. This page lists all registered agents for the current project, -and the configuration directory for each agent: +1. On the left sidebar, select **Infrastructure > Kubernetes clusters**. +1. Select **Agent** tab to view clusters connected to GitLab through the Agent. - +On this page, you can view: -Additional management interfaces are planned for the GitLab Agent. -[Provide more feedback in the related epic](https://gitlab.com/groups/gitlab-org/-/epics/4739). +- All the registered Agents for the current project. +- The connection status. +- The version of `agentk` installed on your cluster. +- The path to each Agent's configuration file. -## View Agent activity information +Furthermore, if you select one of the Agents on your list, you can view its +[activity information](#view-the-agents-activity-information). + +### View the Agent's activity information > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/277323) in GitLab 14.6. -Users with at least the [Developer](../../../permissions.md) can view the Agent's activity events. -The activity logs help you to identify problems and get the information you need for troubleshooting. -You can see events from a week before the current date. -To access an agent's activity: +The activity logs help you to identify problems and get the information +you need for troubleshooting. You can see events from a week before the +current date. To access an Agent's activity: + +1. In your Agent's repository, go to the Agents list as described [above](#view-your-agents). +1. Select the Agent you want to see the activity. + +The activity list includes: + +- Agent registration events: when a new token is **created**. +- Connection events: when an Agent is successfully **connected** to a cluster. + +Note that the connection status is logged when you connect an Agent for +the first time or after more than an hour of inactivity. -1. Go to your agent's configuration repository. -1. From the sidebar, select **Infrastructure > Kubernetes clusters**. +To check what else is planned for the Agent's UI and provide feedback, +see the [related epic](https://gitlab.com/groups/gitlab-org/-/epics/4739). + +### View vulnerabilities in cluster images **(ULTIMATE)** + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/6346) in GitLab 14.8 [with a flag](../../../../administration/feature_flags.md) named `cluster_vulnerabilities`. Enabled by default. + +Users with at least the [Developer role](../../../permissions.md) +can view cluster vulnerabilities. You can access them through the [vulnerability report](../../../application_security/vulnerabilities/index.md) +or in your cluster's image through the following process: + +1. Configure [cluster image scanning](../../../application_security/cluster_image_scanning/index.md) + to your build process. +1. Go to your Agent's configuration repository. +1. On the left sidebar, select **Infrastructure > Kubernetes clusters**. 1. Select the **Agent** tab. -1. Select the agent you want to see the activity. +1. Select the Agent you want to see the vulnerabilities for. + + + +## Create multiple Agents + +You can create and install multiple Agents using the same process +documented above. Give each Agent's configuration file a unique name +and you're good to go. You can create multiple Agents, for example: + +- To reach your cluster from different projects. +- To connect multiple clusters to GitLab. + +## Update the Agent version + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340882) in GitLab 14.8, GitLab warns you on the Agent's list page to update the Agent version installed on your cluster. + +To update the Agent's version on your cluster, you need to re-run the [installation command](#install-the-agent-onto-the-cluster) +with a newer `--agent-version`. Make sure to specify the other required parameters: `--kas-address`, `--namespace`, and `--agent-token`. +You can find the available `agentk` versions in [the container registry](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/container_registry/1223205?sort=desc). -You can see the following events on the activity list: +If you don't have access to your Agent's token, you can retrieve it from your cluster: -- Agent registration: - - When a new token is **created**. -- Connection events: - - When an agent is successfully **connected** to a cluster. +1. On your computer, open the terminal and connect to your cluster. +1. To retrieve the namespace, run: -Note that the connection status is logged when you connect an agent for the first time -or after more than an hour of inactivity. + ```shell + kubectl get namespaces + ``` - +1. To retrieve the secret, run: + + ```shell + kubectl -n <namespace> get secrets + ``` + +1. To retrieve the token, run: + + ```shell + kubectl -n <namespace> get secret <secret-name> --template={{.data.token}} | base64 --decode + ``` + +## Example projects + +The following example projects can help you get started with the Agent. + +- [Configuration repository](https://gitlab.com/gitlab-org/configure/examples/kubernetes-agent) +- This basic GitOps example deploys NGINX: [Manifest repository](https://gitlab.com/gitlab-org/configure/examples/gitops-project) ## Upgrades and version compatibility @@ -178,7 +271,8 @@ A feature introduced in a given GitLab minor version might work with other `agen To make sure that it works, use at least the same `agentk` and `kas` minor version. For example, if your GitLab version is 14.2, use at least `agentk` 14.2 and `kas` 14.2. -We recommend upgrading your `kas` installations together with GitLab instances' upgrades, and to upgrade the `agentk` installations after upgrading GitLab. +We recommend upgrading your `kas` installations together with GitLab instances' upgrades, and to +[upgrade the `agentk` installations](#update-the-agent-version) after upgrading GitLab. The available `agentk` and `kas` versions can be found in [the container registry](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/container_registry/). diff --git a/doc/user/clusters/agent/repository.md b/doc/user/clusters/agent/repository.md index 22964fde395..a9d9fd1c13d 100644 --- a/doc/user/clusters/agent/repository.md +++ b/doc/user/clusters/agent/repository.md @@ -40,6 +40,9 @@ with Kubernetes resource definitions in YAML or JSON format. The Agent monitors each project you declare, and when the project changes, GitLab deploys the changes using the Agent. +WARNING: +When using separate GitLab projects for manifest files and configuration repository, the manifests project must be public. + To use multiple YAML files, specify a `paths` attribute in the `gitops.manifest_projects` section. ```yaml @@ -150,26 +153,19 @@ gitops: ## Authorize projects and groups to use an Agent -> - Group authorization [introduced](https://gitlab.com/groups/gitlab-org/-/epics/5784) in GitLab 14.3. -> - Project authorization [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/327850) in GitLab 14.4. - -If you use the same cluster across multiple projects, you can set up the [CI/CD Tunnel](ci_cd_tunnel.md) -to grant access to an Agent from one or more projects or groups. This way, all the authorized -projects can access the same Agent, which facilitates you to save resources and have a scalable setup. +With the [CI/CD Tunnel](ci_cd_tunnel.md), you can authorize [projects](#authorize-projects-to-use-an-agent) +and [groups](#authorize-groups-to-use-an-agent) to use an Agent. -When you authorize a project to use an agent through the [CI/CD Tunnel](ci_cd_tunnel.md), -the selected Kubernetes context is automatically injected into CI/CD jobs, allowing you to -run Kubernetes commands from your authorized projects' scripts. When you authorize a group, -all the projects that belong to that group can access the selected agent. - -An Agent can only authorize projects or groups in the same group hierarchy as the Agent's configuration -project. You can authorize up to 100 projects and 100 groups per Agent. +Then, you can reach your cluster from authorized projects and [run Kubernetes commands from GitLab CI/CD scripts](#run-kubectl-commands-using-the-cicd-tunnel) +in these projects. ### Authorize projects to use an Agent -To grant projects access to the Agent through the [CI/CD Tunnel](ci_cd_tunnel.md): +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/327850) in GitLab 14.4. + +To grant projects access to the Agent through the CI/CD Tunnel: -1. Go to your Agent's configuration project. +1. Go to your Agent's configuration repository. 1. Edit the Agent's configuration file (`config.yaml`). 1. Add the `projects` attribute into `ci_access`. 1. Identify the project through its path: @@ -182,9 +178,11 @@ To grant projects access to the Agent through the [CI/CD Tunnel](ci_cd_tunnel.md ### Authorize groups to use an Agent +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5784) in GitLab 14.3. + To grant access to all projects within a group: -1. Go to your Agent's configuration project. +1. Go to your Agent's configuration repository. 1. Edit the Agent's configuration file (`config.yaml`). 1. Add the `groups` attribute into `ci_access`. 1. Identify the group or subgroup through its path: @@ -195,7 +193,64 @@ To grant access to all projects within a group: - id: path/to/group/subgroup ``` -### Use impersonation to restrict project and group access **(PREMIUM)** +## Run `kubectl` commands using the CI/CD Tunnel + +After you authorize your project or group to use the Agent, you need to +configure the project's `.gitlab-ci.yaml` file to access the Agent. +This makes it possible to deploy applications to your cluster and run +any Kubernetes-specific commands from the authorized project. + +First, configure your Agent: + +1. Go to your Agent's configuration repository. +1. Edit your Agent's `config.yaml` file authorizing the [project](#authorize-projects-to-use-an-agent) or [group](#authorize-groups-to-use-an-agent) you want to run Kubernetes commands from. + +Then, configure the other project: + +1. Go to the project where you want to run Kubernetes commands from. +1. Edit your project's `.gitlab-ci.yml` file. +1. Set your Agent's context in the first command of the script with the format `path/to/agent/repository:agent-name`. +1. Run Kubernetes commands. + +For example: + +```yaml + deploy: + image: + name: bitnami/kubectl:latest + entrypoint: [""] + script: + - kubectl config use-context path/to/agent/repository:agent-name + - kubectl get pods +``` + +When you use the Agent, KubeContexts are named as `path/to/agent/repository:agent-name`. + +To get the list of available contexts: + +1. Open your terminal and connect to your cluster. +1. Run `kubectl config get-contexts`. + +### `kubectl` commands not supported + +The commands `kubectl exec`, `kubectl cp`, and `kubectl attach` are not supported by the CI/CD tunnel. +Anything else that uses the same API endpoints does not work either as they use the deprecated +SPDY protocol. +We [plan to add support for these features](https://gitlab.com/gitlab-org/gitlab/-/issues/346248) +in a future version of GitLab. + +### `kubectl` requires TLS + +`kubectl` would never send credentials over an unencrypted connection. Self-managed users should ensure that their +GitLab instance is configured with TLS for the CI/CD tunnel feature to work. Trying to use it without TLS +would produce errors: + +```shell +$ kubectl get pods +error: You must be logged in to the server (the server has asked for the client to provide credentials) +``` + +## Use impersonation to restrict project and group access **(PREMIUM)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345014) in GitLab 14.5. @@ -211,11 +266,11 @@ You can impersonate: - The CI job that accesses the cluster. - A specific user or system account defined within the cluster. -#### Impersonate the Agent +### Impersonate the Agent The Agent is impersonated by default. You don't need to do anything to impersonate it. -#### Impersonate the CI job that accesses the cluster +### Impersonate the CI job that accesses the cluster To impersonate the CI job that accesses the cluster, add the `ci_job: {}` key-value under the `access_as` key. @@ -261,7 +316,7 @@ ci_access: ci_job: {} ``` -#### Impersonate a static identity +### Impersonate a static identity For the given CI/CD Tunnel connection, you can use a static identity for the impersonation. @@ -278,6 +333,13 @@ See the [official Kubernetes documentation for more details](https://kubernetes. ## Surface network security alerts from cluster to GitLab **(ULTIMATE)** +> [Deprecated](https://gitlab.com/groups/gitlab-org/-/epics/7476) in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) in GitLab 15.0. + +WARNING: +Cilium integration is in its end-of-life process. It's [deprecated](https://gitlab.com/groups/gitlab-org/-/epics/7476) +for use in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) +in GitLab 15.0. + The GitLab Agent provides an [integration with Cilium](index.md#kubernetes-network-security-alerts). To integrate, add a top-level `cilium` section to your `config.yml` file. Currently, the only configuration option is the Hubble relay address: diff --git a/doc/user/clusters/agent/troubleshooting.md b/doc/user/clusters/agent/troubleshooting.md new file mode 100644 index 00000000000..2c9f98b7c45 --- /dev/null +++ b/doc/user/clusters/agent/troubleshooting.md @@ -0,0 +1,193 @@ +--- +stage: Configure +group: Configure +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Troubleshooting the GitLab Agent for Kubernetes + +When you are using the GitLab Agent for Kubernetes, you might experience issues you need to troubleshoot. + +You can start by viewing the service logs: + +```shell +kubectl logs -f -l=app=gitlab-agent -n gitlab-kubernetes-agent +``` + +If you are a GitLab administrator, you can also view the [GitLab Agent Server logs](../../../administration/clusters/kas.md#troubleshooting). + +## Transport: Error while dialing failed to WebSocket dial + +```json +{ + "level": "warn", + "time": "2020-11-04T10:14:39.368Z", + "msg": "GetConfiguration failed", + "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing failed to WebSocket dial: failed to send handshake request: Get \\\"https://gitlab-kas:443/-/kubernetes-agent\\\": dial tcp: lookup gitlab-kas on 10.60.0.10:53: no such host\"" +} +``` + +This error is shown if there are some connectivity issues between the address +specified as `kas-address`, and your Agent pod. To fix it, make sure that you +specified the `kas-address` correctly. + +```json +{ + "level": "error", + "time": "2021-06-25T21:15:45.335Z", + "msg": "Reverse tunnel", + "mod_name": "reverse_tunnel", + "error": "Connect(): rpc error: code = Unavailable desc = connection error: desc= \"transport: Error while dialing failed to WebSocket dial: expected handshake response status code 101 but got 301\"" +} +``` + +This error occurs if the `kas-address` doesn't include a trailing slash. To fix it, make sure that the +`wss` or `ws` URL ends with a trailing slash, such as `wss://GitLab.host.tld:443/-/kubernetes-agent/` +or `ws://GitLab.host.tld:80/-/kubernetes-agent/`. + +## ValidationError(Deployment.metadata) + +```json +{ + "level": "info", + "time": "2020-10-30T08:56:54.329Z", + "msg": "Synced", + "project_id": "root/kas-manifest001", + "resource_key": "apps/Deployment/kas-test001/nginx-deployment", + "sync_result": "error validating data: [ValidationError(Deployment.metadata): unknown field \"replicas\" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta, ValidationError(Deployment.metadata): unknown field \"selector\" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta, ValidationError(Deployment.metadata): unknown field \"template\" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta]" +} +``` + +This error is shown if a manifest file is malformed, and Kubernetes can't +create specified objects. Make sure that your manifest files are valid. You +may try using them to create objects in Kubernetes directly for more troubleshooting. + +## Error while dialing failed to WebSocket dial: failed to send handshake request + +```json +{ + "level": "warn", + "time": "2020-10-30T09:50:51.173Z", + "msg": "GetConfiguration failed", + "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing failed to WebSocket dial: failed to send handshake request: Get \\\"https://GitLabhost.tld:443/-/kubernetes-agent\\\": net/http: HTTP/1.x transport connection broken: malformed HTTP response \\\"\\\\x00\\\\x00\\\\x06\\\\x04\\\\x00\\\\x00\\\\x00\\\\x00\\\\x00\\\\x00\\\\x05\\\\x00\\\\x00@\\\\x00\\\"\"" +} +``` + +This error is shown if you configured `wss` as `kas-address` on the agent side, +but KAS on the server side is not available via `wss`. To fix it, make sure the +same schemes are configured on both sides. + +It's not possible to set the `grpc` scheme due to the issue +[It is not possible to configure KAS to work with `grpc` without directly editing GitLab KAS deployment](https://gitlab.com/gitlab-org/gitlab/-/issues/276888). To use `grpc` while the +issue is in progress, directly edit the deployment with the +`kubectl edit deployment gitlab-kas` command, and change `--listen-websocket=true` to `--listen-websocket=false`. After running that command, you should be able to use +`grpc://gitlab-kas.<YOUR-NAMESPACE>:8150`. + +## Decompressor is not installed for grpc-encoding + +```json +{ + "level": "warn", + "time": "2020-11-05T05:25:46.916Z", + "msg": "GetConfiguration.Recv failed", + "error": "rpc error: code = Unimplemented desc = grpc: Decompressor is not installed for grpc-encoding \"gzip\"" +} +``` + +This error is shown if the version of the agent is newer that the version of KAS. +To fix it, make sure that both `agentk` and KAS use the same versions. + +## Certificate signed by unknown authority + +```json +{ + "level": "error", + "time": "2021-02-25T07:22:37.158Z", + "msg": "Reverse tunnel", + "mod_name": "reverse_tunnel", + "error": "Connect(): rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing failed to WebSocket dial: failed to send handshake request: Get \\\"https://GitLabhost.tld:443/-/kubernetes-agent/\\\": x509: certificate signed by unknown authority\"" +} +``` + +This error is shown if your GitLab instance is using a certificate signed by an internal CA that +is unknown to the agent. One approach to fixing it is to present the CA certificate file to the agent +via a Kubernetes `configmap` and mount the file in the agent `/etc/ssl/certs` directory from where it +will be picked up automatically. + +For example, if your internal CA certificate is `myCA.pem`: + +```plaintext +kubectl -n gitlab-kubernetes-agent create configmap ca-pemstore --from-file=myCA.pem +``` + +Then in `resources.yml`: + +```yaml + spec: + serviceAccountName: gitlab-kubernetes-agent + containers: + - name: agent + image: "registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:<version>" + args: + - --token-file=/config/token + - --kas-address + - wss://kas.host.tld:443 # replace this line with the line below if using Omnibus GitLab or GitLab.com. + # - wss://gitlab.host.tld:443/-/kubernetes-agent/ + # - wss://kas.gitlab.com # for GitLab.com users, use this KAS. + # - grpc://host.docker.internal:8150 # use this attribute when connecting from Docker. + volumeMounts: + - name: token-volume + mountPath: /config + - name: ca-pemstore-volume + mountPath: /etc/ssl/certs/myCA.pem + subPath: myCA.pem + volumes: + - name: token-volume + secret: + secretName: gitlab-kubernetes-agent-token + - name: ca-pemstore-volume + configMap: + name: ca-pemstore + items: + - key: myCA.pem + path: myCA.pem +``` + +Alternatively, you can mount the certificate file at a different location and include it using the +`--ca-cert-file` agent parameter: + +```yaml + containers: + - name: agent + image: "registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:<version>" + args: + - --ca-cert-file=/tmp/myCA.pem + - --token-file=/config/token + - --kas-address + - wss://kas.host.tld:443 # replace this line with the line below if using Omnibus GitLab or GitLab.com. + # - wss://gitlab.host.tld:443/-/kubernetes-agent/ + # - wss://kas.gitlab.com # for GitLab.com users, use this KAS. + # - grpc://host.docker.internal:8150 # use this attribute when connecting from Docker. + volumeMounts: + - name: token-volume + mountPath: /config + - name: ca-pemstore-volume + mountPath: /tmp/myCA.pem + subPath: myCA.pem +``` + +## Project not found + +```json +{ + "level ":"error ", + "time ":"2022-01-05T15:18:11.331Z", + "msg ":"GetObjectsToSynchronize.Recv failed ", + "mod_name ":"gitops ", + "error ":"rpc error: code = NotFound desc = project not found ", +} +``` + +This error is shown if the manifest project is not public. To fix it, +[make sure your manifest project is public](repository.md#synchronize-manifest-projects) or your manifest files +are stored in the Agent's configuration repository. diff --git a/doc/user/clusters/cost_management.md b/doc/user/clusters/cost_management.md index 14850ca85b3..c99eef385ce 100644 --- a/doc/user/clusters/cost_management.md +++ b/doc/user/clusters/cost_management.md @@ -22,7 +22,7 @@ insights within GitLab: ## Configure cluster cost management -To get started with cluster cost management, you need the [Maintainer role](../permissions.md) in a project or group. +To get started with cluster cost management, you need the Maintainer role for a project or group. 1. Clone the [`kubecost-cost-model`](https://gitlab.com/gitlab-examples/kubecost-cost-model/) example repository, which contains minor modifications to the upstream Kubecost diff --git a/doc/user/clusters/img/cluster_agent_security_tab_v14_8.png b/doc/user/clusters/img/cluster_agent_security_tab_v14_8.png Binary files differnew file mode 100644 index 00000000000..91d7e40429e --- /dev/null +++ b/doc/user/clusters/img/cluster_agent_security_tab_v14_8.png diff --git a/doc/user/clusters/img/gitlab_agent_activity_events_v14_6.png b/doc/user/clusters/img/gitlab_agent_activity_events_v14_6.png Binary files differdeleted file mode 100644 index 90b41ee849c..00000000000 --- a/doc/user/clusters/img/gitlab_agent_activity_events_v14_6.png +++ /dev/null diff --git a/doc/user/clusters/img/kubernetes-agent-ui-list_v14_5.png b/doc/user/clusters/img/kubernetes-agent-ui-list_v14_5.png Binary files differdeleted file mode 100644 index 3463eeb5d93..00000000000 --- a/doc/user/clusters/img/kubernetes-agent-ui-list_v14_5.png +++ /dev/null diff --git a/doc/user/clusters/img/kubernetes-agent-ui-list_v14_8.png b/doc/user/clusters/img/kubernetes-agent-ui-list_v14_8.png Binary files differnew file mode 100644 index 00000000000..5b5ba3a9804 --- /dev/null +++ b/doc/user/clusters/img/kubernetes-agent-ui-list_v14_8.png diff --git a/doc/user/clusters/migrating_from_gma_to_project_template.md b/doc/user/clusters/migrating_from_gma_to_project_template.md index 20c4408d57e..b2ba1bef338 100644 --- a/doc/user/clusters/migrating_from_gma_to_project_template.md +++ b/doc/user/clusters/migrating_from_gma_to_project_template.md @@ -23,16 +23,16 @@ See also [video walk-throughs](#video-walk-throughs) with examples. 1. Create a new project based on the [Cluster Management Project template](management_project_template.md#create-a-new-project-based-on-the-cluster-management-template). 1. [Associate your new Cluster Management Project with your cluster](management_project.md#associate-the-cluster-management-project-with-the-cluster). 1. Detect apps deployed through Helm v2 releases by using the pre-configured [`.gitlab-ci.yml`](management_project_template.md#the-gitlab-ciyml-file) file: - - In case you had overwritten the default GitLab Managed Apps namespace, edit `.gitlab-ci.yml`, - and make sure the script is receiving the correct namespace as an argument: + - In case you had overwritten the default GitLab Managed Apps namespace, edit `.gitlab-ci.yml`, + and make sure the script is receiving the correct namespace as an argument: - ```yaml - script: - - gl-fail-if-helm2-releases-exist <your_custom_namespace> - ``` + ```yaml + script: + - gl-fail-if-helm2-releases-exist <your_custom_namespace> + ``` - - If you kept the default name (`gitlab-managed-apps`), then the script is already - set up. + - If you kept the default name (`gitlab-managed-apps`), then the script is already + set up. Either way, [run a pipeline manually](../../ci/pipelines/index.md#run-a-pipeline-manually) and read the logs of the `detect-helm2-releases` job to know if you have any Helm v2 releases and which are they. @@ -53,7 +53,7 @@ See also [video walk-throughs](#video-walk-throughs) with examples. ```shell helm ls -n gitlab-managed-apps - + NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION runner gitlab-managed-apps 1 2021-06-09 19:36:55.739141644 +0000 UTC deployed gitlab-runner-0.28.0 13.11.0 ``` @@ -67,39 +67,42 @@ See also [video walk-throughs](#video-walk-throughs) with examples. 1. Edit the `applications/{app}/values.yaml` associated with your app to match the currently deployed values. For example, for GitLab Runner: - 1. Copy the output of the following command (it might be big): + 1. Copy the output of the following command (it might be big): - ```shell - helm get values runner -n gitlab-managed-apps -a --output yaml - ``` + ```shell + helm get values runner -n gitlab-managed-apps -a --output yaml + ``` - 1. Overwrite `applications/gitlab-runner/values.yaml` with the output of the previous command. + 1. Overwrite `applications/gitlab-runner/values.yaml` with the output of the previous command. This safe step will guarantee that no unexpected default values overwrite your currently deployed values. For instance, your GitLab Runner could have its `gitlabUrl` or `runnerRegistrationToken` overwritten by mistake. 1. Some apps require special attention: - - Ingress: Due to an existing [chart issue](https://github.com/helm/charts/pull/13646), you might see - `spec.clusterIP: Invalid value` when trying to run the [`./gl-helmfile`](management_project_template.md#the-gitlab-ciyml-file) - command. To work around this, after overwriting the release values in `applications/ingress/values.yaml`, - you might need to overwrite all the occurrences of `omitClusterIP: false`, setting it to `omitClusterIP: true`. - Another approach,could be to collect these IPs by running `kubectl get services -n gitlab-managed-apps` - and then overwriting each `ClusterIP` that it complains about with the value you got from that command. - - - Vault: This application introduces a breaking change from the chart we used in Helm v2 to the chart - used in Helm v3. So, the only way to integrate it with this Cluster Management Project is to actually uninstall this app and accept the - chart version proposed in `applications/vault/values.yaml`. - - - Cert-manager: - - For users on Kubernetes version 1.20 or above, the deprecated cert-manager v0.10 is no longer valid and - and the upgrade includes a breaking change. So we suggest that you [backup and uninstall cert-manager v0.10](#backup-and-uninstall-cert-manager-v010) - , and install cert-manager v1.4 instead. To install this version, uncomment the `applications/cert-manager-1-4/helmfile.yaml` - from the [`./helmfile.yaml`](management_project_template.md#the-main-helmfileyml-file). - This triggers a pipeline to install the new version. - - For users on Kubernetes versions lower than 1.20, you can stick to v0.10 by uncommenting - `applications/cert-manager/helmfile.yaml` - in your project's main Helmfile ([`./helmfile.yaml`](management_project_template.md#the-main-helmfileyml-file)). + - Ingress: Due to an existing [chart issue](https://github.com/helm/charts/pull/13646), you might see + `spec.clusterIP: Invalid value` when trying to run the [`./gl-helmfile`](management_project_template.md#the-gitlab-ciyml-file) + command. To work around this, after overwriting the release values in `applications/ingress/values.yaml`, + you might need to overwrite all the occurrences of `omitClusterIP: false`, setting it to `omitClusterIP: true`. + Another approach,could be to collect these IPs by running `kubectl get services -n gitlab-managed-apps` + and then overwriting each `ClusterIP` that it complains about with the value you got from that command. + + - Vault: This application introduces a breaking change from the chart we used in Helm v2 to the chart + used in Helm v3. So, the only way to integrate it with this Cluster Management Project is to actually uninstall this app and accept the + chart version proposed in `applications/vault/values.yaml`. + + - Cert-manager: + - For users on Kubernetes version 1.20 or above, the deprecated cert-manager v0.10 is no longer valid + and the upgrade includes a breaking change. So we suggest that you [backup and uninstall cert-manager v0.10](#backup-and-uninstall-cert-manager-v010), + and install the latest cert-manager instead. To install this version, uncomment `applications/cert-manager/helmfile.yaml` + from [`./helmfile.yaml`](management_project_template.md#the-main-helmfileyml-file). + This triggers a pipeline to install the new version. + - For users on Kubernetes versions lower than 1.20, you can stick to v0.10 by uncommenting + `applications/cert-manager-legacy/helmfile.yaml` + in your project's main Helmfile ([`./helmfile.yaml`](management_project_template.md#the-main-helmfileyml-file)). + + WARNING: + Cert-manager v0.10 breaks when Kubernetes is upgraded to version 1.20 or later. 1. After following all the previous steps, [run a pipeline manually](../../ci/pipelines/index.md#run-a-pipeline-manually) and watch the `apply` job logs to see if any of your applications were successfully detected, installed, and whether they got any @@ -118,7 +121,7 @@ you want to manage with the Cluster Management Project. ## Backup and uninstall cert-manager v0.10 1. Follow the [official docs](https://docs.cert-manager.io/en/release-0.10/tasks/backup-restore-crds.html) on how to - backup your cert-manager v0.10 data. + backup your cert-manager v0.10 data. 1. Uninstall cert-manager by editing the setting all the occurrences of `installed: true` to `installed: false` in the `applications/cert-manager/helmfile.yaml` file. 1. Search for any left-over resources by executing the following command `kubectl get Issuers,ClusterIssuers,Certificates,CertificateRequests,Orders,Challenges,Secrets,ConfigMaps -n gitlab-managed-apps | grep certmanager`. diff --git a/doc/user/compliance/license_compliance/index.md b/doc/user/compliance/license_compliance/index.md index 04d3cc0595e..18de33ea03b 100644 --- a/doc/user/compliance/license_compliance/index.md +++ b/doc/user/compliance/license_compliance/index.md @@ -14,15 +14,9 @@ project's dependencies for their licenses. You can then decide whether to allow each license. For example, if your application uses an external (open source) library whose license is incompatible with yours, then you can deny the use of that license. -You can take advantage of License Compliance by either: +To detect the licenses in use, License Compliance uses the [License Finder](https://github.com/pivotal/LicenseFinder) scan tool that runs as part of the CI/CD pipeline. The License Compliance job is not dependent on any other job in +a pipeline. -- [Including the job](#configuration) - in your existing `.gitlab-ci.yml` file. -- Implicitly using - [Auto License Compliance](../../../topics/autodevops/stages.md#auto-license-compliance), - provided by [Auto DevOps](../../../topics/autodevops/index.md). - -To detect the licenses in use, License Compliance uses the [License Finder](https://github.com/pivotal/LicenseFinder) scan tool that runs as part of the CI/CD pipeline. For the job to activate, License Finder needs to find a compatible package definition in the project directory. For details, see the [Activation on License Finder documentation](https://github.com/pivotal/LicenseFinder#activation). GitLab checks the License Compliance report, compares the licenses between the source and target branches, and shows the information right on the merge @@ -39,6 +33,14 @@ is displayed in the merge request area. That is the case when you add the Consecutive merge requests have something to compare to and the license compliance report is shown properly. +The results are saved as a +[License Compliance report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportslicense_scanning) +that you can later download and analyze. Due to implementation limitations, we +always take the latest License Compliance artifact available. + +WARNING: +License Compliance Scanning does not support run-time installation of compilers and interpreters. +  You can select a license to see more information. @@ -91,27 +93,26 @@ The reported licenses might be incomplete or inaccurate. | Rust | [Cargo](https://crates.io) | | PHP | [Composer](https://getcomposer.org/) | -## Requirements +## Enable License Compliance -WARNING: -License Compliance Scanning does not support run-time installation of compilers and interpreters. +To enable License Compliance in your project's pipeline, either: -To run a License Compliance scanning job, you need GitLab Runner with the -[`docker` executor](https://docs.gitlab.com/runner/executors/docker.html). +- Enable [Auto License Compliance](../../../topics/autodevops/stages.md#auto-license-compliance) + (provided by [Auto DevOps](../../../topics/autodevops/index.md)). +- Include the [`License-Scanning.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/License-Scanning.gitlab-ci.yml) in your `.gitlab-ci.yml` file. -## Configuration +### Include the License Scanning template -For GitLab 12.8 and later, to enable License Compliance, you must -[include](../../../ci/yaml/index.md#includetemplate) the -[`License-Scanning.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/License-Scanning.gitlab-ci.yml) -that's provided as a part of your GitLab installation. -For older versions of GitLab from 11.9 to 12.7, you must -[include](../../../ci/yaml/index.md#includetemplate) the -[`License-Management.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/d2cc841c55d65bc8134bfb3a467e66c36ac32b0a/lib/gitlab/ci/templates/Security/License-Management.gitlab-ci.yml). -For GitLab versions earlier than 11.9, you can copy and use the job as defined -that template. +Prerequisites: -Add the following to your `.gitlab-ci.yml` file: +- [GitLab Runner](../../../ci/runners/index.md) available, with the + [`docker` executor](https://docs.gitlab.com/runner/executors/docker.html). If you're using the + shared runners on GitLab.com, this is enabled by default. +- License Scanning runs in the `test` stage, which is available by default. If you redefine the stages in the + `.gitlab-ci.yml` file, the `test` stage is required. + +To [include](../../../ci/yaml/index.md#includetemplate) the +[`License-Scanning.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/License-Scanning.gitlab-ci.yml), add it to your `.gitlab-ci.yml` file: ```yaml include: @@ -121,26 +122,6 @@ include: The included template creates a `license_scanning` job in your CI/CD pipeline and scans your dependencies to find their licenses. -NOTE: -Before GitLab 12.8, the `license_scanning` job was named `license_management`. GitLab 13.0 removes -the `license_management` job, so you must migrate to the `license_scanning` job and use the new -`License-Scanning.gitlab-ci.yml` template. - -The results are saved as a -[License Compliance report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportslicense_scanning) -that you can later download and analyze. Due to implementation limitations, we -always take the latest License Compliance artifact available. Behind the scenes, the -[GitLab License Compliance Docker image](https://gitlab.com/gitlab-org/security-products/analyzers/license-finder) -is used to detect the languages/frameworks and in turn analyzes the licenses. - -The License Compliance settings can be changed through [CI/CD variables](#available-cicd-variables) by using the -[`variables`](../../../ci/yaml/index.md#variables) parameter in `.gitlab-ci.yml`. - -### When License Compliance runs - -When using the GitLab `License-Scanning.gitlab-ci.yml` template, the License Compliance job doesn't -wait for other stages to complete. - ### Available CI/CD variables License Compliance can be configured using CI/CD variables. @@ -651,7 +632,7 @@ successfully run. For more information, see [Offline environments](../../applica To use License Compliance in an offline environment, you need: -- GitLab Runner with the [`docker` or `kubernetes` executor](#requirements). +- To meet the standard [License Compliance prerequisites](#include-the-license-scanning-template). - Docker Container Registry with locally available copies of License Compliance [analyzer](https://gitlab.com/gitlab-org/security-products/analyzers) images. NOTE: @@ -674,7 +655,7 @@ registry.gitlab.com/gitlab-org/security-products/analyzers/license-finder:latest The process for importing Docker images into a local offline Docker registry depends on **your network security policy**. Please consult your IT staff to find an accepted and approved -process by which external resources can be imported or temporarily accessed. Note that these scanners are [updated periodically](../../application_security/vulnerabilities/index.md#vulnerability-scanner-maintenance) +process by which external resources can be imported or temporarily accessed. Note that these scanners are [updated periodically](../../application_security/index.md#vulnerability-scanner-maintenance) with new definitions, so consider if you are able to make periodic updates yourself. For details on saving and transporting Docker images as a file, see Docker's documentation on @@ -729,7 +710,7 @@ details about them. For the licenses to appear under the license list, the following requirements must be met: -1. The License Compliance CI job must be [configured](#configuration) for your project. +1. The License Compliance CI/CD job must be [enabled](#enable-license-compliance) for your project. 1. Your project must use at least one of the [supported languages and package managers](#supported-languages-and-package-managers). @@ -772,7 +753,7 @@ Developers of the project can view the policies configured in a project. Prerequisites: -- Maintainer or Owner [role](../../permissions.md#project-members-permissions). +- Maintainer or Owner role. `License-Check` is a [merge request approval](../../project/merge_requests/approvals/index.md) rule you can enable to allow an individual or group to approve a merge request that contains a `denied` diff --git a/doc/user/crm/index.md b/doc/user/crm/index.md index f0f9a907a73..305cca33dd5 100644 --- a/doc/user/crm/index.md +++ b/doc/user/crm/index.md @@ -6,7 +6,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Customer relations management (CRM) **(FREE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2256) in GitLab 14.6 [with a flag](../../administration/feature_flags.md) named `customer_relations`. Disabled by default. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2256) in GitLab 14.6 [with a flag](../../administration/feature_flags.md) named `customer_relations`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `customer_relations`. +On GitLab.com, this feature is not available. With customer relations management (CRM) you can create a record of contacts (individuals) and organizations (companies) and relate them to issues. @@ -133,7 +137,7 @@ API. ### Add contacts to an issue -To add contacts to an issue use the `/add_contacts` +To add contacts to an issue use the `/add_contacts [contact:address@example.com]` [quick action](../project/quick_actions.md). You can also add, remove, or replace issue contacts using the @@ -142,9 +146,25 @@ API. ### Remove contacts from an issue -To remove contacts from an issue use the `/remove_contacts` +To remove contacts from an issue use the `/remove_contacts [contact:address@example.com]` [quick action](../project/quick_actions.md). You can also add, remove, or replace issue contacts using the [GraphQL](../../api/graphql/reference/index.md#mutationissuesetcrmcontacts) API. + +## Autocomplete contacts **(FREE SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2256) in GitLab 14.8 [with a flag](../../administration/feature_flags.md) named `contacts_autocomplete`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `contacts_autocomplete`. +On GitLab.com, this feature is not available. +This feature is not ready for production use. + +When you use the `/add_contacts` or `/remove_contacts` quick actions, follow them with `[contact:` and an autocomplete list appears: + +```plaintext +/add_contacts [contact: +/remove_contacts [contact: +``` diff --git a/doc/user/discussions/index.md b/doc/user/discussions/index.md index 7831fdff249..37047e9f8d0 100644 --- a/doc/user/discussions/index.md +++ b/doc/user/discussions/index.md @@ -115,8 +115,7 @@ You can use [Markdown](../markdown.md) and [quick actions](../project/quick_acti You can edit your own comment at any time. -Anyone with the [Maintainer role](../permissions.md) or -higher can also edit a comment made by someone else. +Anyone with at least the Maintainer role can also edit a comment made by someone else. ## Prevent comments by locking an issue @@ -209,7 +208,7 @@ When you reply to a standard comment, you create a thread. Prerequisites: -- You must have at least the [Guest role](../permissions.md#project-members-permissions). +- You must have at least the Guest role. - You must be in an issue, merge request, or epic. Threads in commits and snippets are not supported. To create a thread by replying to a comment: @@ -231,7 +230,7 @@ You can create a thread without replying to a standard comment. Prerequisites: -- You must have at least the [Guest role](../permissions.md#project-members-permissions). +- You must have at least the Guest role. - You must be in an issue, merge request, commit, or snippet. To create a thread: @@ -253,7 +252,7 @@ In a merge request, you can resolve a thread when you want to finish a conversat Prerequisites: -- You must have at least the [Developer role](../permissions.md#project-members-permissions) +- You must have at least the Developer role or be the author of the change being reviewed. - Resolvable threads can be added only to merge requests. It doesn't work for comments in issues, commits, or snippets. diff --git a/doc/user/gitlab_com/index.md b/doc/user/gitlab_com/index.md index f02073b477b..80ee8d23a35 100644 --- a/doc/user/gitlab_com/index.md +++ b/doc/user/gitlab_com/index.md @@ -122,6 +122,8 @@ Below are the settings for [GitLab Pages](https://about.gitlab.com/stages-devops The maximum size of your Pages site is regulated by the artifacts maximum size, which is part of [GitLab CI/CD](#gitlab-cicd). +There are also [rate limits set for GitLab Pages](#gitlabcom-specific-rate-limits). + ## GitLab CI/CD Below are the current settings regarding [GitLab CI/CD](../../ci/index.md). @@ -310,17 +312,19 @@ limiting responses](#rate-limiting-responses). The following table describes the rate limits for GitLab.com, both before and after the limits change in January, 2021: -| Rate limit | Before 2021-01-18 | From 2021-01-18 | From 2021-02-12 | -|:--------------------------------------------------------------------------|:----------------------------|:------------------------------|:------------------------------| -| **Protected paths** (for a given **IP address**) | **10** requests per minute | **10** requests per minute | **10** requests per minute | -| **Raw endpoint** traffic (for a given **project, commit, and file path**) | **300** requests per minute | **300** requests per minute | **300** requests per minute | -| **Unauthenticated** traffic (from a given **IP address**) | No specific limit | **500** requests per minute | **500** requests per minute | -| **Authenticated** API traffic (for a given **user**) | No specific limit | **2,000** requests per minute | **2,000** requests per minute | -| **Authenticated** non-API HTTP traffic (for a given **user**) | No specific limit | **1,000** requests per minute | **1,000** requests per minute | -| **All** traffic (from a given **IP address**) | **600** requests per minute | **2,000** requests per minute | **2,000** requests per minute | -| **Issue creation** | | **300** requests per minute | **300** requests per minute | -| **Note creation** (on issues and merge requests) | | **300** requests per minute | **60** requests per minute | -| **Advanced, project, and group search** API (for a given **IP address**) | | | **10** requests per minute | +| Rate limit | Before 2021-02-12 | From 2021-02-12 | From 2022-02-03 | +|:--------------------------------------------------------------------------|:------------------------------|:------------------------------|:----------------------------------------| +| **Protected paths** (for a given **IP address**) | **10** requests per minute | **10** requests per minute | **10** requests per minute | +| **Raw endpoint** traffic (for a given **project, commit, and file path**) | **300** requests per minute | **300** requests per minute | **300** requests per minute | +| **Unauthenticated** traffic (from a given **IP address**) | **500** requests per minute | **500** requests per minute | **500** requests per minute | +| **Authenticated** API traffic (for a given **user**) | **2,000** requests per minute | **2,000** requests per minute | **2,000** requests per minute | +| **Authenticated** non-API HTTP traffic (for a given **user**) | **1,000** requests per minute | **1,000** requests per minute | **1,000** requests per minute | +| **All** traffic (from a given **IP address**) | **2,000** requests per minute | **2,000** requests per minute | **2,000** requests per minute | +| **Issue creation** | **300** requests per minute | **300** requests per minute | **300** requests per minute | +| **Note creation** (on issues and merge requests) | **300** requests per minute | **60** requests per minute | **60** requests per minute | +| **Advanced, project, and group search** API (for a given **IP address**) | | **10** requests per minute | **10** requests per minute | +| **GitLab Pages** requests (for a given **IP address**) | | | **1000** requests per **50 seconds** | +| **GitLab Pages** requests (for a given **GitLab Pages domain**) | | | **5000** requests per **10 seconds** | More details are available on the rate limits for [protected paths](#protected-paths-throttle) and [raw diff --git a/doc/user/group/contribution_analytics/index.md b/doc/user/group/contribution_analytics/index.md index 76a8eb77e72..3b866c4a1b0 100644 --- a/doc/user/group/contribution_analytics/index.md +++ b/doc/user/group/contribution_analytics/index.md @@ -11,8 +11,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w With Contribution Analytics, you can get an overview of the [contribution events](../../index.md#user-contribution-events) in your group. -- Analyze your team's contributions over a period of time, and offer a bonus for the top - contributors. +- Analyze your team's contributions over a period of time. - Identify opportunities for improvement with group members who may benefit from additional support. diff --git a/doc/user/group/custom_project_templates.md b/doc/user/group/custom_project_templates.md index 2214a18475a..00e6bc671c1 100644 --- a/doc/user/group/custom_project_templates.md +++ b/doc/user/group/custom_project_templates.md @@ -24,7 +24,7 @@ You can also configure [custom templates for the instance](../admin_area/custom_ Prerequisite: -- You must have the [Owner role for the group](../permissions.md#group-members-permissions). +- You must have the Owner role for the group. To set up custom project templates in a group, add the subgroup that contains the project templates to the group settings: diff --git a/doc/user/group/devops_adoption/index.md b/doc/user/group/devops_adoption/index.md index 4151745189d..333bef84dcc 100644 --- a/doc/user/group/devops_adoption/index.md +++ b/doc/user/group/devops_adoption/index.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Group DevOps Adoption **(ULTIMATE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/321083) in GitLab 13.11 as a [Beta feature](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta). +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/321083) in GitLab 13.11 as a [Beta feature](../../../policy/alpha-beta-support.md#beta-features). > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/333556) in GitLab 14.1. > - The Overview tab [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/330401) in GitLab 14.1. > - DAST and SAST metrics [added](https://gitlab.com/gitlab-org/gitlab/-/issues/328033) in GitLab 14.1. @@ -32,7 +32,7 @@ how to use those features. Prerequisite: -- You must have at least the [Reporter role](../../permissions.md) for the group. +- You must have at least the Reporter role for the group. To view DevOps Adoption: diff --git a/doc/user/group/epics/epic_boards.md b/doc/user/group/epics/epic_boards.md index d184030718a..f03a56d8a00 100644 --- a/doc/user/group/epics/epic_boards.md +++ b/doc/user/group/epics/epic_boards.md @@ -24,7 +24,7 @@ To view an epic board: Prerequisites: -- You must have at least the [Reporter role](../../permissions.md#group-members-permissions) for a group. +- You must have at least the Reporter role for a group. To create a new epic board: @@ -49,7 +49,7 @@ To change these options later, [edit the board](#edit-the-scope-of-an-epic-board Prerequisites: -- You must have at least the [Reporter role](../../permissions.md#group-members-permissions) for a group. +- You must have at least the Reporter role for a group. - A minimum of two boards present in a group. To delete the active epic board: @@ -73,7 +73,7 @@ To delete the active epic board: Prerequisites: -- You must have at least the [Reporter role](../../permissions.md#group-members-permissions) for a group. +- You must have at least the Reporter role for a group. To create a new list: @@ -91,7 +91,7 @@ list view that's removed. You can always create it again later if you need. Prerequisites: -- You must have at least the [Reporter role](../../permissions.md#group-members-permissions) for a group. +- You must have at least the Reporter role for a group. To remove a list from an epic board: @@ -106,7 +106,7 @@ To remove a list from an epic board: Prerequisites: -- You must have at least the [Reporter role](../../permissions.md#group-members-permissions) for a group. +- You must have at least the Reporter role for a group. - You must have [created a list](#create-a-new-list) first. To create an epic from a list in epic board: @@ -147,7 +147,7 @@ You can move epics and lists by dragging them. Prerequisites: -- You must have at least the [Reporter role](../../permissions.md#group-members-permissions) for a group. +- You must have at least the Reporter role for a group. To move an epic, select the epic card and drag it to another position in its current list or into another list. Learn about possible effects in [Dragging epics between lists](#dragging-epics-between-lists). @@ -170,7 +170,7 @@ and the target list. Prerequisites: -- You must have at least the [Reporter role](../../permissions.md#group-members-permissions) for a group. +- You must have at least the Reporter role for a group. To edit the scope of an epic board: diff --git a/doc/user/group/epics/manage_epics.md b/doc/user/group/epics/manage_epics.md index caca10a05a2..3350b0f1169 100644 --- a/doc/user/group/epics/manage_epics.md +++ b/doc/user/group/epics/manage_epics.md @@ -83,7 +83,7 @@ To edit an epic's start date, due date, or labels: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7250) in GitLab 12.2. -Users with at least the [Reporter role](../../permissions.md) can manage epics. +Users with at least the Reporter role can manage epics. When bulk editing epics in a group, you can edit their labels. @@ -98,8 +98,7 @@ To update multiple epics at the same time: ## Delete an epic NOTE: -To delete an epic, you must be an [Owner](../../permissions.md#group-members-permissions) of a group -or subgroup. +To delete an epic, you must be an Owner of a group or subgroup. To delete the epic: @@ -165,6 +164,7 @@ than 1000. The cached value is rounded to thousands or millions and updated ever > - Sorting by epic titles [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/331625) in GitLab 14.1. > - Searching by milestone and confidentiality [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/268372) in GitLab 14.2 [with a flag](../../../administration/feature_flags.md) named `vue_epics_list`. Disabled by default. > - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/276189) in GitLab 14.7. +> - [Feature flag `vue_epics_list`](https://gitlab.com/gitlab-org/gitlab/-/issues/327320) removed in GitLab 14.8. You can search for an epic from the list of epics using filtered search bar based on following parameters: diff --git a/doc/user/group/import/index.md b/doc/user/group/import/index.md index 70406cfe8e8..cdc3fe02a53 100644 --- a/doc/user/group/import/index.md +++ b/doc/user/group/import/index.md @@ -10,20 +10,81 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/249160) in GitLab 13.7 [with a flag](../../feature_flags.md) named `bulk_import`. Disabled by default. > - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/338985) in GitLab 14.3. +FLAG: +On self-managed GitLab, by default this feature is available. To hide the feature, ask an administrator to [disable the feature flag](../../../administration/feature_flags.md) named `bulk_import`. On GitLab.com, this feature is available. + +You can migrate your existing top-level groups to any of the following: + +- Another GitLab instance, including GitLab.com. +- Another top-level group. +- The subgroup of any existing top-level group. + +Migrating groups is not the same as [group import/export](../settings/import_export.md). + +- Group import/export requires you to export a group to a file and then import that file in + another GitLab instance. +- Group migration automates this process. + +## Import your groups into GitLab + +When you migrate a group, you connect to your GitLab instance and then choose +groups to import. Not all the data is migrated. View the +[Migrated resources](#migrated-resources) list for details. + +Leave feedback about group migration in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/284495). + NOTE: -The importer migrates **only** the group data listed on this page. To leave feedback on this -feature, see [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/284495). +You might need to reconfigure your firewall to prevent blocking the connection on the self-managed +instance. + +### Connect to the remote GitLab instance -Using GitLab Group Migration, you can migrate existing top-level groups from GitLab.com or a self-managed instance. Groups can be migrated to a target instance, as a top-level group, or as a subgroup of any existing top-level group. +Before you begin, ensure that the target GitLab instance can communicate with the source over HTTPS +(HTTP is not supported). You might need to reconfigure your firewall to prevent blocking the connection on the self-managed +instance. -The following resources are migrated to the target instance: +Then create the group you want to import into, and connect: + +1. Create a new group or subgroup: + + - On the top bar, select `+` and then **New group**. + - Or, on an existing group's page, in the top right, select **New subgroup**. + +1. Select **Import group**. +1. Enter the source URL of your GitLab instance. +1. Generate or copy a [personal access token](../../../user/profile/personal_access_tokens.md) + with the `api` and `read_repository` scopes on your remote GitLab instance. +1. Enter the [personal access token](../../../user/profile/personal_access_tokens.md) for your remote GitLab instance. +1. Select **Connect instance**. + +### Select the groups to import + +After you have authorized access to the GitLab instance, you are redirected to the GitLab Group +Migration importer page. The remote groups you have the Owner role for are listed. + +1. By default, the proposed group namespaces match the names as they exist in remote instance, but based on your permissions, you can choose to edit these names before you proceed to import any of them. +1. Next to the groups you want to import, select **Import**. +1. The **Status** column shows the import status of each group. If you leave the page open, it updates in real-time. +1. After a group has been imported, select its GitLab path to open its GitLab URL. + + + +## Automate group and project import **(PREMIUM)** + +For information on automating user, group, and project import API calls, see +[Automate group and project import](../../project/import/index.md#automate-group-and-project-import). + +## Migrated resources + +Only the following resources are migrated to the target instance. Any other items are **not** +migrated: - Groups ([Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4374) in 13.7) - description - attributes - subgroups - avatar ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322904) in 14.0) -- Group Labels ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292429) in 13.9) +- Group labels ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292429) in 13.9) - title - description - color @@ -71,67 +132,3 @@ The following resources are migrated to the target instance: - image URL - Boards - Board Lists - -Any other items are **not** migrated. - -## Enable or disable GitLab Group Migration - -GitLab Migration is deployed behind the `bulk_import` feature flag, which is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) -can disable it. - -To disable it: - -```ruby -Feature.disable(:bulk_import) -``` - -To enable it: - -```ruby -Feature.enable(:bulk_import) -``` - -## Import your groups into GitLab - -Before you begin, ensure that the target instance of GitLab can communicate with the source -over HTTPS (HTTP is not supported). - -NOTE: -This might involve reconfiguring your firewall to prevent blocking connection on the side of self-managed instance. - -### Connect to the remote GitLab instance - -1. Go to the New Group page: - - - On the top bar, select `+` and then **New group**. - - Or, on an existing group's page, in the top right, select **New subgroup**. - -  - -1. On the New Group page, select **Import group**. - -  - -1. Enter the source URL of your GitLab instance. -1. Generate or copy a [personal access token](../../../user/profile/personal_access_tokens.md) - with the `api` and `read_repository` scopes on your remote GitLab instance. -1. Enter the [personal access token](../../../user/profile/personal_access_tokens.md) for your remote GitLab instance. -1. Select **Connect instance**. - -### Selecting which groups to import - -After you have authorized access to the GitLab instance, you are redirected to the GitLab Group -Migration importer page. The remote groups you have the Owner role for are listed. - -1. By default, the proposed group namespaces match the names as they exist in remote instance, but based on your permissions, you can choose to edit these names before you proceed to import any of them. -1. Next to the groups you want to import, select **Import**. -1. The **Status** column shows the import status of each group. If you leave the page open, it updates in real-time. -1. After a group has been imported, select its GitLab path to open its GitLab URL. - - - -## Automate group and project import **(PREMIUM)** - -For information on automating user, group, and project import API calls, see -[Automate group and project import](../../project/import/index.md#automate-group-and-project-import). diff --git a/doc/user/group/index.md b/doc/user/group/index.md index 8aa9b8e799d..ec76dc52516 100644 --- a/doc/user/group/index.md +++ b/doc/user/group/index.md @@ -1,7 +1,7 @@ --- type: reference, howto stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -68,19 +68,9 @@ To create a group: - Select **Menu > Groups**, and on the right, select **Create group**. - To the left of the search box, select the plus sign and then **New group**. 1. Select **Create group**. -1. For the **Group name**, use only: - - Alphanumeric characters - - Emojis - - Underscores - - Dashes, dots, spaces, and parentheses (however, it cannot start with any of these characters) - - For a list of words that cannot be used as group names, see [reserved names](../reserved_names.md). - -1. For the **Group URL**, which is used for the [namespace](#namespaces), - use only: - - Alphanumeric characters - - Underscores - - Dashes and dots (it cannot start with dashes or end in a dot) +1. Enter a name for the group in **Group name**. For a list of words that cannot be used as group names, see + [reserved names](../reserved_names.md). +1. Enter a path for the group in **Group URL**, which is used for the [namespace](#namespaces). 1. Choose the [visibility level](../../public_access/public_access.md). 1. Personalize your GitLab experience by answering the following questions: - What is your role? @@ -138,7 +128,7 @@ your group. ## Change the owner of a group You can change the owner of a group. Each group must always have at least one -member with the [Owner role](../permissions.md#group-members-permissions). +member with the Owner role. - As an administrator: 1. Go to the group and from the left menu, select **Group information > Members**. @@ -153,7 +143,7 @@ member with the [Owner role](../permissions.md#group-members-permissions). Prerequisites: -- You must have the [Owner role](../permissions.md#group-members-permissions). +- You must have the Owner role. - The member must have direct membership in the group. If membership is inherited from a parent group, then the member can be removed from the parent group only. @@ -250,7 +240,7 @@ There are two different ways to add a new project to a group: > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2534) in GitLab 10.5. > - [Moved](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/25975) from GitLab Premium to GitLab Free in 11.10. -By default, [Developers and Maintainers](../permissions.md#group-members-permissions) can create projects under a group. +By default, users with at least the Developer role can create projects under a group. To change this setting for a specific group: @@ -290,10 +280,15 @@ To view the activity feed in Atom format, select the ## Share a group with another group -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18328) in GitLab 12.7. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18328) in GitLab 12.7. +> - [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/247208) in GitLab 13.11 from a form to a modal window [with a flag](../feature_flags.md). Disabled by default. +> - Modal window [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/247208) in GitLab 14.8. -NOTE: -In GitLab 13.11, you can [replace this form with a modal window](#share-a-group-modal-window). +FLAG: +On self-managed GitLab, by default the modal window feature is available. +To hide the feature, ask an administrator to [disable the feature flag](../../administration/feature_flags.md) +named `invite_members_group_modal`. +On GitLab.com, this feature is available. Similar to how you [share a project with a group](../project/members/share_project_with_groups.md), you can share a group with another group. Members get direct access @@ -303,35 +298,14 @@ To share a given group, for example, `Frontend` with another group, for example, `Engineering`: 1. Go to the `Frontend` group. -1. From the left menu, select **Group information > Members**. -1. Select the **Invite group** tab. +1. On the left sidebar, select **Group information > Members**. +1. Select **Invite a group**. 1. In the **Select a group to invite** list, select `Engineering`. -1. For the **Max role**, select a [role](../permissions.md). +1. Select a [role](../permissions.md). 1. Select **Invite**. All the members of the `Engineering` group are added to the `Frontend` group. -### Share a group modal window - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/247208) in GitLab 13.11. -> - [Deployed behind a feature flag](../feature_flags.md), disabled by default. -> - Enabled on GitLab.com. -> - Recommended for production use. -> - Replaces the existing form with buttons to open a modal window. -> - To use in GitLab self-managed instances, ask a GitLab administrator to [enable it](../project/members/index.md#enable-or-disable-modal-window). - -WARNING: -This feature might not be available to you. Check the **version history** note above for details. - -In GitLab 13.11, you can optionally replace the sharing form with a modal window. -To share a group after enabling this feature: - -1. Go to your group's page. -1. On the left sidebar, go to **Group information > Members**, and then select **Invite a group**. -1. Select a group, and select a **Max role**. -1. Optional. Select an **Access expiration date**. -1. Select **Invite**. - ## Manage group memberships via LDAP **(PREMIUM SELF)** Group syncing allows LDAP groups to be mapped to GitLab groups. This provides more control over per-group user management. To configure group syncing, edit the `group_base` **DN** (`'OU=Global Groups,OU=GitLab INT,DC=GitLab,DC=org'`). This **OU** contains all groups that will be associated with GitLab groups. @@ -432,10 +406,22 @@ for the group's projects to meet your group's needs. To remove a group and its contents: -1. Go to your group's **Settings > General** page. -1. Expand the **Path, transfer, remove** section. +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Settings > General**. +1. Expand the **Advanced** section. +1. In the **Remove group** section, select **Remove group**. +1. Type the group name. +1. Select **Confirm**. + +A group can also be removed from the groups dashboard: + +1. On the top bar, select **Menu > Groups**. +1. Select **Your Groups**. +1. Select (**{ellipsis_v}**) for the group you want to delete. +1. Select **Delete**. 1. In the Remove group section, select **Remove group**. -1. Confirm the action. +1. Type the group name. +1. Select **Confirm**. This action removes the group. It also adds a background job to delete all projects in the group. @@ -534,7 +520,7 @@ disabled for the group and its subgroups. Prerequisite: -- You must be assigned the [Owner role](../permissions.md#group-members-permissions) for the group. +- You must be assigned the Owner role) for the group. To specify a user cap: @@ -556,7 +542,7 @@ You can remove the user cap, so there is no limit on the number of members you c Prerequisite: -- You must be assigned the [Owner role](../permissions.md#group-members-permissions) for the group. +- You must be assigned the Owner role) for the group. To remove the user cap: @@ -575,7 +561,7 @@ and must be approved. Prerequisite: -- You must be assigned the [Owner role](../permissions.md#group-members-permissions) for the group. +- You must be assigned the Owner role) for the group. To approve members that are pending because they've exceeded the user cap: @@ -651,7 +637,7 @@ To restrict group access by IP address: 1. Go to the group's **Settings > General** page. 1. Expand the **Permissions and group features** section. -1. In the **Allow access to the following IP addresses** field, enter IP address ranges in CIDR notation. +1. In the **Allow access to the following IP addresses** field, enter IPv4 or IPv6 address ranges in CIDR notation. 1. Select **Save changes**.  @@ -803,7 +789,7 @@ Existing forks are not removed. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/224129) in GitLab 13.4. Group push rules allow group maintainers to set -[push rules](../../push_rules/push_rules.md) for newly created projects in the specific group. +[push rules](../project/repository/push_rules.md) for newly created projects in the specific group. To configure push rules for a group: diff --git a/doc/user/group/iterations/index.md b/doc/user/group/iterations/index.md index 8a79effba15..c0bd6f1a672 100644 --- a/doc/user/group/iterations/index.md +++ b/doc/user/group/iterations/index.md @@ -52,7 +52,7 @@ With iteration cadences enabled, you must first Prerequisites: -- You must have at least the [Developer role](../../permissions.md) for a group. +- You must have at least the Developer role for a group. To create an iteration cadence: @@ -65,7 +65,7 @@ To create an iteration cadence: Prerequisites: -- You must have at least the [Developer role](../../permissions.md) for a group. +- You must have at least the Developer role for a group. Deleting an iteration cadence also deletes all iterations within that cadence. @@ -86,7 +86,7 @@ From there you can create a new iteration or select an iteration to get a more d Prerequisites: -- You must have at least the [Developer role](../../permissions.md) for a group. +- You must have at least the Developer role for a group. For manually scheduled iteration cadences, you create and add iterations yourself. @@ -104,7 +104,7 @@ To create an iteration: Prerequisites: -- You must have at least the [Developer role](../../permissions.md) for a group. +- You must have at least the Developer role for a group. To edit an iteration, select the three-dot menu (**{ellipsis_v}**) > **Edit**. @@ -114,7 +114,7 @@ To edit an iteration, select the three-dot menu (**{ellipsis_v}**) > **Edit**. Prerequisites: -- You must have at least the [Developer role](../../permissions.md) for a group. +- You must have at least the Developer role for a group. To delete an iteration, select the three-dot menu (**{ellipsis_v}**) > **Delete**. diff --git a/doc/user/group/planning_hierarchy/img/view-project-work-item-hierarchy_v14_8.png b/doc/user/group/planning_hierarchy/img/view-project-work-item-hierarchy_v14_8.png Binary files differnew file mode 100644 index 00000000000..2ddd551ee46 --- /dev/null +++ b/doc/user/group/planning_hierarchy/img/view-project-work-item-hierarchy_v14_8.png diff --git a/doc/user/group/planning_hierarchy/index.md b/doc/user/group/planning_hierarchy/index.md index 5887328abe4..934421e8a9a 100644 --- a/doc/user/group/planning_hierarchy/index.md +++ b/doc/user/group/planning_hierarchy/index.md @@ -5,7 +5,7 @@ group: Product Planning info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Planning hierarchies **(PREMIUM)** +# Planning hierarchies **(FREE)** Planning hierarchies are an integral part of breaking down your work in GitLab. To understand how you can use epics and issues together in hierarchies, remember the following: @@ -20,7 +20,21 @@ To learn about hierarchies in general, common frameworks, and using GitLab for portfolio management, see [How to use GitLab for Agile portfolio planning and project management](https://about.gitlab.com/blog/2020/11/11/gitlab-for-agile-portfolio-planning-project-management/). -## Hierarchies with epics +## View planning hierarchies + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340844/) in GitLab 14.8. + +To view the planning hierarchy in a project: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Project information > Planning hierarchy**. + +Under **Current structure**, you can see a hierarchy diagram that matches your current planning hierarchy. +The work items outside your subscription plan show up below **Unavailable structure**. + + + +## Hierarchies with epics **(PREMIUM)** With epics, you can achieve the following hierarchy: @@ -54,14 +68,14 @@ Epic "1"*-- "0..*" Issue  -## View ancestry of an epic - -In an epic, you can view the ancestors as parents in the right sidebar under **Ancestors**. - - - ## View ancestry of an issue In an issue, you can view the parented epic above the issue in the right sidebar under **Epic**.  + +## View ancestry of an epic **(PREMIUM)** + +In an epic, you can view the ancestors as parents in the right sidebar under **Ancestors**. + + diff --git a/doc/user/group/repositories_analytics/index.md b/doc/user/group/repositories_analytics/index.md index 2487ab188e8..6e3ea7d6c0f 100644 --- a/doc/user/group/repositories_analytics/index.md +++ b/doc/user/group/repositories_analytics/index.md @@ -1,7 +1,6 @@ --- -type: reference stage: Verify -group: Testing +group: Pipeline Insights info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/group/roadmap/index.md b/doc/user/group/roadmap/index.md index 7d489bc5b2d..f10dc3bc22a 100644 --- a/doc/user/group/roadmap/index.md +++ b/doc/user/group/roadmap/index.md @@ -72,6 +72,26 @@ You can also filter epics in the Roadmap view by the epics': Roadmaps can also be [visualized inside an epic](../epics/index.md#roadmap-in-epics). +### Roadmap settings + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345158) in GitLab 14.8 [with a flag](../../../administration/feature_flags.md) named `roadmap_settings`. Enabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `roadmap_settings`. +On GitLab.com, this feature is available but can be configured by GitLab.com administrators only. + +When you enable the roadmap settings sidebar, you can use it to refine epics shown in the roadmap. + +You can configure the following: + +- Select date range. +- Turn milestones on or off and select whether to show all, group, sub-group, or project milestones. +- Show all, open, or closed epics. +- Turn progress tracking for child issues on or off and select whether + to use issue weights or counts. + + The progress tracking setting is not saved in user preferences but is saved or shared using URL parameters. + ## Timeline duration > - Introduced in GitLab 11.0. diff --git a/doc/user/group/saml_sso/group_managed_accounts.md b/doc/user/group/saml_sso/group_managed_accounts.md index 06e666f4d24..aeb7db923a9 100644 --- a/doc/user/group/saml_sso/group_managed_accounts.md +++ b/doc/user/group/saml_sso/group_managed_accounts.md @@ -1,7 +1,7 @@ --- type: reference, howto stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/group/saml_sso/index.md b/doc/user/group/saml_sso/index.md index 20ff4a201f5..14c4447c5c6 100644 --- a/doc/user/group/saml_sso/index.md +++ b/doc/user/group/saml_sso/index.md @@ -1,7 +1,7 @@ --- type: reference, howto stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -21,7 +21,7 @@ SAML SSO is only configurable at the top-level group. If required, you can find [a glossary of common terms](../../../integration/saml.md#glossary-of-common-terms). -## Configuring your identity provider +## Configure your identity provider 1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Settings > SAML SSO**. @@ -32,7 +32,7 @@ If required, you can find [a glossary of common terms](../../../integration/saml 1. Configure the required [user attributes](#user-attributes), ensuring you include the user's email address. 1. While the default is enabled for most SAML providers, please ensure the app is set to have service provider initiated calls in order to link existing GitLab accounts. -1. Once the identity provider is set up, move on to [configuring GitLab](#configuring-gitlab). +1. Once the identity provider is set up, move on to [configuring GitLab](#configure-gitlab).  @@ -82,7 +82,7 @@ GitLab provides metadata XML that can be used to configure your identity provide 1. Copy the provided **GitLab metadata URL**. 1. Follow your identity provider's documentation and paste the metadata URL when it's requested. -## Configuring GitLab +## Configure GitLab After you set up your identity provider to work with GitLab, you must configure GitLab to use it for authentication: @@ -108,7 +108,9 @@ The certificate [fingerprint algorithm](../../../integration/saml.md#notes-on-co > - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/9152) in GitLab 13.11 with enforcing open SSO session to use Git if this setting is switched on. > - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/339888) in GitLab 14.7 to not enforce SSO checks for Git activity originating from CI/CD jobs. -With this option enabled, users (except users with the Owner role) must access GitLab using your group GitLab single sign-on URL to access group resources. Users added manually as members can't access group resources. +With this option enabled, users must access GitLab using your group GitLab single sign-on URL to access group resources. +Users can't be added as new members manually. +Users with the Owner role can use the standard sign in process to make necessary changes to top-level group settings. SSO enforcement does not affect sign in or access to any resources outside of the group. Users can view which groups and projects they are a member of without SSO sign in. @@ -116,7 +118,7 @@ However, users are not prompted to sign in through SSO on each visit. GitLab che has authenticated through SSO. If it's been more than 1 day since the last sign-in, GitLab prompts the user to sign in again through SSO. -We intend to add a similar SSO requirement for [API activity](https://gitlab.com/gitlab-org/gitlab/-/issues/297389). +An [issue exists](https://gitlab.com/gitlab-org/gitlab/-/issues/297389) to add a similar SSO requirement for API activity. SSO has the following effects when enabled: @@ -127,7 +129,6 @@ SSO has the following effects when enabled: - Git activity originating from CI/CD jobs do not have the SSO check enforced. - Credentials that are not tied to regular users (for example, access tokens and deploy keys) do not have the SSO check enforced. - Users must be signed-in through SSO before they can pull images using the [Dependency Proxy](../../packages/dependency_proxy/index.md). -<!-- Add bullet for API activity when https://gitlab.com/gitlab-org/gitlab/-/issues/9152 is complete --> When SSO is enforced, users are not immediately revoked. If the user: @@ -139,7 +140,7 @@ When SSO is enforced, users are not immediately revoked. If the user: The SAML standard means that you can use a wide range of identity providers with GitLab. Your identity provider might have relevant documentation. It can be generic SAML documentation or specifically targeted for GitLab. -When [configuring your identity provider](#configuring-your-identity-provider), please consider the notes below for specific providers to help avoid common issues and as a guide for terminology used. +When [configuring your identity provider](#configure-your-identity-provider), please consider the notes below for specific providers to help avoid common issues and as a guide for terminology used. For providers not listed below, you can refer to the [instance SAML notes on configuring an identity provider](../../../integration/saml.md#notes-on-configuring-your-identity-provider) for additional guidance on information your identity provider may require. @@ -293,12 +294,16 @@ convert the information to XML. An example SAML response is shown here. ### Role -Starting from [GitLab 13.3](https://gitlab.com/gitlab-org/gitlab/-/issues/214523), group owners can set a 'Default membership role' other than 'Guest'. To do so, [configure the SAML SSO for the group](#configuring-gitlab). That role becomes the starting access level of all users added to the group. +Starting from [GitLab 13.3](https://gitlab.com/gitlab-org/gitlab/-/issues/214523), group owners can set a +"Default membership role" other than Guest. To do so, [configure the SAML SSO for the group](#configure-gitlab). +That role becomes the starting access level of all users added to the group. Existing members with appropriate privileges can promote or demote users, as needed. If a user is already a member of the group, linking the SAML identity does not change their role. +Users given a "minimal access" role have [specific restrictions](../../permissions.md#users-with-minimal-access). + ### Blocking access To rescind a user's access to the group when only SAML SSO is configured, either: @@ -336,6 +341,13 @@ For example, to unlink the `MyOrg` account: ## Group Sync +WARNING: +Changing Group Sync configuration can remove users from the relevant GitLab group. +Removal happens if there is any mismatch between the group names and the list of `groups` in the SAML response. +If changes must be made, ensure either the SAML response includes the `groups` attribute +and the `AttributeValue` value matches the **SAML Group Name** in GitLab, +or that all groups are removed from GitLab to disable Group Sync. + <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For a demo of Group Sync using Azure, see [Demo: SAML Group Sync](https://youtu.be/Iqvo2tJfXjg). @@ -353,10 +365,6 @@ Ensure your SAML identity provider sends an attribute statement named `Groups` o </saml:AttributeStatement> ``` -WARNING: -Setting up Group Sync can disconnect users from SAML IDP if there is any mismatch in the configuration. Ensure the -`Groups` attribute is included in the SAML response, and the **SAML Group Name** matches the `AttributeValue` attribute. - Other attribute names such as `http://schemas.microsoft.com/ws/2008/06/identity/claims/groups` are not accepted as a source of groups. See the [SAML troubleshooting page](../../../administration/troubleshooting/group_saml_scim.md) @@ -533,7 +541,7 @@ This can then be compared to the [NameID](#nameid) being sent by the identity pr If you receive a `404` during setup when using "verify configuration", make sure you have used the correct [SHA-1 generated fingerprint](../../../integration/saml.md#notes-on-configuring-your-identity-provider). -If a user is trying to sign in for the first time and the GitLab single sign-on URL has not [been configured](#configuring-your-identity-provider), they may see a 404. +If a user is trying to sign in for the first time and the GitLab single sign-on URL has not [been configured](#configure-your-identity-provider), they may see a 404. As outlined in the [user access section](#linking-saml-to-your-existing-gitlabcom-account), a group Owner needs to provide the URL to users. ### Message: "SAML authentication failed: Extern UID has already been taken" diff --git a/doc/user/group/saml_sso/scim_setup.md b/doc/user/group/saml_sso/scim_setup.md index d7d663f4115..d1e9ba29378 100644 --- a/doc/user/group/saml_sso/scim_setup.md +++ b/doc/user/group/saml_sso/scim_setup.md @@ -1,7 +1,7 @@ --- type: howto, reference stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -49,22 +49,21 @@ Once [Group Single Sign-On](index.md) has been configured, we can: ### Azure configuration steps -The SAML application that was created during [Single sign-on](index.md) setup for [Azure](https://docs.microsoft.com/en-us/azure/active-directory/manage-apps/view-applications-portal) now needs to be set up for SCIM. +The SAML application that was created during [Single sign-on](index.md) setup for [Azure](https://docs.microsoft.com/en-us/azure/active-directory/manage-apps/view-applications-portal) now needs to be set up for SCIM. You can refer to [Azure SCIM setup documentation](https://docs.microsoft.com/en-us/azure/active-directory/app-provisioning/use-scim-to-provision-users-and-groups#getting-started). -1. Enable automatic provisioning and administrative credentials by following the - [Azure's SCIM setup documentation](https://docs.microsoft.com/en-us/azure/active-directory/app-provisioning/use-scim-to-provision-users-and-groups#provisioning-users-and-groups-to-applications-that-support-scim). +1. In your app, go to the Provisioning tab, and set the **Provisioning Mode** to **Automatic**. + Then fill in the **Admin Credentials**, and save. The **Tenant URL** and **secret token** are the items + retrieved in the [previous step](#gitlab-configuration). -During this configuration, note the following: +1. After saving, two more tabs appear: -- The `Tenant URL` and `secret token` are the items retrieved in the - [previous step](#gitlab-configuration). -- We recommend setting a notification email and selecting the **Send an email notification when a failure occurs** checkbox. -- For mappings, we only leave `Synchronize Azure Active Directory Users to AppName` enabled. - `Synchronize Azure Active Directory Groups to AppName` is usually disabled. However, this - does not mean Azure AD users cannot be provisioned in groups. Leaving it enabled does not break - the SCIM user provisioning, but causes errors in Azure AD that may be confusing and misleading. + - **Settings**: We recommend setting a notification email and selecting the **Send an email notification when a failure occurs** checkbox. + You also control what is actually synced by selecting the **Scope**. For example, **Sync only assigned users and groups** only syncs the users and groups assigned to the application. Otherwise, it syncs the whole Active Directory. -You can then test the connection by clicking on **Test Connection**. If the connection is successful, be sure to save your configuration before moving on. See below for [troubleshooting](#troubleshooting). + - **Mappings**: We recommend keeping **Provision Azure Active Directory Users** enabled, and disable **Provision Azure Active Directory Groups**. + Leaving **Provision Azure Active Directory Groups** enabled does not break the SCIM user provisioning, but it causes errors in Azure AD that may be confusing and misleading. + +1. You can then test the connection by selecting **Test Connection**. If the connection is successful, save your configuration before moving on. See below for [troubleshooting](#troubleshooting). #### Configure attribute mapping @@ -93,11 +92,6 @@ For guidance, you can view [an example configuration in the troubleshooting refe 1. Save all changes. 1. In the **Provisioning** step, set the `Provisioning Status` to `On`. - NOTE: - You can control what is actually synced by selecting the `Scope`. For example, - `Sync only assigned users and groups` only syncs the users assigned to - the application (`Users and groups`), otherwise, it syncs the whole Active Directory. - Once enabled, the synchronization details and any errors appears on the bottom of the **Provisioning** screen, together with a link to the audit events. diff --git a/doc/user/group/settings/group_access_tokens.md b/doc/user/group/settings/group_access_tokens.md index 816edb629f5..769943cd5d8 100644 --- a/doc/user/group/settings/group_access_tokens.md +++ b/doc/user/group/settings/group_access_tokens.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" type: reference, howto --- @@ -140,8 +140,11 @@ To enable or disable group access token creation for all sub-groups in a top-lev Even when creation is disabled, you can still use and revoke existing group access tokens. -## Bot users +## Bot users for groups Each time you create a group access token, a bot user is created and added to the group. -These bot users are similar to [project bot users](../../project/settings/project_access_tokens.md#project-bot-users), but are added to groups instead of projects. For more information, see -[Project bot users](../../project/settings/project_access_tokens.md#project-bot-users). +These bot users are similar to [bot users for projects](../../project/settings/project_access_tokens.md#bot-users-for-projects), +except they are added to groups instead of projects. +These bot users do not count as licensed seats. + +For more information, see [Bot users for projects](../../project/settings/project_access_tokens.md#bot-users-for-projects). diff --git a/doc/user/group/settings/import_export.md b/doc/user/group/settings/import_export.md index 5745c499c5c..7b63466656d 100644 --- a/doc/user/group/settings/import_export.md +++ b/doc/user/group/settings/import_export.md @@ -16,7 +16,7 @@ You can also [export projects](../../project/settings/import_export.md). Prerequisite: -- You must have the [Owner role](../../permissions.md) for the group. +- You must have the Owner role for the group. To enable import and export for a group: @@ -69,7 +69,7 @@ in GitLab 14.6 and replaced by [GitLab Migration](../import/). Prerequisites: -- You must have the [Owner role](../../permissions.md) for the group. +- You must have the Owner role for the group. To export the contents of a group: diff --git a/doc/user/group/subgroups/index.md b/doc/user/group/subgroups/index.md index ef984a76a7d..46dea81ae9f 100644 --- a/doc/user/group/subgroups/index.md +++ b/doc/user/group/subgroups/index.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference, howto, concepts --- @@ -158,7 +158,7 @@ added to), add the user to the new subgroup again with a higher set of permissio For example, if User 1 was first added to group `one/two` with Developer permissions, then they inherit those permissions in every other subgroup -of `one/two`. To give them the [Maintainer role](../../permissions.md) for group `one/two/three/four`, +of `one/two`. To give them the Maintainer role for group `one/two/three/four`, you would add them again in that group as Maintainer. Removing them from that group, the permissions fall back to those of the ancestor group. diff --git a/doc/user/group/value_stream_analytics/index.md b/doc/user/group/value_stream_analytics/index.md index 4663cfc8bfd..ccf77f79fd9 100644 --- a/doc/user/group/value_stream_analytics/index.md +++ b/doc/user/group/value_stream_analytics/index.md @@ -117,16 +117,15 @@ production environment for all merge requests deployed in the given time period. The "Recent Activity" metrics near the top of the page are measured as follows: - **New Issues:** the number of issues created in the date range. -- **Deploys:** the number of deployments <sup>1</sup> to production <sup>2</sup> in the date range. -- **Deployment Frequency:** the average number of deployments <sup>1</sup> to production <sup>2</sup> +- **Deploys:** the number of deployments to production in the date range. +- **Deployment Frequency:** the average number of deployments to production per day in the date range. -1. To give a more accurate representation of deployments that actually completed successfully, - the calculation for these two metrics changed in GitLab 13.9 from using the time a deployment was - created to the time a deployment finished. If you were referencing this metric prior to 13.9, please - keep this slight change in mind. -1. To see deployment metrics, you must have a - [production environment configured](../../../ci/environments/index.md#deployment-tier-of-environments). +To see deployment metrics, you must have a [production environment configured](../../../ci/environments/index.md#deployment-tier-of-environments). + +NOTE: +In GitLab 13.9 and later, deployment metrics are calculated based on when the deployment was finished. +In GitLab 13.8 and earlier, deployment metrics are calculated based on when the deployment was created. You can learn more about these metrics in our [analytics definitions](../../analytics/index.md). diff --git a/doc/user/index.md b/doc/user/index.md index a3b7cfc4b3c..f4a7cd6a5d9 100644 --- a/doc/user/index.md +++ b/doc/user/index.md @@ -44,7 +44,7 @@ GitLab is a Git-based platform that integrates a great number of essential tools - Tracking proposals for new implementations, bug reports, and feedback with a fully featured [Issue tracker](project/issues/index.md). - Organizing and prioritizing with [issue boards](project/issue_board.md). -- Reviewing code in [Merge Requests](project/merge_requests/index.md) with live-preview changes per +- Reviewing code in [merge requests](project/merge_requests/index.md) with live-preview changes per branch with [Review Apps](../ci/review_apps/index.md). - Building, testing, and deploying with built-in [Continuous Integration](../ci/index.md). - Deploying personal and professional static websites with [GitLab Pages](project/pages/index.md). @@ -56,7 +56,7 @@ GitLab is a Git-based platform that integrates a great number of essential tools With GitLab Enterprise Edition, you can also: - Improve collaboration with: - - [Merge Request Approvals](project/merge_requests/approvals/index.md). + - [Merge request approvals](project/merge_requests/approvals/index.md). - [Multiple Assignees for Issues](project/issues/multiple_assignees_for_issues.md). - [Multiple issue boards](project/issue_board.md#multiple-issue-boards). - Create formal relationships between issues with [linked issues](project/issues/related_issues.md). @@ -152,8 +152,8 @@ it all at once, from one single project. - [Repositories](project/repository/index.md): Host your codebase in repositories with version control and as part of a fully integrated platform. - [Issues](project/issues/index.md): Explore the best of GitLab Issues' features. -- [Merge Requests](project/merge_requests/index.md): Collaborate on code, - reviews, live preview changes per branch, and request approvals with Merge Requests. +- [Merge requests](project/merge_requests/index.md): Collaborate on code, + reviews, live preview changes per branch, and request approvals with merge requests. - [Milestones](project/milestones/index.md): Work on multiple issues and merge requests towards the same target date with Milestones. diff --git a/doc/user/infrastructure/clusters/connect/index.md b/doc/user/infrastructure/clusters/connect/index.md index 9e57622875d..06d1307984d 100644 --- a/doc/user/infrastructure/clusters/connect/index.md +++ b/doc/user/infrastructure/clusters/connect/index.md @@ -15,28 +15,6 @@ If you don't have a cluster yet, create one and connect it to GitLab through the You can also create a new cluster from GitLab using [Infrastructure as Code](../../iac/index.md#create-a-new-cluster-through-iac). --> -## Supported cluster versions - -GitLab is committed to support at least two production-ready Kubernetes minor -versions at any given time. We regularly review the versions we support, and -provide a three-month deprecation period before we remove support of a specific -version. The range of supported versions is based on the evaluation of: - -- The versions supported by major managed Kubernetes providers. -- The versions [supported by the Kubernetes community](https://kubernetes.io/releases/version-skew-policy/#supported-versions). - -GitLab supports the following Kubernetes versions, and you can upgrade your -Kubernetes version to any supported version at any time: - -- 1.20 (support ends on July 22, 2022) -- 1.19 (support ends on February 22, 2022) -- 1.18 (support ends on November 22, 2021) -- 1.17 (support ends on September 22, 2021) - -[Adding support to other versions of Kubernetes is managed under this epic](https://gitlab.com/groups/gitlab-org/-/epics/4827). - -Some GitLab features may support versions outside the range provided here. - ## Cluster levels (DEPRECATED) > [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. diff --git a/doc/user/infrastructure/clusters/manage/management_project_applications/certmanager.md b/doc/user/infrastructure/clusters/manage/management_project_applications/certmanager.md index 12f99af8d8d..58de5f5e368 100644 --- a/doc/user/infrastructure/clusters/manage/management_project_applications/certmanager.md +++ b/doc/user/infrastructure/clusters/manage/management_project_applications/certmanager.md @@ -8,48 +8,33 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/project-templates/cluster-management/-/merge_requests/5) in GitLab 14.0. > - Support for cert-manager v1.4 was [introduced](https://gitlab.com/gitlab-org/project-templates/cluster-management/-/merge_requests/69405) in GitLab 14.3. +> - [Upgraded](https://gitlab.com/gitlab-org/project-templates/cluster-management/-/merge_requests/23) to cert-manager 1.7 in GitLab 14.8. Assuming you already have a [Cluster management project](../../../../../user/clusters/management_project.md) created from a [management project template](../../../../../user/clusters/management_project_template.md), to install cert-manager you should uncomment this line from your `helmfile.yaml`: ```yaml - - path: applications/cert-manager-1-4/helmfile.yaml + - path: applications/cert-manager/helmfile.yaml ``` NOTE: -We kept the `- path: applications/cert-manager/helmfile.yaml` with cert-manager v0.10 to facilitate -the [migration from GitLab Managed Apps to a cluster management project](../../../../clusters/migrating_from_gma_to_project_template.md). +If your Kubernetes version is earlier than 1.20 and you are [migrating from GitLab +Managed Apps to a cluster management +project](../../../../clusters/migrating_from_gma_to_project_template.md), then +you can instead use `- path: applications/cert-manager-legacy/helmfile.yaml` to +take over an existing release of cert-manager v0.10. cert-manager: - Is installed by default into the `gitlab-managed-apps` namespace of your cluster. -- Can be installed with or without a default - [Let's Encrypt `ClusterIssuer`](https://cert-manager.io/docs/configuration/acme/), which requires an - email address to be specified. The email address is used by Let's Encrypt to +- Includes a + [Let's Encrypt + `ClusterIssuer`](https://cert-manager.io/docs/configuration/acme/) enabled by + default. In the `certmanager-issuer` release, the issuer requires a valid email address + for `letsEncryptClusterIssuer.email`. Let's Encrypt uses this email address to contact you about expiring certificates and issues related to your account. - -To install cert-manager in your cluster, configure your `applications/cert-manager-1-4/helmfile.yaml` to: - -```yaml -certManager: - installed: true - letsEncryptClusterIssuer: - installed: true - email: "user@example.com" -``` - -Or without the default `ClusterIssuer`: - -```yaml -certManager: - installed: true - letsEncryptClusterIssuer: - installed: false -``` - -You can customize the installation of cert-manager by defining a -`.gitlab/managed-apps/cert-manager/values.yaml` file in your cluster -management project. Refer to the -[chart](https://github.com/jetstack/cert-manager) for the -available configuration options. +- Can be customized in `applications/cert-manager/helmfile.yaml` by passing custom + `values` to the `certmanager` release. Refer to the + [chart](https://github.com/jetstack/cert-manager) for the available + configuration options. diff --git a/doc/user/infrastructure/clusters/manage/management_project_applications/elasticstack.md b/doc/user/infrastructure/clusters/manage/management_project_applications/elasticstack.md index 3bd675b7439..f9d0948a2bb 100644 --- a/doc/user/infrastructure/clusters/manage/management_project_applications/elasticstack.md +++ b/doc/user/infrastructure/clusters/manage/management_project_applications/elasticstack.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/infrastructure/clusters/manage/management_project_applications/prometheus.md b/doc/user/infrastructure/clusters/manage/management_project_applications/prometheus.md index fd2eed25997..f76c7363a83 100644 --- a/doc/user/infrastructure/clusters/manage/management_project_applications/prometheus.md +++ b/doc/user/infrastructure/clusters/manage/management_project_applications/prometheus.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/infrastructure/clusters/manage/management_project_applications/sentry.md b/doc/user/infrastructure/clusters/manage/management_project_applications/sentry.md index 9e5d7860a67..b968e63d632 100644 --- a/doc/user/infrastructure/clusters/manage/management_project_applications/sentry.md +++ b/doc/user/infrastructure/clusters/manage/management_project_applications/sentry.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/infrastructure/iac/index.md b/doc/user/infrastructure/iac/index.md index ceb6101688b..6fef1aa7879 100644 --- a/doc/user/infrastructure/iac/index.md +++ b/doc/user/infrastructure/iac/index.md @@ -6,36 +6,33 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Infrastructure as Code with Terraform and GitLab **(FREE)** -## Motivation +With Terraform in GitLab, you can use GitLab authentication and authorization with +your GitOps and Infrastructure-as-Code (IaC) workflows. +Use these features if you want to collaborate on Terraform code within GitLab or would like to use GitLab as a Terraform state storage that incorporates best practices out of the box. -The Terraform integration features in GitLab enable your GitOps / Infrastructure-as-Code (IaC) -workflows to tie into GitLab authentication and authorization. These features focus on -lowering the barrier to entry for teams to adopt Terraform, collaborate effectively in -GitLab, and support Terraform best practices. - -## Quick Start +## Integrate your project with Terraform > SAST test was [introduced](https://gitlab.com/groups/gitlab-org/-/epics/6655) in GitLab 14.6. -Use the following `.gitlab-ci.yml` to set up a basic Terraform project integration -for GitLab versions 14.0 and later: +In GitLab 14.0 and later, to integrate your project with Terraform, add the following +to your `.gitlab-ci.yml` file: ```yaml include: - template: Terraform.latest.gitlab-ci.yml variables: - # If not using GitLab's HTTP backend, remove this line and specify TF_HTTP_* variables + # If you do not use the GitLab HTTP backend, remove this line and specify TF_HTTP_* variables TF_STATE_NAME: default TF_CACHE_KEY: default # If your terraform files are in a subdirectory, set TF_ROOT accordingly # TF_ROOT: terraform/production ``` -This template includes the following parameters that you can override: +The `Terraform.latest.gitlab-ci.yml` template: - Uses the latest [GitLab Terraform image](https://gitlab.com/gitlab-org/terraform-images). -- Uses the [GitLab-managed Terraform State](#gitlab-managed-terraform-state) as +- Uses the [GitLab-managed Terraform state](#gitlab-managed-terraform-state) as the Terraform state storage backend. - Creates [four pipeline stages](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Terraform.latest.gitlab-ci.yml): `test`, `validate`, `build`, and `deploy`. These stages @@ -44,10 +41,12 @@ This template includes the following parameters that you can override: - Runs the [Terraform SAST scanner](../../application_security/iac_scanning/index.md#configure-iac-scanning-manually), that you can disable by creating a `SAST_DISABLED` environment variable and setting it to `1`. -The latest template described above might contain breaking changes between major GitLab releases. For users requiring more stable setups, we -recommend using the stable templates: +You can override the values in the default template by updating your `.gitlab-ci.yml` file. + +The latest template might contain breaking changes between major GitLab releases. +For a more stable template, we recommend: -- [A ready to use version](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Terraform.gitlab-ci.yml) +- [A ready-to-use version](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Terraform.gitlab-ci.yml) - [A base template for customized setups](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Terraform/Base.gitlab-ci.yml) This video from January 2021 walks you through all the GitLab Terraform integration features: @@ -59,7 +58,7 @@ This video from January 2021 walks you through all the GitLab Terraform integrat <iframe src="https://www.youtube.com/embed/iGXjUrkkzDI" frameborder="0" allowfullscreen="true"> </iframe> </figure> -## GitLab Managed Terraform state +## GitLab-managed Terraform state [Terraform remote backends](https://www.terraform.io/docs/language/settings/backends/index.html) enable you to store the state file in a remote, shared store. GitLab uses the @@ -67,7 +66,7 @@ enable you to store the state file in a remote, shared store. GitLab uses the to securely store the state files in local storage (the default) or [the remote store of your choice](../../../administration/terraform_state.md). -The GitLab managed Terraform state backend can store your Terraform state easily and +The GitLab-managed Terraform state backend can store your Terraform state easily and securely. It spares you from setting up additional remote resources like Amazon S3 or Google Cloud Storage. Its features include: @@ -75,7 +74,7 @@ Amazon S3 or Google Cloud Storage. Its features include: - Locking and unlocking state. - Remote Terraform plan and apply execution. -Read more on setting up and [using GitLab Managed Terraform states](terraform_state.md). +Read more about setting up and [using GitLab-managed Terraform states](terraform_state.md). ## Terraform module registry @@ -83,12 +82,12 @@ GitLab can be used as a [Terraform module registry](../../packages/terraform_mod to create and publish Terraform modules to a private registry specific to your top-level namespace. -## Terraform integration in Merge Requests +## Terraform integration in merge requests Collaborating around Infrastructure as Code (IaC) changes requires both code changes and expected infrastructure changes to be checked and approved. GitLab provides a solution to help collaboration around Terraform code changes and their expected -effects using the Merge Request pages. This way users don't have to build custom +effects using the merge request pages. This way users don't have to build custom tools or rely on 3rd party solutions to streamline their IaC workflows. Read more on setting up and [using the merge request integrations](mr_integration.md). @@ -104,7 +103,7 @@ to manage various aspects of GitLab using Terraform. The provider is an open sou owned by GitLab, where everyone can contribute. The [documentation of the provider](https://registry.terraform.io/providers/gitlabhq/gitlab/latest/docs) -is available as part of the official Terraform provider documentations. +is available as part of the official Terraform provider documentation. ## Create a new cluster through IaC (DEPRECATED) @@ -114,66 +113,9 @@ NOTE: The linked tutorial connects the cluster to GitLab through cluster certificates, and this method was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. You can still create a cluster through IaC and then connect it to GitLab -through the [Agent](../../clusters/agent/index.md), the default and fully supported +through the [agent](../../clusters/agent/index.md), the default and fully supported method to connect clusters to GitLab. ## Troubleshooting -### `gitlab_group_share_group` resources not detected when subgroup state is refreshed - -The GitLab Terraform provider can fail to detect existing `gitlab_group_share_group` resources -due to the issue ["User with permissions cannot retrieve `share_with_groups` from the API"](https://gitlab.com/gitlab-org/gitlab/-/issues/328428). -This results in an error when running `terraform apply` because Terraform attempts to recreate an -existing resource. - -For example, consider the following group/subgroup configuration: - -```plaintext -parent-group -├── subgroup-A -└── subgroup-B -``` - -Where: - -- User `user-1` creates `parent-group`, `subgroup-A`, and `subgroup-B`. -- `subgroup-A` is shared with `subgroup-B`. -- User `terraform-user` is member of `parent-group` with inherited `owner` access to both subgroups. - -When the Terraform state is refreshed, the API query `GET /groups/:subgroup-A_id` issued by the provider does not return the -details of `subgroup-B` in the `shared_with_groups` array. This leads to the error. - -To workaround this issue, make sure to apply one of the following conditions: - -1. The `terraform-user` creates all subgroup resources. -1. Grant Maintainer or Owner role to the `terraform-user` user on `subgroup-B`. -1. The `terraform-user` inherited access to `subgroup-B` and `subgroup-B` contains at least one project. - -### Invalid CI/CD syntax error when using the "latest" base template - -On GitLab 14.2 and later, you might get a CI/CD syntax error when using the -`latest` Base Terraform template: - -```yaml -include: - - template: Terraform/Base.latest.gitlab-ci.yml - -my-Terraform-job: - extends: .init -``` - -The base template's [jobs were renamed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67719/) -with better Terraform-specific names. To resolve the syntax error, you can: - -- Use the stable `Terraform/Base.gitlab-ci.yml` template, which has not changed. -- Update your pipeline configuration to use the new job names in - `https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates/Terraform/Base.latest.gitlab-ci.yml`. - For example: - - ```yaml - include: - - template: Terraform/Base.latest.gitlab-ci.yml - - my-Terraform-job: - extends: .terraform:init # The updated name. - ``` +See the [troubleshooting](troubleshooting.md) documentation. diff --git a/doc/user/infrastructure/iac/mr_integration.md b/doc/user/infrastructure/iac/mr_integration.md index ab59f8ad64b..8c135f18bc1 100644 --- a/doc/user/infrastructure/iac/mr_integration.md +++ b/doc/user/infrastructure/iac/mr_integration.md @@ -4,9 +4,9 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Terraform integration in Merge Requests **(FREE)** +# Terraform integration in merge requests **(FREE)** -Collaborating around Infrastructure as Code (IaC) changes requires both code changes and expected infrastructure changes to be checked and approved. GitLab provides a solution to help collaboration around Terraform code changes and their expected effects using the Merge Request pages. This way users don't have to build custom tools or rely on 3rd party solutions to streamline their IaC workflows. +Collaborating around Infrastructure as Code (IaC) changes requires both code changes and expected infrastructure changes to be checked and approved. GitLab provides a solution to help collaboration around Terraform code changes and their expected effects using the merge request pages. This way users don't have to build custom tools or rely on 3rd party solutions to streamline their IaC workflows. ## Output Terraform Plan information into a merge request @@ -16,14 +16,14 @@ enabling you to see statistics about the resources that Terraform creates, modifies, or destroys. WARNING: -Like any other job artifact, Terraform Plan data is viewable by anyone with the Guest [role](../../permissions.md) on the repository. +Like any other job artifact, Terraform Plan data is viewable by anyone with the Guest role for the repository. Neither Terraform nor GitLab encrypts the plan file by default. If your Terraform Plan includes sensitive data such as passwords, access tokens, or certificates, we strongly recommend encrypting plan output or modifying the project visibility settings. ## Configure Terraform report artifacts -GitLab ships with a [pre-built CI template](index.md#quick-start) that uses GitLab Managed Terraform state and integrates Terraform changes into merge requests. We recommend customizing the pre-built image and relying on the `gitlab-terraform` helper provided within for a quick setup. +GitLab ships with a [pre-built CI template](index.md#integrate-your-project-with-terraform) that uses GitLab Managed Terraform state and integrates Terraform changes into merge requests. We recommend customizing the pre-built image and relying on the `gitlab-terraform` helper provided within for a quick setup. To manually configure a GitLab Terraform Report artifact: @@ -83,7 +83,7 @@ To manually configure a GitLab Terraform Report artifact: 1. Running the pipeline displays the widget in the merge request, like this: -  +  1. Clicking the **View Full Log** button in the widget takes you directly to the plan output present in the pipeline logs: @@ -151,8 +151,8 @@ apply: ### Multiple Terraform Plan reports -Starting with GitLab version 13.2, you can display multiple reports on the Merge Request -page. The reports also display the `artifacts: name:`. See example below for a suggested setup. +Starting with GitLab version 13.2, you can display multiple reports on a merge request. +The reports also display the `artifacts: name:`. See example below for a suggested setup. ```yaml default: diff --git a/doc/user/infrastructure/iac/terraform_state.md b/doc/user/infrastructure/iac/terraform_state.md index a45ef02622f..8fd38215b04 100644 --- a/doc/user/infrastructure/iac/terraform_state.md +++ b/doc/user/infrastructure/iac/terraform_state.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab managed Terraform State **(FREE)** +# GitLab-managed Terraform state **(FREE)** > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2673) in GitLab 13.0. @@ -19,7 +19,7 @@ Using local storage (the default) on clustered deployments of GitLab will result a split state across nodes, making subsequent executions of Terraform inconsistent. You are highly advised to use a remote storage resource in that case. -The GitLab managed Terraform state backend can store your Terraform state easily and +The GitLab-managed Terraform state backend can store your Terraform state easily and securely, and spares you from setting up additional remote resources like Amazon S3 or Google Cloud Storage. Its features include: @@ -33,11 +33,13 @@ before using this feature. ## Permissions for using Terraform -In GitLab version 13.1, the [Maintainer role](../../permissions.md) was required to use a -GitLab managed Terraform state backend. In GitLab versions 13.2 and greater, the -[Maintainer role](../../permissions.md) is required to lock, unlock, and write to the state -(using `terraform apply`), while the [Developer role](../../permissions.md) is required to read -the state (using `terraform plan -lock=false`). +In GitLab version 13.1, at least the Maintainer role was required to use a +GitLab managed Terraform state backend. + +In GitLab versions 13.2 and later, at least: + +- The Maintainer role is required to lock, unlock, and write to the state (using `terraform apply`). +- The Developer role is required to read the state (using `terraform plan -lock=false`). ## Set up GitLab-managed Terraform state @@ -72,9 +74,8 @@ local machine, this is a simple way to get started: 1. On your local machine, run `terraform init`, passing in the following options, replacing `<YOUR-STATE-NAME>`, `<YOUR-PROJECT-ID>`, `<YOUR-USERNAME>` and `<YOUR-ACCESS-TOKEN>` with the relevant values. This command initializes your - Terraform state, and stores that state in your GitLab project. The name of - your state can contain only uppercase and lowercase letters, decimal digits, - hyphens, and underscores. This example uses `gitlab.com`: + Terraform state, and stores that state in your GitLab project. This example + uses `gitlab.com`: ```shell terraform init \ @@ -88,6 +89,10 @@ local machine, this is a simple way to get started: -backend-config="retry_wait_min=5" ``` + WARNING: + The name of your state can contain only uppercase and lowercase letters, decimal digits, + hyphens, and underscores. + If you already have a GitLab-managed Terraform state, you can use the `terraform init` command with the pre-populated parameters values: @@ -205,7 +210,7 @@ and the CI YAML file: The output from the above `terraform` commands should be viewable in the job logs. WARNING: -Like any other job artifact, Terraform plan data is viewable by anyone with the Guest [role](../../permissions.md) on the repository. +Like any other job artifact, Terraform plan data is viewable by anyone with the Guest role on the repository. Neither Terraform nor GitLab encrypts the plan file by default. If your Terraform plan includes sensitive data such as passwords, access tokens, or certificates, GitLab strongly recommends encrypting plan output or modifying the project visibility settings. @@ -214,7 +219,7 @@ recommends encrypting plan output or modifying the project visibility settings. See [this reference project](https://gitlab.com/gitlab-org/configure/examples/gitlab-terraform-aws) using GitLab and Terraform to deploy a basic AWS EC2 in a custom VPC. -## Using a GitLab managed Terraform state backend as a remote data source +## Using a GitLab-managed Terraform state backend as a remote data source You can use a GitLab-managed Terraform state as a [Terraform data source](https://www.terraform.io/docs/language/state/remote-state-data.html). @@ -255,16 +260,16 @@ An example setup is shown below: Outputs from the data source can now be referenced in your Terraform resources using `data.terraform_remote_state.example.outputs.<OUTPUT-NAME>`. -You need at least the [Developer role](../../permissions.md) in the target project +You need at least the Developer role in the target project to read the Terraform state. -## Migrating to GitLab Managed Terraform state +## Migrating to GitLab-managed Terraform state Terraform supports copying the state when the backend is changed or reconfigured. This can be useful if you need to migrate from another backend to -GitLab managed Terraform state. Using a local terminal is recommended to run the commands needed for migrating to GitLab Managed Terraform state. +GitLab-managed Terraform state. Using a local terminal is recommended to run the commands needed for migrating to GitLab-managed Terraform state. -The following example demonstrates how to change the state name, the same workflow is needed to migrate to GitLab Managed Terraform state from a different state storage backend. +The following example demonstrates how to change the state name, the same workflow is needed to migrate to GitLab-managed Terraform state from a different state storage backend. ### Setting up the initial backend @@ -313,6 +318,7 @@ the old state is, you can tell it about the new location: TF_ADDRESS="https://gitlab.com/api/v4/projects/${PROJECT_ID}/terraform/state/new-state-name" terraform init \ + -migrate-state \ -backend-config=address=${TF_ADDRESS} \ -backend-config=lock_address=${TF_ADDRESS}/lock \ -backend-config=unlock_address=${TF_ADDRESS}/lock \ @@ -364,7 +370,7 @@ location. You can then go back to running it in GitLab CI/CD. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/273592) in GitLab 13.8. -Users with Developer and greater [permissions](../../permissions.md) can view the +Users with at least the Developer role can view the state files attached to a project at **Infrastructure > Terraform**. Users with the Maintainer role can perform commands on the state files. The user interface contains these fields: @@ -383,9 +389,27 @@ Additional improvements to the [graphical interface for managing state files](https://gitlab.com/groups/gitlab-org/-/epics/4563) are planned. +## Manage individual Terraform state versions + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207347) in GitLab 13.4. + +Individual state versions can be managed using the GitLab REST API. + +Users with the [Developer role](../../permissions.md) can retrieve state versions using their serial number. To retrieve a version: + +```shell +curl --header "Private-Token: <your_access_token>" "https://gitlab.example.com/api/v4/projects/<your_project_id>/terraform/state/<your_state_name>/versions/<version-serial>" +``` + +Users with the [Maintainer role](../../permissions.md) can remove state versions using their serial number. To remove a version: + +```shell +curl --header "Private-Token: <your_access_token>" --request DELETE "https://gitlab.example.com/api/v4/projects/<your_project_id>/terraform/state/<your_state_name>/versions/<version-serial>" +``` + ## Remove a state file -Users with Maintainer and greater [permissions](../../permissions.md) can use the +Users with at least the Maintainer role can use the following options to remove a state file: - **GitLab UI**: Go to **Infrastructure > Terraform**. In the **Actions** column, @@ -444,3 +468,15 @@ This happens because the value of `$CI_JOB_TOKEN` is only valid for the duration As a workaround, use [http backend configuration variables](https://www.terraform.io/docs/language/settings/backends/http.html#configuration-variables) in your CI job, which is what happens behind the scenes when following the [Get started using GitLab CI](#get-started-using-gitlab-ci) instructions. + +### Error: "address": required field is not set + +By default, we set `TF_ADDRESS` to `${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${TF_STATE_NAME}`. +If you don't set `TF_STATE_NAME` or `TF_ADDRESS` in your job, the job fails with the error message +`Error: "address": required field is not set`. + +To resolve this, ensure that either `TF_ADDRESS` or `TF_STATE_NAME` is accessible in the +job that returned the error: + +1. Configure the [CI/CD environment scope](../../../ci/variables/#add-a-cicd-variable-to-a-project) for the job. +1. Set the job's [environment](../../../ci/yaml/#environment), matching the environment scope from the previous step. diff --git a/doc/user/infrastructure/iac/troubleshooting.md b/doc/user/infrastructure/iac/troubleshooting.md new file mode 100644 index 00000000000..ecefa20db99 --- /dev/null +++ b/doc/user/infrastructure/iac/troubleshooting.md @@ -0,0 +1,68 @@ +--- +stage: Configure +group: Configure +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Troubleshooting the Terraform integration with GitLab + +When you are using the integration with Terraform and GitLab, you might experience issues you need to troubleshoot. + +## `gitlab_group_share_group` resources not detected when subgroup state is refreshed + +The GitLab Terraform provider can fail to detect existing `gitlab_group_share_group` resources +due to the issue ["User with permissions cannot retrieve `share_with_groups` from the API"](https://gitlab.com/gitlab-org/gitlab/-/issues/328428). +This results in an error when running `terraform apply` because Terraform attempts to recreate an +existing resource. + +For example, consider the following group/subgroup configuration: + +```plaintext +parent-group +├── subgroup-A +└── subgroup-B +``` + +Where: + +- User `user-1` creates `parent-group`, `subgroup-A`, and `subgroup-B`. +- `subgroup-A` is shared with `subgroup-B`. +- User `terraform-user` is member of `parent-group` with inherited `owner` access to both subgroups. + +When the Terraform state is refreshed, the API query `GET /groups/:subgroup-A_id` issued by the provider does not return the +details of `subgroup-B` in the `shared_with_groups` array. This leads to the error. + +To workaround this issue, make sure to apply one of the following conditions: + +1. The `terraform-user` creates all subgroup resources. +1. Grant Maintainer or Owner role to the `terraform-user` user on `subgroup-B`. +1. The `terraform-user` inherited access to `subgroup-B` and `subgroup-B` contains at least one project. + +### Invalid CI/CD syntax error when using the `latest` base template + +On GitLab 14.2 and later, you might get a CI/CD syntax error when using the +`latest` Base Terraform template: + +```yaml +include: + - template: Terraform/Base.latest.gitlab-ci.yml + +my-Terraform-job: + extends: .init +``` + +The base template's [jobs were renamed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67719/) +with better Terraform-specific names. To resolve the syntax error, you can: + +- Use the stable `Terraform/Base.gitlab-ci.yml` template, which has not changed. +- Update your pipeline configuration to use the new job names in + `https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates/Terraform/Base.latest.gitlab-ci.yml`. + For example: + + ```yaml + include: + - template: Terraform/Base.latest.gitlab-ci.yml + + my-Terraform-job: + extends: .terraform:init # The updated name. + ``` diff --git a/doc/user/markdown.md b/doc/user/markdown.md index 3d640185a55..a16f8fd39b1 100644 --- a/doc/user/markdown.md +++ b/doc/user/markdown.md @@ -539,6 +539,7 @@ GitLab Flavored Markdown recognizes the following: | repository file references | `[README](doc/README.md)` | | | | repository file line references | `[README](doc/README.md#L13)` | | | | [alert](../operations/incident_management/alerts.md) | `^alert#123` | `namespace/project^alert#123` | `project^alert#123` | +| contact | `[contact:test@example.com]` | | | 1. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/222483) in GitLab 13.7. diff --git a/doc/user/packages/composer_repository/index.md b/doc/user/packages/composer_repository/index.md index 23bd140d4b7..5cfb4278a2c 100644 --- a/doc/user/packages/composer_repository/index.md +++ b/doc/user/packages/composer_repository/index.md @@ -349,6 +349,16 @@ used to access them: subgroups. - A project deploy token only has access to packages published to that particular project. +## Troubleshooting + +To improve performance, Composer caches files related to a package. Note that Composer doesn't remove data by +itself. The cache grows as new packages are installed. If you encounter issues, clear the cache with +this command: + +```shell +composer clearcache +``` + ## Supported CLI commands The GitLab Composer repository supports the following Composer CLI commands: diff --git a/doc/user/packages/container_registry/index.md b/doc/user/packages/container_registry/index.md index c77fc5a0f4b..b28f0cbfb35 100644 --- a/doc/user/packages/container_registry/index.md +++ b/doc/user/packages/container_registry/index.md @@ -489,11 +489,13 @@ Container Registry. ## Limitations - Moving or renaming existing Container Registry repositories is not supported -once you have pushed images, because the images are stored in a path that matches -the repository path. To move or rename a repository with a -Container Registry, you must delete all existing images. + once you have pushed images, because the images are stored in a path that matches + the repository path. To move or rename a repository with a + Container Registry, you must delete all existing images. + Community suggestions to work around this limitation have been shared in + [issue 18383](https://gitlab.com/gitlab-org/gitlab/-/issues/18383#possible-workaround). - Prior to GitLab 12.10, any tags that use the same image ID as the `latest` tag -are not deleted by the cleanup policy. + are not deleted by the cleanup policy. ## Disable the Container Registry for a project @@ -581,103 +583,27 @@ For information on how to update your images, see the [Docker help](https://docs ### `Blob unknown to registry` error when pushing a manifest list -When [pushing a Docker manifest list](https://docs.docker.com/engine/reference/commandline/manifest/#create-and-push-a-manifest-list) to the GitLab Container Registry, you may receive the error `manifest blob unknown: blob unknown to registry`. [This issue](https://gitlab.com/gitlab-org/gitlab/-/issues/209008) occurs when the individual child manifests referenced in the manifest list were not pushed to the same repository. +When [pushing a Docker manifest list](https://docs.docker.com/engine/reference/commandline/manifest/#create-and-push-a-manifest-list) +to the GitLab Container Registry, you may receive the error +`manifest blob unknown: blob unknown to registry`. This is likely caused by having multiple images +with different architectures, spread out over several repositories instead of the same repository. -For example, you may have two individual images, one for `amd64` and another for `arm64v8`, and you want to build a multi-arch image with them. The `amd64` and `arm64v8` images must be pushed to the same repository where you want to push the multi-arch image. +For example, you may have two images, each representing an architecture: -As a workaround, you should include the architecture in the tag name of individual images. For example, use `mygroup/myapp:1.0.0-amd64` instead of using sub repositories, like `mygroup/myapp/amd64:1.0.0`. You can then tag the manifest list with `mygroup/myapp:1.0.0`. +- The `amd64` platform +- The `arm64v8` platform -### The cleanup policy doesn't delete any tags +To build a multi-arch image with these images, you must push them to the same repository as the +multi-arch image. -There can be different reasons behind this: - -- In GitLab 13.6 and earlier, when you run the cleanup policy you may expect it to delete tags and - it does not. This occurs when the cleanup policy is saved without editing the value in the - **Remove tags matching** field. This field has a grayed out `.*` value as a placeholder. Unless - `.*` (or another regex pattern) is entered explicitly into the field, a `nil` value is submitted. - This value prevents the saved cleanup policy from matching any tags. As a workaround, edit the - cleanup policy. In the **Remove tags matching** field, enter `.*` and save. This value indicates - that all tags should be removed. - -- If you are on GitLab self-managed instances and you have 1000+ tags in a container repository, you - might run into a [Container Registry token expiration issue](https://gitlab.com/gitlab-org/gitlab/-/issues/288814), - with `error authorizing context: invalid token` in the logs. - - To fix this, there are two workarounds: - - - If you are on GitLab 13.9 or later, you can [set limits for the cleanup policy](reduce_container_registry_storage.md#set-cleanup-limits-to-conserve-resources). - This limits the cleanup execution in time, and avoids the expired token error. - - - Extend the expiration delay of the Container Registry authentication tokens. This defaults to 5 - minutes. You can set a custom value by running - `ApplicationSetting.last.update(container_registry_token_expire_delay: <integer>)` in the Rails - console, where `<integer>` is the desired number of minutes. For reference, 15 minutes is the - value currently in use for GitLab.com. Be aware that by extending this value you increase the - time required to revoke permissions. - -If the previous fixes didn't work or you are on earlier versions of GitLab, you can generate a list -of the tags that you want to delete, and then use that list to delete the tags. To do this, follow -these steps: - -1. Run the following shell script. The command just before the `for` loop ensures that - `list_o_tags.out` is always reinitialized when starting the loop. After running this command, all - the tags' names will be in the `list_o_tags.out` file: - - ```shell - # Get a list of all tags in a certain container repository while considering [pagination](../../../api/index.md#pagination) - echo -n "" > list_o_tags.out; for i in {1..N}; do curl --header 'PRIVATE-TOKEN: <PAT>' "https://gitlab.example.com/api/v4/projects/<Project_id>/registry/repositories/<container_repo_id>/tags?per_page=100&page=${i}" | jq '.[].name' | sed 's:^.\(.*\).$:\1:' >> list_o_tags.out; done - ``` - - If you have Rails console access, you can enter the following commands to retrieve a list of tags limited by date: - - ```shell - output = File.open( "/tmp/list_o_tags.out","w" ) - Project.find(<Project_id>).container_repositories.find(<container_repo_id>).tags.each do |tag| - output << tag.name + "\n" if tag.created_at < 1.month.ago - end;nil - output.close - ``` - - This set of commands creates a `/tmp/list_o_tags.out` file listing all tags with a `created_at` date of older than one month. - -1. Remove from the `list_o_tags.out` file any tags that you want to keep. Here are some example - `sed` commands for this. Note that these commands are simply examples. You may change them to - best suit your needs: - - ```shell - # Remove the `latest` tag from the file - sed -i '/latest/d' list_o_tags.out - - # Remove the first N tags from the file - sed -i '1,Nd' list_o_tags.out - - # Remove the tags starting with `Av` from the file - sed -i '/^Av/d' list_o_tags.out - - # Remove the tags ending with `_v3` from the file - sed -i '/_v3$/d' list_o_tags.out - ``` - - If you are running macOS, you must add `.bak` to the commands. For example: - - ```shell - sed -i .bak '/latest/d' list_o_tags.out - ``` - -1. Double-check the `list_o_tags.out` file to make sure it contains only the tags that you want to - delete. - -1. Run this shell script to delete the tags in the `list_o_tags.out` file: - - ```shell - # loop over list_o_tags.out to delete a single tag at a time - while read -r LINE || [[ -n $LINE ]]; do echo ${LINE}; curl --request DELETE --header 'PRIVATE-TOKEN: <PAT>' "https://gitlab.example.com/api/v4/projects/<Project_id>/registry/repositories/<container_repo_id>/tags/${LINE}"; sleep 0.1; echo; done < list_o_tags.out > delete.logs - ``` +To address the `Blob unknown to registry` error, include the architecture in the tag name of +individual images. For example, use `mygroup/myapp:1.0.0-amd64` and `mygroup/myapp:1.0.0-arm64v8`. +You can then tag the manifest list with `mygroup/myapp:1.0.0`. ### Troubleshoot as a GitLab server administrator Troubleshooting the GitLab Container Registry, most of the times, requires -you to log in to GitLab server with the Administrator role. +you to log in to GitLab server with administrator access. [Read how to troubleshoot the Container Registry](../../../administration/packages/container_registry.md#troubleshooting). diff --git a/doc/user/packages/container_registry/reduce_container_registry_data_transfer.md b/doc/user/packages/container_registry/reduce_container_registry_data_transfer.md new file mode 100644 index 00000000000..5f678a661f8 --- /dev/null +++ b/doc/user/packages/container_registry/reduce_container_registry_data_transfer.md @@ -0,0 +1,133 @@ +--- +stage: Package +group: Package +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Reduce Container Registry data transfers **(FREE)** + +Depending on the frequency with which images or tags are downloaded from the Container Registry, +data transfers can exceed the GitLab limit. This page offers several recommendations and tips for +reducing the amount of data you transfer with the Container Registry. +are downloaded from the Container Registry, data transfers can exceed the GitLab limit. This page +offers several recommendations and tips for reducing the amount of data you transfer with the +Container Registry. + +## Check data transfer use + +Transfer usage is not available within the GitLab UI. [GitLab-#350905](https://gitlab.com/gitlab-org/gitlab/-/issues/350905) +is the epic tracking the work to surface this information. In the interim, GitLab team members may reach out to customers that have +significantly exceeded their transfer limits to better understand their use case and offer ways to reduce data transfer +usage. + +## Determine image size + +Use these tools and techniques to determine your image's size: + +- [Skopeo](https://github.com/containers/skopeo): + use Skopeo's `inspect` command to examine layer count and sizes through API calls. You can + therefore inspect this data prior to running `docker pull IMAGE`. + +- Docker in CI: examine and record the image size when using GitLab CI prior to pushing an image + with Docker. For example: + + ```yaml + docker inspect "$CI_REGISTRY_IMAGE:$IMAGE_TAG" \ + | awk '/"Size": ([0-9]+)[,]?/{ printf "Final Image Size: %d\n", $2 }' + ``` + +- [Dive](https://github.com/wagoodman/dive) + is a tool for exploring a Docker image, layer contents, and discovering ways to reduce its size. + +## Reduce image size + +### Use a smaller base image + +Consider using a smaller base image, such as [Alpine Linux](https://alpinelinux.org/). +An Alpine image is around 5MB, which is several times smaller than popular base images such as +[Debian](https://hub.docker.com/_/debian). +If your application is distributed as a self-contained static binary, such as for Go applications, +you can also consider using Docker's [scratch](https://hub.docker.com/_/scratch/) +base image. + +If you need to use a specific base image OS, look for `-slim` or `-minimal` variants, as this helps +to reduce the image size. + +Also be mindful about the operating system packages you install on top of your base image. These can +add up to hundreds of megabytes. Try keeping the number of installed packages to the bare minimum. + +[Multi-stage builds](#use-multi-stage-builds) can be a powerful ally in cleaning up transient build +dependencies. + +You may also consider using tools such as these: + +- [DockerSlim](https://github.com/docker-slim/docker-slim) + provides a set of commands to reduce the size of your container images. +- [Distroless](https://github.com/GoogleContainerTools/distroless) images contain only your + application and its runtime dependencies. They don't contain package managers, shells, or any + other programs you would expect to find in a standard Linux distribution. + +### Minimize layers + +Every instruction in your Dockerfile leads to a new layer, which records the file system changes +applied during such an instruction. In general, more or larger layers lead to larger images. Try to +minimize the number of layers to install the packages in the Dockerfile. Otherwise, this may cause +each step in the build process to increase the image size. + +There are multiple strategies to reduce the number and size of layers. For example, instead of using +a `RUN` command per operating system package that you want to install (which would lead to a layer +per package), you can install all the packages on a single `RUN` command to reduce the number of +steps in the build process and reduce the size of the image. + +Another useful strategy is to ensure that you remove all transient build dependencies and disable or +empty the operating system package manager cache before and after installing a package. + +When building your images, make sure you only copy the relevant files. For Docker, using a +[`.dockerignore`](https://docs.docker.com/engine/reference/builder/#dockerignore-file) +helps ensure that the build process ignores irrelevant files. + +You can use other third-party tools to minify your images, such as [DockerSlim](https://github.com/docker-slim/docker-slim). +Be aware that if used improperly, such tools may remove dependencies that your application needs to +operate under certain conditions. Therefore, it's preferable to strive for smaller images during the +build process instead of trying to minify images afterward. + +### Use multi-stage builds + +With [multi-stage builds](https://docs.docker.com/develop/develop-images/multistage-build/), +you use multiple `FROM` statements in your Dockerfile. Each `FROM` instruction can use a different +base, and each begins a new build stage. You can selectively copy artifacts from one stage to +another, leaving behind everything you don't want in the final image. This is especially useful when +you need to install build dependencies, but you don't need them to be present in your final image. + +## Use an image pull policy + +When using the `docker` or `docker+machine` executors, you can set a [`pull_policy`](https://docs.gitlab.com/runner/executors/docker.html#using-the-if-not-present-pull-policy) +parameter in your runner `config.toml` that defines how the runner works when pulling Docker images. +To avoid transferring data when using large and rarely updated images, consider using the +`if-not-present` pull policy when pulling images from remote registries. + +## Use Docker layer caching + +When running `docker build`, each command in `Dockerfile` results in a layer. These layers are kept +as a cache and can be reused if there haven't been any changes. You can specify a tagged image to be +used as a cache source for the `docker build` command by using the `--cache-from` argument. Multiple +images can be specified as a cache source by using multiple `--cache-from` arguments. This can speed +up your builds and reduce the amount of data transferred. For more information, see the +[documentation on Docker layer caching](../../../ci/docker/using_docker_build.md#make-docker-in-docker-builds-faster-with-docker-layer-caching). + +## Move to GitLab Premium or Ultimate + +GitLab data transfer limits are set at the tier level. If you need a higher limit, consider +upgrading to [GitLab Premium or Ultimate](https://about.gitlab.com/upgrade/). + +## Purchase additional data transfer + +Read more about managing your [data transfer limits](../../../subscriptions/gitlab_com/#purchase-more-storage-and-transfer). + +## Related issues + +- You may want to rebuild your image when the base Docker image is updated. However, the + [pipeline subscription limit is too low](https://gitlab.com/gitlab-org/gitlab/-/issues/225278) + to leverage this feature. As a workaround, you can rebuild daily or multiple times per day. + [GitLab-#225278](https://gitlab.com/gitlab-org/gitlab/-/issues/225278) + proposes raising the limit to help with this workflow. diff --git a/doc/user/packages/container_registry/reduce_container_registry_storage.md b/doc/user/packages/container_registry/reduce_container_registry_storage.md index e2242a85b75..a2a22f138e3 100644 --- a/doc/user/packages/container_registry/reduce_container_registry_storage.md +++ b/doc/user/packages/container_registry/reduce_container_registry_storage.md @@ -16,8 +16,9 @@ to automatically manage your container registry usage. ## Check Container Registry Storage Use -The Usage Quotas page (**Settings > Usage Quotas > Storage**) displays storage usage for Packages, which includes Container Registry, -however, the storage is not being calculated. +The Usage Quotas page (**Settings > Usage Quotas > Storage**) displays storage usage for Packages, +which doesn't include the Container Registry. To track work on this, see the epic +[Storage management for the Container Registry](https://gitlab.com/groups/gitlab-org/-/epics/7226). ## Cleanup policy @@ -252,21 +253,108 @@ It is recommended you only enable container cleanup policies for projects that were created before GitLab 12.8 if you are confident the number of tags being cleaned up is minimal. -## Related topics +## More Container Registry storage reduction options -- [Delete images](index.md#delete-images) -- [Delete registry repository](../../../api/container_registry.md#delete-registry-repository) -- [Delete a registry repository tag](../../../api/container_registry.md#delete-a-registry-repository-tag) -- [Delete registry repository tags in bulk](../../../api/container_registry.md#delete-registry-repository-tags-in-bulk) -- [Delete a package](../package_registry/index.md#delete-a-package) +Here are some other options to reduce your project's use of Container Registry storage: -## Troubleshooting cleanup policies +- Use the [GitLab UI](index.md#delete-images) + to delete individual image tags or the entire repository containing all the tags. +- Use the API to [delete individual image tags](../../../api/container_registry.md#delete-a-registry-repository-tag). +- Use the API to [delete the entire container registry repository containing all the tags](../../../api/container_registry.md#delete-registry-repository). +- Use the API to [delete registry repository tags in bulk](../../../api/container_registry.md#delete-registry-repository-tags-in-bulk). -If you see the following message: +## Troubleshooting cleanup policies -"Something went wrong while updating the cleanup policy." +### `Something went wrong while updating the cleanup policy.` -Check the regex patterns to ensure they are valid. +If you see this error message, check the regex patterns to ensure they are valid. GitLab uses [RE2 syntax](https://github.com/google/re2/wiki/Syntax) for regular expressions in the cleanup policy. You can test them with the [regex101 regex tester](https://regex101.com/). View some common [regex pattern examples](#regex-pattern-examples). + +### The cleanup policy doesn't delete any tags + +There can be different reasons behind this: + +- In GitLab 13.6 and earlier, when you run the cleanup policy you may expect it to delete tags and + it does not. This occurs when the cleanup policy is saved without editing the value in the + **Remove tags matching** field. This field has a grayed out `.*` value as a placeholder. Unless + `.*` (or another regex pattern) is entered explicitly into the field, a `nil` value is submitted. + This value prevents the saved cleanup policy from matching any tags. As a workaround, edit the + cleanup policy. In the **Remove tags matching** field, enter `.*` and save. This value indicates + that all tags should be removed. + +- If you are on GitLab self-managed instances and you have 1000+ tags in a container repository, you + might run into a [Container Registry token expiration issue](https://gitlab.com/gitlab-org/gitlab/-/issues/288814), + with `error authorizing context: invalid token` in the logs. + + To fix this, there are two workarounds: + + - If you are on GitLab 13.9 or later, you can [set limits for the cleanup policy](reduce_container_registry_storage.md#set-cleanup-limits-to-conserve-resources). + This limits the cleanup execution in time, and avoids the expired token error. + + - Extend the expiration delay of the Container Registry authentication tokens. This defaults to 5 + minutes. You can set a custom value by running + `ApplicationSetting.last.update(container_registry_token_expire_delay: <integer>)` in the Rails + console, where `<integer>` is the desired number of minutes. For reference, 15 minutes is the + value currently in use for GitLab.com. Be aware that by extending this value you increase the + time required to revoke permissions. + +If the previous fixes didn't work or you are on earlier versions of GitLab, you can generate a list +of the tags that you want to delete, and then use that list to delete the tags. To do this, follow +these steps: + +1. Run the following shell script. The command just before the `for` loop ensures that + `list_o_tags.out` is always reinitialized when starting the loop. After running this command, all + the tags' names will be in the `list_o_tags.out` file: + + ```shell + # Get a list of all tags in a certain container repository while considering [pagination](../../../api/index.md#pagination) + echo -n "" > list_o_tags.out; for i in {1..N}; do curl --header 'PRIVATE-TOKEN: <PAT>' "https://gitlab.example.com/api/v4/projects/<Project_id>/registry/repositories/<container_repo_id>/tags?per_page=100&page=${i}" | jq '.[].name' | sed 's:^.\(.*\).$:\1:' >> list_o_tags.out; done + ``` + + If you have Rails console access, you can enter the following commands to retrieve a list of tags limited by date: + + ```shell + output = File.open( "/tmp/list_o_tags.out","w" ) + Project.find(<Project_id>).container_repositories.find(<container_repo_id>).tags.each do |tag| + output << tag.name + "\n" if tag.created_at < 1.month.ago + end;nil + output.close + ``` + + This set of commands creates a `/tmp/list_o_tags.out` file listing all tags with a `created_at` date of older than one month. + +1. Remove from the `list_o_tags.out` file any tags that you want to keep. Here are some example + `sed` commands for this. Note that these commands are simply examples. You may change them to + best suit your needs: + + ```shell + # Remove the `latest` tag from the file + sed -i '/latest/d' list_o_tags.out + + # Remove the first N tags from the file + sed -i '1,Nd' list_o_tags.out + + # Remove the tags starting with `Av` from the file + sed -i '/^Av/d' list_o_tags.out + + # Remove the tags ending with `_v3` from the file + sed -i '/_v3$/d' list_o_tags.out + ``` + + If you are running macOS, you must add `.bak` to the commands. For example: + + ```shell + sed -i .bak '/latest/d' list_o_tags.out + ``` + +1. Double-check the `list_o_tags.out` file to make sure it contains only the tags that you want to + delete. + +1. Run this shell script to delete the tags in the `list_o_tags.out` file: + + ```shell + # loop over list_o_tags.out to delete a single tag at a time + while read -r LINE || [[ -n $LINE ]]; do echo ${LINE}; curl --request DELETE --header 'PRIVATE-TOKEN: <PAT>' "https://gitlab.example.com/api/v4/projects/<Project_id>/registry/repositories/<container_repo_id>/tags/${LINE}"; sleep 0.1; echo; done < list_o_tags.out > delete.logs + ``` diff --git a/doc/user/packages/dependency_proxy/index.md b/doc/user/packages/dependency_proxy/index.md index 52f5a1fcc0d..925a1b3e8e6 100644 --- a/doc/user/packages/dependency_proxy/index.md +++ b/doc/user/packages/dependency_proxy/index.md @@ -93,9 +93,8 @@ You can authenticate using: - A [personal access token](../../../user/profile/personal_access_tokens.md) with the scope set to `read_registry` and `write_registry`. - A [group deploy token](../../../user/project/deploy_tokens/index.md#group-deploy-token) with the scope set to `read_registry` and `write_registry`. -Users accessing the Dependency Proxy with a personal access token or username and password require -at least [Guest membership](../../permissions.md#group-members-permissions) -to the group they pull images from. +Users accessing the Dependency Proxy with a personal access token or username and password must +have at least the Guest role for the group they pull images from. #### SAML SSO @@ -210,65 +209,8 @@ from the GitLab server. ## Reduce storage usage -Blobs are kept forever on the GitLab server, and there is no hard limit on how much data can be -stored. - -### Using the API to clear the cache - -To reclaim disk space used by image blobs that are no longer needed, use -the [Dependency Proxy API](../../../api/dependency_proxy.md) to clear the entire -cache. - -If you clear the cache, the next time a pipeline runs it must pull an image or tag from Docker Hub. - -### Cleanup policies - -#### Enable cleanup policies from within GitLab - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340777) in GitLab 14.6 - -You can enable an automatic time-to-live (TTL) policy for the Dependency Proxy from the user -interface. To do this, navigate to your group's **Settings > Packages & Registries > Dependency Proxy** -and enable the setting to automatically clear items from the cache after 90 days. - -#### Enable cleanup policies with GraphQL - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/294187) in GitLab 14.4. - -The cleanup policy is a scheduled job you can use to clear cached images that are no longer used, -freeing up additional storage space. The policies use time-to-live (TTL) logic: - -- The number of days is configured. -- All cached dependency proxy files that have not been pulled in that many days are deleted. - -Use the [GraphQL API](../../../api/graphql/reference/index.md#mutationupdatedependencyproxyimagettlgrouppolicy) -to enable and configure cleanup policies: - -```graphql -mutation { - updateDependencyProxyImageTtlGroupPolicy(input: - { - groupPath: "<your-full-group-path>", - enabled: true, - ttl: 90 - } - ) { - dependencyProxyImageTtlPolicy { - enabled - ttl - } - errors - } -} -``` - -See the [Getting started with GraphQL](../../../api/graphql/getting_started.md) -guide to learn how to make GraphQL queries. - -When the policy is initially enabled, the default TTL setting is 90 days. Once enabled, stale -dependency proxy files are queued for deletion each day. Deletion may not occur right away due to -processing time. If the image is pulled after the cached files are marked as expired, the expired -files are ignored and new files are downloaded and cached from the external registry. +For information on reducing your storage use on the Dependency Proxy, see +[Reduce Dependency Proxy storage use](reduce_dependency_proxy_storage.md). ## Docker Hub rate limits and the Dependency Proxy diff --git a/doc/user/packages/dependency_proxy/reduce_dependency_proxy_storage.md b/doc/user/packages/dependency_proxy/reduce_dependency_proxy_storage.md new file mode 100644 index 00000000000..cd04d2e696b --- /dev/null +++ b/doc/user/packages/dependency_proxy/reduce_dependency_proxy_storage.md @@ -0,0 +1,74 @@ +--- +stage: Package +group: Package +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Reduce Dependency Proxy Storage **(FREE)** + +There's no automatic removal process for blobs. Unless you delete them manually, they're stored +indefinitely. Since this impacts your +[storage usage quota](../../usage_quotas.md), +it's important that you clear unused items from the cache. This page covers several options for +doing so. + +## Check Dependency Proxy Storage Use + +The Usage Quotas page (**Settings > Usage Quotas > Storage**) displays storage usage for Packages, which includes the Dependency Proxy, +however, the storage is not yet displayed. + +## Use the API to clear the cache + +To reclaim disk space used by image blobs that are no longer needed, use the +[Dependency Proxy API](../../../api/dependency_proxy.md) +to clear the entire cache. If you clear the cache, the next time a pipeline runs it must pull an +image or tag from Docker Hub. + +## Cleanup policies + +### Enable cleanup policies from within GitLab + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340777) in GitLab 14.6 + +You can enable an automatic time-to-live (TTL) policy for the Dependency Proxy from the user +interface. To do this, navigate to your group's **Settings > Packages & Registries > Dependency Proxy** +and enable the setting to automatically clear items from the cache after 90 days. + +### Enable cleanup policies with GraphQL + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/294187) in GitLab 14.4. + +The cleanup policy is a scheduled job you can use to clear cached images that are no longer used, +freeing up additional storage space. The policies use time-to-live (TTL) logic: + +- The number of days is configured. +- All cached dependency proxy files that have not been pulled in that many days are deleted. + +Use the [GraphQL API](../../../api/graphql/reference/index.md#mutationupdatedependencyproxyimagettlgrouppolicy) +to enable and configure cleanup policies: + +```graphql +mutation { + updateDependencyProxyImageTtlGroupPolicy(input: + { + groupPath: "<your-full-group-path>", + enabled: true, + ttl: 90 + } + ) { + dependencyProxyImageTtlPolicy { + enabled + ttl + } + errors + } +} +``` + +See the [Getting started with GraphQL](../../../api/graphql/getting_started.md) +guide to learn how to make GraphQL queries. + +When the policy is initially enabled, the default TTL setting is 90 days. Once enabled, stale +dependency proxy files are queued for deletion each day. Deletion may not occur right away due to +processing time. If the image is pulled after the cached files are marked as expired, the expired +files are ignored and new files are downloaded and cached from the external registry. diff --git a/doc/user/packages/generic_packages/index.md b/doc/user/packages/generic_packages/index.md index 7b44b5bcbb7..ada6f033288 100644 --- a/doc/user/packages/generic_packages/index.md +++ b/doc/user/packages/generic_packages/index.md @@ -57,7 +57,7 @@ Example request using a personal access token: ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" \ --upload-file path/to/file.txt \ - "https://gitlab.example.com/api/v4/projects/24/packages/generic/my_package/0.0.1/file.txt?status=hidden" + "https://gitlab.example.com/api/v4/projects/24/packages/generic/my_package/0.0.1/file.txt" ``` Example response without attribute `select`: diff --git a/doc/user/packages/maven_repository/index.md b/doc/user/packages/maven_repository/index.md index 21fecc16602..6e3af6a92d5 100644 --- a/doc/user/packages/maven_repository/index.md +++ b/doc/user/packages/maven_repository/index.md @@ -263,7 +263,7 @@ The `name` must be `Job-Token`. <httpHeaders> <property> <name>Job-Token</name> - <value>${env.CI_JOB_TOKEN}</value> + <value>${CI_JOB_TOKEN}</value> </property> </httpHeaders> </configuration> @@ -725,7 +725,7 @@ You can create a new package each time the `main` branch is updated. <httpHeaders> <property> <name>Job-Token</name> - <value>${env.CI_JOB_TOKEN}</value> + <value>${CI_JOB_TOKEN}</value> </property> </httpHeaders> </configuration> @@ -742,17 +742,17 @@ You can create a new package each time the `main` branch is updated. <repositories> <repository> <id>gitlab-maven</id> - <url>${env.CI_API_V4_URL}/projects/${env.CI_PROJECT_ID}/packages/maven</url> + <url>${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/maven</url> </repository> </repositories> <distributionManagement> <repository> <id>gitlab-maven</id> - <url>${CI_API_V4_URL}/projects/${env.CI_PROJECT_ID}/packages/maven</url> + <url>${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/maven</url> </repository> <snapshotRepository> <id>gitlab-maven</id> - <url>${CI_API_V4_URL}/projects/${env.CI_PROJECT_ID}/packages/maven</url> + <url>${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/maven</url> </snapshotRepository> </distributionManagement> ``` diff --git a/doc/user/packages/npm_registry/index.md b/doc/user/packages/npm_registry/index.md index 5338e546625..1f37e5639c6 100644 --- a/doc/user/packages/npm_registry/index.md +++ b/doc/user/packages/npm_registry/index.md @@ -255,7 +255,8 @@ This regex allows almost all of the characters that npm allows, with a few excep The regex also allows for capital letters, while npm does not. -WARNING: +## Limitations + When you update the path of a user or group, or transfer a subgroup or project, you must remove any npm packages first. You cannot update the root namespace of a project with npm packages. Make sure you update your `.npmrc` files to follow diff --git a/doc/user/packages/package_registry/index.md b/doc/user/packages/package_registry/index.md index 3311b271126..a2228148447 100644 --- a/doc/user/packages/package_registry/index.md +++ b/doc/user/packages/package_registry/index.md @@ -12,6 +12,15 @@ With the GitLab Package Registry, you can use GitLab as a private or public regi of [supported package managers](#supported-package-managers). You can publish and share packages, which can be consumed as a dependency in downstream projects. +## Package workflows + +Learn how to use the GitLab Package Registry to build your own custom package workflow: + +- [Use a project as a package registry](../workflows/project_registry.md) + to publish all of your packages to one project. + +- Publish multiple different packages from one [monorepo project](../workflows/working_with_monorepos.md). + ## View packages You can view packages for your project or group. @@ -77,46 +86,10 @@ when you view the package details: You can view which pipeline published the package, and the commit and user who triggered it. However, the history is limited to five updates of a given package. -## Download a package +## Reduce storage usage -To download a package: - -1. Go to **Packages & Registries > Package Registry**. -1. Select the name of the package you want to download. -1. In the **Activity** section, select the name of the package you want to download. - -## Delete a package - -You cannot edit a package after you publish it in the Package Registry. Instead, you -must delete and recreate it. - -To delete a package, you must have suitable [permissions](../../permissions.md). - -You can delete packages by using [the API](../../../api/packages.md#delete-a-project-package) or the UI. - -To delete a package in the UI, from your group or project: - -1. Go to **Packages & Registries > Package Registry**. -1. Find the name of the package you want to delete. -1. Click **Delete**. - -The package is permanently deleted. - -## Delete files associated with a package - -To delete package files, you must have suitable [permissions](../../permissions.md). - -You can delete packages by using [the API](../../../api/packages.md#delete-a-package-file) or the UI. - -To delete package files in the UI, from your group or project: - -1. Go to **Packages & Registries > Package Registry**. -1. Find the name of the package you want to delete. -1. Select the package to view additional details. -1. Find the name of the file you would like to delete. -1. Expand the ellipsis and select **Delete file**. - -The package files are permanently deleted. +For information on reducing your storage use for the Package Registry, see +[Reduce Package Registry storage use](reduce_package_registry_storage.md). ## Disable the Package Registry @@ -135,15 +108,6 @@ You can also remove the Package Registry for your project specifically: The **Packages & Registries > Package Registry** entry is removed from the sidebar. -## Package workflows - -Learn how to use the GitLab Package Registry to build your own custom package workflow: - -- [Use a project as a package registry](../workflows/project_registry.md) - to publish all of your packages to one project. - -- Publish multiple different packages from one [monorepo project](../workflows/working_with_monorepos.md). - ## Supported package managers WARNING: @@ -166,7 +130,7 @@ The Package Registry supports the following formats: | [Go](../go_proxy/index.md) | 13.1+ | [Alpha](https://gitlab.com/groups/gitlab-org/-/epics/3043) | | [Ruby gems](../rubygems_registry/index.md) | 13.10+ | [Alpha](https://gitlab.com/groups/gitlab-org/-/epics/3200) | -[Status](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga): +[Status](../../../policy/alpha-beta-support.md): - Alpha: behind a feature flag and not officially supported. - Beta: several known issues that may prevent expected use. diff --git a/doc/user/packages/package_registry/reduce_package_registry_storage.md b/doc/user/packages/package_registry/reduce_package_registry_storage.md new file mode 100644 index 00000000000..c2e4cd8d889 --- /dev/null +++ b/doc/user/packages/package_registry/reduce_package_registry_storage.md @@ -0,0 +1,52 @@ +--- +stage: Package +group: Package +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Reduce Package Registry Storage **(FREE)** + +Without cleanup, package registries become large over time. When a large number of packages and +their files are added: + +- Fetching the list of packages becomes slower. +- They take up a large amount of storage space on the server, impacting your [storage usage quota](../../usage_quotas.md). + +We recommend deleting unnecessary packages and files. This page offers examples of how to do so. + +## Check Package Registry Storage Use + +The Usage Quotas page (**Settings > Usage Quotas > Storage**) displays storage usage for Packages. + +## Delete a package + +You cannot edit a package after you publish it in the Package Registry. Instead, you +must delete and recreate it. + +To delete a package, you must have suitable [permissions](../../permissions.md). + +You can delete packages by using [the API](../../../api/packages.md#delete-a-project-package) or the UI. + +To delete a package in the UI, from your group or project: + +1. Go to **Packages & Registries > Package Registry**. +1. Find the name of the package you want to delete. +1. Click **Delete**. + +The package is permanently deleted. + +## Delete files associated with a package + +To delete package files, you must have suitable [permissions](../../permissions.md). + +You can delete packages by using [the API](../../../api/packages.md#delete-a-package-file) or the UI. + +To delete package files in the UI, from your group or project: + +1. Go to **Packages & Registries > Package Registry**. +1. Find the name of the package you want to delete. +1. Select the package to view additional details. +1. Find the name of the file you would like to delete. +1. Expand the ellipsis and select **Delete file**. + +The package files are permanently deleted. diff --git a/doc/user/packages/pypi_repository/index.md b/doc/user/packages/pypi_repository/index.md index 4151346ec98..bf007572ac7 100644 --- a/doc/user/packages/pypi_repository/index.md +++ b/doc/user/packages/pypi_repository/index.md @@ -407,6 +407,16 @@ characters are removed. A `pip install` request for `my.package` looks for packages that match any of the three characters, such as `my-package`, `my_package`, and `my....package`. +## Troubleshooting + +To improve performance, PyPI caches files related to a package. Note that PyPI doesn't remove data by +itself. The cache grows as new packages are installed. If you encounter issues, clear the cache with +this command: + +```shell +pip cache purge +``` + ## Supported CLI commands The GitLab PyPI repository supports the following CLI commands: diff --git a/doc/user/packages/terraform_module_registry/index.md b/doc/user/packages/terraform_module_registry/index.md index bb9f32d1144..6fc47a23373 100644 --- a/doc/user/packages/terraform_module_registry/index.md +++ b/doc/user/packages/terraform_module_registry/index.md @@ -41,7 +41,7 @@ PUT /projects/:id/packages/terraform/modules/:module-name/:module-system/:module Provide the file content in the request body. -Note that, in the following example, the request must end with `/file`. +As the following example shows, requests must end with `/file`. If you send a request ending with something else, it results in a 404 error `{"error":"404 Not Found"}`. diff --git a/doc/user/permissions.md b/doc/user/permissions.md index 36c49e39151..4476b0dd75b 100644 --- a/doc/user/permissions.md +++ b/doc/user/permissions.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -46,199 +46,266 @@ The following table lists project permissions available for each role: <!-- Keep this table sorted: By topic first, then by minimum role, then alphabetically. --> -| Action | Guest | Reporter | Developer | Maintainer | Owner | -|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|----------|-----------|------------|-------| -| [Analytics](analytics/index.md):<br>View issue analytics **(PREMIUM)** | ✓ | ✓ | ✓ | ✓ | ✓ | -| [Analytics](analytics/index.md):<br>View [merge request analytics](analytics/merge_request_analytics.md) **(PREMIUM)** | ✓ | ✓ | ✓ | ✓ | ✓ | -| [Analytics](analytics/index.md):<br>View value stream analytics | ✓ | ✓ | ✓ | ✓ | ✓ | -| [Analytics](analytics/index.md):<br>View [DORA metrics](analytics/ci_cd_analytics.md) | | ✓ | ✓ | ✓ | ✓ | -| [Analytics](analytics/index.md):<br>View [CI/CD analytics](analytics/ci_cd_analytics.md) | | ✓ | ✓ | ✓ | ✓ | -| [Analytics](analytics/index.md):<br>View [code review analytics](analytics/code_review_analytics.md) **(PREMIUM)** | | ✓ | ✓ | ✓ | ✓ | -| [Analytics](analytics/index.md):<br>View [repository analytics](analytics/repository_analytics.md) | | ✓ | ✓ | ✓ | ✓ | -| [Application security](application_security/index.md):<br>View licenses in [dependency list](application_security/dependency_list/index.md) **(ULTIMATE)** | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | -| [Application security](application_security/index.md):<br>Create and run [on-demand DAST scans](application_security/dast/index.md#on-demand-scans) **(ULTIMATE)** | | | ✓ | ✓ | ✓ | -| [Application security](application_security/index.md):<br>Manage [security policy](application_security/policies/index.md) **(ULTIMATE)** | | | ✓ | ✓ | ✓ | -| [Application security](application_security/index.md):<br>View [dependency list](application_security/dependency_list/index.md) **(ULTIMATE)** | | | ✓ | ✓ | ✓ | -| [Application security](application_security/index.md):<br>View [threats list](application_security/threat_monitoring/index.md#threat-monitoring) **(ULTIMATE)** | | | ✓ | ✓ | ✓ | -| [Application security](application_security/index.md):<br>Create a [CVE ID Request](application_security/cve_id_request.md) **(FREE SAAS)** | | | | ✓ | ✓ | -| [Application security](application_security/index.md):<br>Create or assign [security policy project](application_security/policies/index.md) **(ULTIMATE)** | | | | | ✓ | -| [CI/CD](../ci/index.md):<br>Download and browse job artifacts | ✓ (*3*) | ✓ | ✓ | ✓ | ✓ | -| [CI/CD](../ci/index.md):<br>View a job log | ✓ (*3*) | ✓ | ✓ | ✓ | ✓ | -| [CI/CD](../ci/index.md):<br>View list of jobs | ✓ (*3*) | ✓ | ✓ | ✓ | ✓ | -| [CI/CD](../ci/index.md):<br>View [environments](../ci/environments/index.md) | | ✓ | ✓ | ✓ | ✓ | -| [CI/CD](../ci/index.md):<br>Cancel and retry jobs | | | ✓ | ✓ | ✓ | -| [CI/CD](../ci/index.md):<br>Create new [environments](../ci/environments/index.md) | | | ✓ | ✓ | ✓ | -| [CI/CD](../ci/index.md):<br>Run CI/CD pipeline against a protected branch | | | ✓ (*5*) | ✓ | ✓ | -| [CI/CD](../ci/index.md):<br>Stop [environments](../ci/environments/index.md) | | | ✓ | ✓ | ✓ | -| [CI/CD](../ci/index.md):<br>View a job with [debug logging](../ci/variables/index.md#debug-logging) | | | ✓ | ✓ | ✓ | -| [CI/CD](../ci/index.md):<br>Manage CI/CD variables | | | | ✓ | ✓ | -| [CI/CD](../ci/index.md):<br>Manage job triggers | | | | ✓ | ✓ | -| [CI/CD](../ci/index.md):<br>Manage group runners | | | | | ✓ | -| [CI/CD](../ci/index.md):<br>Manage project runners | | | | ✓ | ✓ | -| [CI/CD](../ci/index.md):<br>Run Web IDE's Interactive Web Terminals **(ULTIMATE ONLY)** | | | | ✓ | ✓ | -| [CI/CD](../ci/index.md):<br>Use [environment terminals](../ci/environments/index.md#web-terminals-deprecated) | | | | ✓ | ✓ | -| [CI/CD](../ci/index.md):<br>Delete pipelines | | | | | ✓ | -| [Clusters](infrastructure/clusters/index.md):<br>View [pod logs](project/clusters/kubernetes_pod_logs.md) | | | ✓ | ✓ | ✓ | -| [Clusters](infrastructure/clusters/index.md):<br>Manage clusters | | | | ✓ | ✓ | -| [Container Registry](packages/container_registry/index.md):<br>Create, edit, delete cleanup policies | | | ✓ | ✓ | ✓ | -| [Container Registry](packages/container_registry/index.md):<br>Remove a container registry image | | | ✓ | ✓ | ✓ | -| [Container Registry](packages/container_registry/index.md):<br>Update container registry | | | ✓ | ✓ | ✓ | -| [GitLab Pages](project/pages/index.md):<br>View Pages protected by [access control](project/pages/introduction.md#gitlab-pages-access-control) | ✓ | ✓ | ✓ | ✓ | ✓ | -| [GitLab Pages](project/pages/index.md):<br>Manage | | | | ✓ | ✓ | -| [GitLab Pages](project/pages/index.md):<br>Manage GitLab Pages domains and certificates | | | | ✓ | ✓ | -| [GitLab Pages](project/pages/index.md):<br>Remove GitLab Pages | | | | ✓ | ✓ | -| [Incident Management](../operations/incident_management/index.md):<br>View [alerts](../operations/incident_management/alerts.md) | | ✓ | ✓ | ✓ | ✓ | -| [Incident Management](../operations/incident_management/index.md):<br>Assign an alert | ✓| ✓ | ✓ | ✓ | ✓ | -| [Incident Management](../operations/incident_management/index.md):<br>View [incident](../operations/incident_management/incidents.md) | ✓| ✓ | ✓ | ✓ | ✓ | -| [Incident Management](../operations/incident_management/index.md):<br>Create [incident](../operations/incident_management/incidents.md) | (*17*) | ✓ | ✓ | ✓ | ✓ | -| [Incident Management](../operations/incident_management/index.md):<br>View [on-call schedules](../operations/incident_management/oncall_schedules.md) | | ✓ | ✓ | ✓ | ✓ | -| [Incident Management](../operations/incident_management/index.md):<br>Participate in on-call rotation | ✓| ✓ | ✓ | ✓ | ✓ | -| [Incident Management](../operations/incident_management/index.md):<br>View [escalation policies](../operations/incident_management/escalation_policies.md) | | ✓ | ✓ | ✓ | ✓ | -| [Incident Management](../operations/incident_management/index.md):<br>Manage [on-call schedules](../operations/incident_management/oncall_schedules.md) | | | | ✓ | ✓ | -| [Incident Management](../operations/incident_management/index.md):<br>Manage [escalation policies](../operations/incident_management/escalation_policies.md) | | | | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>Add Labels | ✓ (*16*) | ✓ | ✓ | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>Assign | ✓ (*16*) | ✓ | ✓ | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>Create | ✓ | ✓ | ✓ | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>Create [confidential issues](project/issues/confidential_issues.md) | ✓ | ✓ | ✓ | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>View [Design Management](project/issues/design_management.md) pages | ✓ | ✓ | ✓ | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>View related issues | ✓ | ✓ | ✓ | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>Set weight | ✓ (*16*) | ✓ | ✓ | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>View [confidential issues](project/issues/confidential_issues.md) | (*2*) | ✓ | ✓ | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>Close / reopen | | ✓ | ✓ | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>Lock threads | | ✓ | ✓ | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>Manage related issues | | ✓ | ✓ | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>Manage tracker | | ✓ | ✓ | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>Move issues (*15*) | | ✓ | ✓ | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>Set issue [time tracking](project/time_tracking.md) estimate and time spent | | ✓ | ✓ | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>Upload [Design Management](project/issues/design_management.md) files | | | ✓ | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>Delete | | | | | ✓ | -| [License Compliance](compliance/license_compliance/index.md):<br>View allowed and denied licenses **(ULTIMATE)** | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | -| [License Compliance](compliance/license_compliance/index.md):<br>View License Compliance reports **(ULTIMATE)** | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | -| [License Compliance](compliance/license_compliance/index.md):<br>View License list **(ULTIMATE)** | | ✓ | ✓ | ✓ | ✓ | -| [License Compliance](compliance/license_compliance/index.md):<br>Manage license policy **(ULTIMATE)** | | | | ✓ | ✓ | -| [Merge requests](project/merge_requests/index.md):<br>Assign reviewer | | ✓ | ✓ | ✓ | ✓ | -| [Merge requests](project/merge_requests/index.md):<br>See list | | ✓ | ✓ | ✓ | ✓ | -| [Merge requests](project/merge_requests/index.md):<br>Apply code change suggestions | | | ✓ | ✓ | ✓ | -| [Merge requests](project/merge_requests/index.md):<br>Approve (*9*) | | | ✓ | ✓ | ✓ | -| [Merge requests](project/merge_requests/index.md):<br>Assign | | | ✓ | ✓ | ✓ | -| [Merge requests](project/merge_requests/index.md):<br>Create (*18*) | | | ✓ | ✓ | ✓ | -| [Merge requests](project/merge_requests/index.md):<br>Add labels | | | ✓ | ✓ | ✓ | -| [Merge requests](project/merge_requests/index.md):<br>Lock threads | | | ✓ | ✓ | ✓ | -| [Merge requests](project/merge_requests/index.md):<br>Manage or accept | | | ✓ | ✓ | ✓ | -| [Merge requests](project/merge_requests/index.md):<br>Manage merge approval rules (project settings) | | | | ✓ | ✓ | -| [Merge requests](project/merge_requests/index.md):<br>Delete | | | | | ✓ | -| [Metrics dashboards](../operations/metrics/dashboards/index.md):<br>Manage user-starred metrics dashboards (*7*) | ✓ | ✓ | ✓ | ✓ | ✓ | -| [Metrics dashboards](../operations/metrics/dashboards/index.md):<br>View metrics dashboard annotations | | ✓ | ✓ | ✓ | ✓ | -| [Metrics dashboards](../operations/metrics/dashboards/index.md):<br>Create/edit/delete metrics dashboard annotations | | | ✓ | ✓ | ✓ | -| [Package registry](packages/index.md):<br>Pull package | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | -| [Package registry](packages/index.md):<br>Publish package | | | ✓ | ✓ | ✓ | -| [Package registry](packages/index.md):<br>Delete package | | | | ✓ | ✓ | -| [Project operations](../operations/index.md):<br>View [Error Tracking](../operations/error_tracking.md) list | | ✓ | ✓ | ✓ | ✓ | -| [Project operations](../operations/index.md):<br>Manage [Feature Flags](../operations/feature_flags.md) **(PREMIUM)** | | | ✓ | ✓ | ✓ | -| [Project operations](../operations/index.md):<br>Manage [Error Tracking](../operations/error_tracking.md) | | | | ✓ | ✓ | -| [Projects](project/index.md):<br>Download project | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | -| [Projects](project/index.md):<br>Leave comments | ✓ | ✓ | ✓ | ✓ | ✓ | -| [Projects](project/index.md):<br>Reposition comments on images (posted by any user) | ✓ (*10*) | ✓ (*10*) | ✓ (*10*) | ✓ | ✓ | -| [Projects](project/index.md):<br>View Insights **(ULTIMATE)** | ✓ | ✓ | ✓ | ✓ | ✓ | -| [Projects](project/index.md):<br>View [releases](project/releases/index.md) | ✓ (*6*) | ✓ | ✓ | ✓ | ✓ | -| [Projects](project/index.md):<br>View Requirements **(ULTIMATE)** | ✓ | ✓ | ✓ | ✓ | ✓ | -| [Projects](project/index.md):<br>View [time tracking](project/time_tracking.md) reports | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | -| [Projects](project/index.md):<br>View [wiki](project/wiki/index.md) pages | ✓ | ✓ | ✓ | ✓ | ✓ | -| [Projects](project/index.md):<br>Create [snippets](snippets.md) | | ✓ | ✓ | ✓ | ✓ | -| [Projects](project/index.md):<br>Manage labels | | ✓ | ✓ | ✓ | ✓ | -| [Projects](project/index.md):<br>View [project traffic statistics](../api/project_statistics.md) | | ✓ | ✓ | ✓ | ✓ | -| [Projects](project/index.md):<br>Create, edit, delete [milestones](project/milestones/index.md). | | | ✓ | ✓ | ✓ | -| [Projects](project/index.md):<br>Create, edit, delete [releases](project/releases/index.md) | | | ✓ (*13*) | ✓ (*13*) | ✓ (*13*) | -| [Projects](project/index.md):<br>Create, edit [wiki](project/wiki/index.md) pages | | | ✓ | ✓ | ✓ | -| [Projects](project/index.md):<br>Enable Review Apps | | | ✓ | ✓ | ✓ | -| [Projects](project/index.md):<br>View project [Audit Events](../administration/audit_events.md) | | | ✓ (*11*) | ✓ | ✓ | -| [Projects](project/index.md):<br>Add deploy keys | | | | ✓ | ✓ | -| [Projects](project/index.md):<br>Add new team members | | | | ✓ | ✓ | -| [Projects](project/index.md):<br>Change [project features visibility](../public_access/public_access.md) level | | | | ✓ (14) | ✓ | -| [Projects](project/index.md):<br>Configure [webhooks](project/integrations/webhooks.md) | | | | ✓ | ✓ | -| [Projects](project/index.md):<br>Delete [wiki](project/wiki/index.md) pages | | | ✓ | ✓ | ✓ | -| [Projects](project/index.md):<br>Edit comments (posted by any user) | | | | ✓ | ✓ | -| [Projects](project/index.md):<br>Edit project badges | | | | ✓ | ✓ | -| [Projects](project/index.md):<br>Edit project settings | | | | ✓ | ✓ | -| [Projects](project/index.md):<br>Export project | | | | ✓ | ✓ | -| [Projects](project/index.md):<br>Manage [project access tokens](project/settings/project_access_tokens.md) **(FREE SELF)** **(PREMIUM SAAS)** (*12*) | | | | ✓ | ✓ | -| [Projects](project/index.md):<br>Manage [Project Operations](../operations/index.md) | | | | ✓ | ✓ | -| [Projects](project/index.md):<br>Share (invite) projects with groups | | | | ✓ (*8*) | ✓ (*8*) | -| [Projects](project/index.md):<br>View 2FA status of members | | | | ✓ | ✓ | -| [Projects](project/index.md):<br>Administer project compliance frameworks | | | | | ✓ | -| [Projects](project/index.md):<br>Archive project | | | | | ✓ | -| [Projects](project/index.md):<br>Change project visibility level | | | | | ✓ | -| [Projects](project/index.md):<br>Delete project | | | | | ✓ | -| [Projects](project/index.md):<br>Disable notification emails | | | | | ✓ | -| [Projects](project/index.md):<br>Rename project | | | | | ✓ | -| [Projects](project/index.md):<br>Transfer project to another namespace | | | | | ✓ | -| [Repository](project/repository/index.md):<br>Pull project code | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | -| [Repository](project/repository/index.md):<br>View project code | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | -| [Repository](project/repository/index.md):<br>View a commit status | | ✓ | ✓ | ✓ | ✓ | -| [Repository](project/repository/index.md):<br>Add tags | | | ✓ | ✓ | ✓ | -| [Repository](project/repository/index.md):<br>Create new branches | | | ✓ | ✓ | ✓ | -| [Repository](project/repository/index.md):<br>Create or update commit status | | | ✓ (*5*) | ✓ | ✓ | -| [Repository](project/repository/index.md):<br>Force push to non-protected branches | | | ✓ | ✓ | ✓ | -| [Repository](project/repository/index.md):<br>Push to non-protected branches | | | ✓ | ✓ | ✓ | -| [Repository](project/repository/index.md):<br>Remove non-protected branches | | | ✓ | ✓ | ✓ | -| [Repository](project/repository/index.md):<br>Rewrite or remove Git tags | | | ✓ | ✓ | ✓ | -| [Repository](project/repository/index.md):<br>Enable or disable branch protection | | | | ✓ | ✓ | -| [Repository](project/repository/index.md):<br>Enable or disable tag protection | | | | ✓ | ✓ | -| [Repository](project/repository/index.md):<br>Manage [push rules](../push_rules/push_rules.md) | | | | ✓ | ✓ | -| [Repository](project/repository/index.md):<br>Push to protected branches (*5*) | | | | ✓ | ✓ | -| [Repository](project/repository/index.md):<br>Turn on or off protected branch push for developers | | | | ✓ | ✓ | -| [Repository](project/repository/index.md):<br>Remove fork relationship | | | | | ✓ | -| [Repository](project/repository/index.md):<br>Force push to protected branches (*4*) | | | | | | -| [Repository](project/repository/index.md):<br>Remove protected branches (*4*) | | | | | | -| [Requirements Management](project/requirements/index.md):<br>Archive / reopen **(ULTIMATE)** | | ✓ | ✓ | ✓ | ✓ | -| [Requirements Management](project/requirements/index.md):<br>Create / edit **(ULTIMATE)** | | ✓ | ✓ | ✓ | ✓ | -| [Requirements Management](project/requirements/index.md):<br>Import / export **(ULTIMATE)** | | ✓ | ✓ | ✓ | ✓ | -| [Security dashboard](application_security/security_dashboard/index.md):<br>View Security reports **(ULTIMATE)** | ✓ (*3*) | ✓ | ✓ | ✓ | ✓ | -| [Security dashboard](application_security/security_dashboard/index.md):<br>Create issue from vulnerability finding **(ULTIMATE)** | | | ✓ | ✓ | ✓ | -| [Security dashboard](application_security/security_dashboard/index.md):<br>Create vulnerability from vulnerability finding **(ULTIMATE)** | | | ✓ | ✓ | ✓ | -| [Security dashboard](application_security/security_dashboard/index.md):<br>Dismiss vulnerability **(ULTIMATE)** | | | ✓ | ✓ | ✓ | -| [Security dashboard](application_security/security_dashboard/index.md):<br>Dismiss vulnerability finding **(ULTIMATE)** | | | ✓ | ✓ | ✓ | -| [Security dashboard](application_security/security_dashboard/index.md):<br>Resolve vulnerability **(ULTIMATE)** | | | ✓ | ✓ | ✓ | -| [Security dashboard](application_security/security_dashboard/index.md):<br>Revert vulnerability to detected state **(ULTIMATE)** | | | ✓ | ✓ | ✓ | -| [Security dashboard](application_security/security_dashboard/index.md):<br>Use security dashboard **(ULTIMATE)** | | | ✓ | ✓ | ✓ | -| [Security dashboard](application_security/security_dashboard/index.md):<br>View vulnerability **(ULTIMATE)** | | | ✓ | ✓ | ✓ | -| [Security dashboard](application_security/security_dashboard/index.md):<br>View vulnerability findings in [dependency list](application_security/dependency_list/index.md) **(ULTIMATE)** | | | ✓ | ✓ | ✓ | -| [Terraform](infrastructure/index.md):<br>Read Terraform state | | | ✓ | ✓ | ✓ | -| [Terraform](infrastructure/index.md):<br>Manage Terraform state | | | | ✓ | ✓ | -| [Test cases](../ci/test_cases/index.md):<br>Archive | | ✓ | ✓ | ✓ | ✓ | -| [Test cases](../ci/test_cases/index.md):<br>Create | | ✓ | ✓ | ✓ | ✓ | -| [Test cases](../ci/test_cases/index.md):<br>Move | | ✓ | ✓ | ✓ | ✓ | -| [Test cases](../ci/test_cases/index.md):<br>Reopen | | ✓ | ✓ | ✓ | ✓ | +| Action | Guest | Reporter | Developer | Maintainer | Owner | +|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|----------|-----------|------------|-------| +| [Analytics](analytics/index.md):<br>View issue analytics **(PREMIUM)** | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Analytics](analytics/index.md):<br>View [merge request analytics](analytics/merge_request_analytics.md) **(PREMIUM)** | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Analytics](analytics/index.md):<br>View value stream analytics | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Analytics](analytics/index.md):<br>View [DORA metrics](analytics/ci_cd_analytics.md) | | ✓ | ✓ | ✓ | ✓ | +| [Analytics](analytics/index.md):<br>View [CI/CD analytics](analytics/ci_cd_analytics.md) | | ✓ | ✓ | ✓ | ✓ | +| [Analytics](analytics/index.md):<br>View [code review analytics](analytics/code_review_analytics.md) **(PREMIUM)** | | ✓ | ✓ | ✓ | ✓ | +| [Analytics](analytics/index.md):<br>View [repository analytics](analytics/repository_analytics.md) | | ✓ | ✓ | ✓ | ✓ | +| [Application security](application_security/index.md):<br>View licenses in [dependency list](application_security/dependency_list/index.md) **(ULTIMATE)** | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | +| [Application security](application_security/index.md):<br>Create and run [on-demand DAST scans](application_security/dast/index.md#on-demand-scans) **(ULTIMATE)** | | | ✓ | ✓ | ✓ | +| [Application security](application_security/index.md):<br>Manage [security policy](application_security/policies/index.md) **(ULTIMATE)** | | | ✓ | ✓ | ✓ | +| [Application security](application_security/index.md):<br>View [dependency list](application_security/dependency_list/index.md) **(ULTIMATE)** | | | ✓ | ✓ | ✓ | +| [Application security](application_security/index.md):<br>View [threats list](application_security/threat_monitoring/index.md#threat-monitoring) **(ULTIMATE)** | | | ✓ | ✓ | ✓ | +| [Application security](application_security/index.md):<br>Create a [CVE ID Request](application_security/cve_id_request.md) **(FREE SAAS)** | | | | ✓ | ✓ | +| [Application security](application_security/index.md):<br>Create or assign [security policy project](application_security/policies/index.md) **(ULTIMATE)** | | | | | ✓ | +| [Clusters](infrastructure/clusters/index.md):<br>View [pod logs](project/clusters/kubernetes_pod_logs.md) | | | ✓ | ✓ | ✓ | +| [Clusters](infrastructure/clusters/index.md):<br>View clusters | | | ✓ | ✓ | ✓ | +| [Clusters](infrastructure/clusters/index.md):<br>Manage clusters | | | | ✓ | ✓ | +| [Container Registry](packages/container_registry/index.md):<br>Create, edit, delete cleanup policies | | | ✓ | ✓ | ✓ | +| [Container Registry](packages/container_registry/index.md):<br>Push an image to the Container Registry | | | ✓ | ✓ | ✓ | +| [Container Registry](packages/container_registry/index.md):<br>Pull an image from the Container Registry | ✓ (*20*) | ✓ (*20*) | ✓ | ✓ | ✓ | +| [Container Registry](packages/container_registry/index.md):<br>Remove a Container Registry image | | | ✓ | ✓ | ✓ | +| [GitLab Pages](project/pages/index.md):<br>View Pages protected by [access control](project/pages/introduction.md#gitlab-pages-access-control) | ✓ | ✓ | ✓ | ✓ | ✓ | +| [GitLab Pages](project/pages/index.md):<br>Manage | | | | ✓ | ✓ | +| [GitLab Pages](project/pages/index.md):<br>Manage GitLab Pages domains and certificates | | | | ✓ | ✓ | +| [GitLab Pages](project/pages/index.md):<br>Remove GitLab Pages | | | | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>View [alerts](../operations/incident_management/alerts.md) | | ✓ | ✓ | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>Assign an alert | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>View [incident](../operations/incident_management/incidents.md) | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>Create [incident](../operations/incident_management/incidents.md) | (*16*) | ✓ | ✓ | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>View [on-call schedules](../operations/incident_management/oncall_schedules.md) | | ✓ | ✓ | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>Participate in on-call rotation | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>View [escalation policies](../operations/incident_management/escalation_policies.md) | | ✓ | ✓ | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>Manage [on-call schedules](../operations/incident_management/oncall_schedules.md) | | | | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>Manage [escalation policies](../operations/incident_management/escalation_policies.md) | | | | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>Add Labels | ✓ (*15*) | ✓ | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>Assign | ✓ (*15*) | ✓ | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>Create (*18*) | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>Create [confidential issues](project/issues/confidential_issues.md) | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>View [Design Management](project/issues/design_management.md) pages | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>View related issues | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>Set weight | ✓ (*15*) | ✓ | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>View [confidential issues](project/issues/confidential_issues.md) | (*2*) | ✓ | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>Close / reopen (*19*) | | ✓ | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>Lock threads | | ✓ | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>Manage related issues | | ✓ | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>Manage tracker | | ✓ | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>Move issues (*14*) | | ✓ | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>Set issue [time tracking](project/time_tracking.md) estimate and time spent | | ✓ | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>Upload [Design Management](project/issues/design_management.md) files | | | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>Delete | | | | | ✓ | +| [License Compliance](compliance/license_compliance/index.md):<br>View allowed and denied licenses **(ULTIMATE)** | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | +| [License Compliance](compliance/license_compliance/index.md):<br>View License Compliance reports **(ULTIMATE)** | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | +| [License Compliance](compliance/license_compliance/index.md):<br>View License list **(ULTIMATE)** | | ✓ | ✓ | ✓ | ✓ | +| [License Compliance](compliance/license_compliance/index.md):<br>Manage license policy **(ULTIMATE)** | | | | ✓ | ✓ | +| [Merge requests](project/merge_requests/index.md):<br>Assign reviewer | | ✓ | ✓ | ✓ | ✓ | +| [Merge requests](project/merge_requests/index.md):<br>See list | | ✓ | ✓ | ✓ | ✓ | +| [Merge requests](project/merge_requests/index.md):<br>Apply code change suggestions | | | ✓ | ✓ | ✓ | +| [Merge requests](project/merge_requests/index.md):<br>Approve (*8*) | | | ✓ | ✓ | ✓ | +| [Merge requests](project/merge_requests/index.md):<br>Assign | | | ✓ | ✓ | ✓ | +| [Merge requests](project/merge_requests/index.md):<br>Create (*17*) | | | ✓ | ✓ | ✓ | +| [Merge requests](project/merge_requests/index.md):<br>Add labels | | | ✓ | ✓ | ✓ | +| [Merge requests](project/merge_requests/index.md):<br>Lock threads | | | ✓ | ✓ | ✓ | +| [Merge requests](project/merge_requests/index.md):<br>Manage or accept | | | ✓ | ✓ | ✓ | +| [Merge requests](project/merge_requests/index.md):<br>Manage merge approval rules (project settings) | | | | ✓ | ✓ | +| [Merge requests](project/merge_requests/index.md):<br>Delete | | | | | ✓ | +| [Metrics dashboards](../operations/metrics/dashboards/index.md):<br>Manage user-starred metrics dashboards (*6*) | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Metrics dashboards](../operations/metrics/dashboards/index.md):<br>View metrics dashboard annotations | | ✓ | ✓ | ✓ | ✓ | +| [Metrics dashboards](../operations/metrics/dashboards/index.md):<br>Create/edit/delete metrics dashboard annotations | | | ✓ | ✓ | ✓ | +| [Package registry](packages/index.md):<br>Pull a package | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | +| [Package registry](packages/index.md):<br>Publish a package | | | ✓ | ✓ | ✓ | +| [Package registry](packages/index.md):<br>Delete a package | | | | ✓ | ✓ | +| [Package registry](packages/index.md):<br>Delete a file associated with a package | | | | ✓ | ✓ | +| [Project operations](../operations/index.md):<br>View [Error Tracking](../operations/error_tracking.md) list | | ✓ | ✓ | ✓ | ✓ | +| [Project operations](../operations/index.md):<br>Manage [Feature Flags](../operations/feature_flags.md) **(PREMIUM)** | | | ✓ | ✓ | ✓ | +| [Project operations](../operations/index.md):<br>Manage [Error Tracking](../operations/error_tracking.md) | | | | ✓ | ✓ | +| [Projects](project/index.md):<br>Download project | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | +| [Projects](project/index.md):<br>Leave comments | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Projects](project/index.md):<br>Reposition comments on images (posted by any user) | ✓ (*9*) | ✓ (*9*) | ✓ (*9*) | ✓ | ✓ | +| [Projects](project/index.md):<br>View Insights **(ULTIMATE)** | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Projects](project/index.md):<br>View [releases](project/releases/index.md) | ✓ (*5*) | ✓ | ✓ | ✓ | ✓ | +| [Projects](project/index.md):<br>View Requirements **(ULTIMATE)** | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Projects](project/index.md):<br>View [time tracking](project/time_tracking.md) reports | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | +| [Projects](project/index.md):<br>View [wiki](project/wiki/index.md) pages | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Projects](project/index.md):<br>Create [snippets](snippets.md) | | ✓ | ✓ | ✓ | ✓ | +| [Projects](project/index.md):<br>Manage labels | | ✓ | ✓ | ✓ | ✓ | +| [Projects](project/index.md):<br>View [project traffic statistics](../api/project_statistics.md) | | ✓ | ✓ | ✓ | ✓ | +| [Projects](project/index.md):<br>Create, edit, delete [milestones](project/milestones/index.md). | | | ✓ | ✓ | ✓ | +| [Projects](project/index.md):<br>Create, edit, delete [releases](project/releases/index.md) | | | ✓ (*12*) | ✓ (*12*) | ✓ (*12*) | +| [Projects](project/index.md):<br>Create, edit [wiki](project/wiki/index.md) pages | | | ✓ | ✓ | ✓ | +| [Projects](project/index.md):<br>Enable Review Apps | | | ✓ | ✓ | ✓ | +| [Projects](project/index.md):<br>View project [Audit Events](../administration/audit_events.md) | | | ✓ (*10*) | ✓ | ✓ | +| [Projects](project/index.md):<br>Add deploy keys | | | | ✓ | ✓ | +| [Projects](project/index.md):<br>Add new team members | | | | ✓ | ✓ | +| [Projects](project/index.md):<br>Change [project features visibility](../public_access/public_access.md) level | | | | ✓ (*13*) | ✓ | +| [Projects](project/index.md):<br>Configure [webhooks](project/integrations/webhooks.md) | | | | ✓ | ✓ | +| [Projects](project/index.md):<br>Delete [wiki](project/wiki/index.md) pages | | | ✓ | ✓ | ✓ | +| [Projects](project/index.md):<br>Edit comments (posted by any user) | | | | ✓ | ✓ | +| [Projects](project/index.md):<br>Edit project badges | | | | ✓ | ✓ | +| [Projects](project/index.md):<br>Edit project settings | | | | ✓ | ✓ | +| [Projects](project/index.md):<br>Export project | | | | ✓ | ✓ | +| [Projects](project/index.md):<br>Manage [project access tokens](project/settings/project_access_tokens.md) **(FREE SELF)** **(PREMIUM SAAS)** (*11*) | | | | ✓ | ✓ | +| [Projects](project/index.md):<br>Manage [Project Operations](../operations/index.md) | | | | ✓ | ✓ | +| [Projects](project/index.md):<br>Rename project | | | | ✓ | ✓ | +| [Projects](project/index.md):<br>Share (invite) projects with groups | | | | ✓ (*7*) | ✓ (*7*) | +| [Projects](project/index.md):<br>View 2FA status of members | | | | ✓ | ✓ | +| [Projects](project/index.md):<br>Administer project compliance frameworks | | | | | ✓ | +| [Projects](project/index.md):<br>Archive project | | | | | ✓ | +| [Projects](project/index.md):<br>Change project visibility level | | | | | ✓ | +| [Projects](project/index.md):<br>Delete project | | | | | ✓ | +| [Projects](project/index.md):<br>Disable notification emails | | | | | ✓ | +| [Projects](project/index.md):<br>Transfer project to another namespace | | | | | ✓ | +| [Repository](project/repository/index.md):<br>Pull project code | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | +| [Repository](project/repository/index.md):<br>View project code | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | +| [Repository](project/repository/index.md):<br>View a commit status | | ✓ | ✓ | ✓ | ✓ | +| [Repository](project/repository/index.md):<br>Add tags | | | ✓ | ✓ | ✓ | +| [Repository](project/repository/index.md):<br>Create new branches | | | ✓ | ✓ | ✓ | +| [Repository](project/repository/index.md):<br>Create or update commit status | | | ✓ (*4*) | ✓ | ✓ | +| [Repository](project/repository/index.md):<br>Force push to non-protected branches | | | ✓ | ✓ | ✓ | +| [Repository](project/repository/index.md):<br>Push to non-protected branches | | | ✓ | ✓ | ✓ | +| [Repository](project/repository/index.md):<br>Remove non-protected branches | | | ✓ | ✓ | ✓ | +| [Repository](project/repository/index.md):<br>Rewrite or remove Git tags | | | ✓ | ✓ | ✓ | +| [Repository](project/repository/index.md):<br>Enable or disable branch protection | | | | ✓ | ✓ | +| [Repository](project/repository/index.md):<br>Enable or disable tag protection | | | | ✓ | ✓ | +| [Repository](project/repository/index.md):<br>Manage [push rules](project/repository/push_rules.md) | | | | ✓ | ✓ | +| [Repository](project/repository/index.md):<br>Push to protected branches (*4*) | | | | ✓ | ✓ | +| [Repository](project/repository/index.md):<br>Turn on or off protected branch push for developers | | | | ✓ | ✓ | +| [Repository](project/repository/index.md):<br>Remove fork relationship | | | | | ✓ | +| [Repository](project/repository/index.md):<br>Force push to protected branches (*3*) | | | | | | +| [Repository](project/repository/index.md):<br>Remove protected branches (*3*) | | | | | | +| [Requirements Management](project/requirements/index.md):<br>Archive / reopen **(ULTIMATE)** | | ✓ | ✓ | ✓ | ✓ | +| [Requirements Management](project/requirements/index.md):<br>Create / edit **(ULTIMATE)** | | ✓ | ✓ | ✓ | ✓ | +| [Requirements Management](project/requirements/index.md):<br>Import / export **(ULTIMATE)** | | ✓ | ✓ | ✓ | ✓ | +| [Security dashboard](application_security/security_dashboard/index.md):<br>Create issue from vulnerability finding **(ULTIMATE)** | | | ✓ | ✓ | ✓ | +| [Security dashboard](application_security/security_dashboard/index.md):<br>Create vulnerability from vulnerability finding **(ULTIMATE)** | | | ✓ | ✓ | ✓ | +| [Security dashboard](application_security/security_dashboard/index.md):<br>Dismiss vulnerability **(ULTIMATE)** | | | ✓ | ✓ | ✓ | +| [Security dashboard](application_security/security_dashboard/index.md):<br>Dismiss vulnerability finding **(ULTIMATE)** | | | ✓ | ✓ | ✓ | +| [Security dashboard](application_security/security_dashboard/index.md):<br>Resolve vulnerability **(ULTIMATE)** | | | ✓ | ✓ | ✓ | +| [Security dashboard](application_security/security_dashboard/index.md):<br>Revert vulnerability to detected state **(ULTIMATE)** | | | ✓ | ✓ | ✓ | +| [Security dashboard](application_security/security_dashboard/index.md):<br>Use security dashboard **(ULTIMATE)** | | | ✓ | ✓ | ✓ | +| [Security dashboard](application_security/security_dashboard/index.md):<br>View vulnerability **(ULTIMATE)** | | | ✓ | ✓ | ✓ | +| [Security dashboard](application_security/security_dashboard/index.md):<br>View vulnerability findings in [dependency list](application_security/dependency_list/index.md) **(ULTIMATE)** | | | ✓ | ✓ | ✓ | +| [Terraform](infrastructure/index.md):<br>Read Terraform state | | | ✓ | ✓ | ✓ | +| [Terraform](infrastructure/index.md):<br>Manage Terraform state | | | | ✓ | ✓ | +| [Test cases](../ci/test_cases/index.md):<br>Archive | | ✓ | ✓ | ✓ | ✓ | +| [Test cases](../ci/test_cases/index.md):<br>Create | | ✓ | ✓ | ✓ | ✓ | +| [Test cases](../ci/test_cases/index.md):<br>Move | | ✓ | ✓ | ✓ | ✓ | +| [Test cases](../ci/test_cases/index.md):<br>Reopen | | ✓ | ✓ | ✓ | ✓ | + +<!-- markdownlint-disable MD029 --> 1. On self-managed GitLab instances, guest users are able to perform this action only on public and internal projects (not on private projects). [External users](#external-users) must be given explicit access even if the project is internal. For GitLab.com, see the [GitLab.com visibility settings](gitlab_com/index.md#visibility-settings). -1. Guest users can only view the [confidential issues](project/issues/confidential_issues.md) they created themselves. -1. If **Public pipelines** is enabled in **Project Settings > CI/CD**. -1. Not allowed for Guest, Reporter, Developer, Maintainer, or Owner. See [protected branches](project/protected_branches.md). -1. If the [branch is protected](project/protected_branches.md), this depends on the access Developers and Maintainers are given. -1. Guest users can access GitLab [**Releases**](project/releases/index.md) for downloading assets but are not allowed to download the source code nor see [repository information like commits and release evidence](project/releases/index.md#view-a-release-and-download-assets). -1. Actions are limited only to records owned (referenced) by user. -1. When [Share Group Lock](group/index.md#prevent-a-project-from-being-shared-with-groups) is enabled the project can't be shared with other groups. It does not affect group with group sharing. -1. For information on eligible approvers for merge requests, see +2. Guest users can only view the [confidential issues](project/issues/confidential_issues.md) they created themselves. +3. Not allowed for Guest, Reporter, Developer, Maintainer, or Owner. See [protected branches](project/protected_branches.md). +4. If the [branch is protected](project/protected_branches.md), this depends on the access Developers and Maintainers are given. +5. Guest users can access GitLab [**Releases**](project/releases/index.md) for downloading assets but are not allowed to download the source code nor see [repository information like commits and release evidence](project/releases/index.md#view-a-release-and-download-assets). +6. Actions are limited only to records owned (referenced) by user. +7. When [Share Group Lock](group/index.md#prevent-a-project-from-being-shared-with-groups) is enabled the project can't be shared with other groups. It does not affect group with group sharing. +8. For information on eligible approvers for merge requests, see [Eligible approvers](project/merge_requests/approvals/rules.md#eligible-approvers). -1. Applies only to comments on [Design Management](project/issues/design_management.md) designs. -1. Users can only view events based on their individual actions. -1. Project access tokens are supported for self-managed instances on Free and above. They are also - supported on GitLab SaaS Premium and above (excluding [trial licenses](https://about.gitlab.com/free-trial/)). -1. If the [tag is protected](#release-permissions-with-protected-tags), this depends on the access Developers and Maintainers are given. -1. A Maintainer can't change project features visibility level if - [project visibility](../public_access/public_access.md) is set to private. -1. Attached design files are moved together with the issue even if the user doesn't have the - Developer role. -1. Guest users can only set metadata (for example, labels, assignees, or milestones) - when creating an issue. They cannot change the metadata on existing issues. -1. In GitLab 14.5 or later, Guests are not allowed to [create incidents](../operations/incident_management/incidents.md#incident-creation). -1. In projects that accept contributions from external members, users can create, edit, and close their own merge requests. +9. Applies only to comments on [Design Management](project/issues/design_management.md) designs. +10. Users can only view events based on their individual actions. +11. Project access tokens are supported for self-managed instances on Free and above. They are also + supported on GitLab SaaS Premium and above (excluding [trial licenses](https://about.gitlab.com/free-trial/)). +12. If the [tag is protected](#release-permissions-with-protected-tags), this depends on the access Developers and Maintainers are given. +13. A Maintainer can't change project features visibility level if + [project visibility](../public_access/public_access.md) is set to private. +14. Attached design files are moved together with the issue even if the user doesn't have the + Developer role. +15. Guest users can only set metadata (for example, labels, assignees, or milestones) + when creating an issue. They cannot change the metadata on existing issues. +16. In GitLab 14.5 or later, Guests are not allowed to [create incidents](../operations/incident_management/incidents.md#incident-creation). + A guest who created an incident when they had the Reporter role or who is assigned to the incident can modify the title, description and metrics. They can also close and reopen the incident. +17. In projects that accept contributions from external members, users can create, edit, and close their own merge requests. +18. Authors and assignees of issues can modify the title and description even if they don't have the Reporter role. +19. Authors and assignees can close and reopen issues even if they don't have the Reporter role. +20. The ability to view the Container Registry and pull images is controlled by the [Container Registry's visibility permissions](packages/container_registry/index.md#container-registry-visibility-permissions). + +<!-- markdownlint-enable MD029 --> ## Project features permissions +More details about the permissions for some project-level features follow. + +### GitLab CI/CD permissions + +[GitLab CI/CD](../ci/index.md) permissions for some roles can be modified by these settings: + +- [Public pipelines](../ci/pipelines/settings.md#change-which-users-can-view-your-pipelines): + When set to public, gives access to certain CI/CD features to *Guest* project members. +- [Pipeline visibility](../ci/enable_or_disable_ci.md#enable-cicd-in-a-project): When set to **Everyone with Access**, + gives access to certain CI/CD "view" features to *non-project* members. + +| Action | Non-member | Guest | Reporter | Developer | Maintainer | Owner | +|-----------------------------------------------------------------------------------|------------|---------|----------|-----------|------------|-------| +| See that artifacts exist | ✓ (*3*) | ✓ (*3*) | ✓ | ✓ | ✓ | ✓ | +| View a list of jobs | ✓ (*1*) | ✓ (*2*) | ✓ | ✓ | ✓ | ✓ | +| View and download artifacts | ✓ (*1*) | ✓ (*2*) | ✓ | ✓ | ✓ | ✓ | +| View [environments](../ci/environments/index.md) | ✓ (*3*) | ✓ (*3*) | ✓ | ✓ | ✓ | ✓ | +| View job logs and job details page | ✓ (*1*) | ✓ (*2*) | ✓ | ✓ | ✓ | ✓ | +| View pipeline details page | ✓ (*1*) | ✓ (*2*) | ✓ | ✓ | ✓ | ✓ | +| View pipelines page | ✓ (*1*) | ✓ (*2*) | ✓ | ✓ | ✓ | ✓ | +| View pipelines tab in MR | ✓ (*3*) | ✓ (*3*) | ✓ | ✓ | ✓ | ✓ | +| [View vulnerabilities in a pipeline](application_security/security_dashboard/index.md#view-vulnerabilities-in-a-pipeline) | | ✓ (*2*) | ✓ | ✓ | ✓ | ✓ | +| Cancel and retry jobs | | | | ✓ | ✓ | ✓ | +| Create new [environments](../ci/environments/index.md) | | | | ✓ | ✓ | ✓ | +| Delete job logs or job artifacts | | | | ✓ (*4*) | ✓ | ✓ | +| Run CI/CD pipeline for a protected branch | | | | ✓ (*5*) | ✓ (*5*) | ✓ | +| Stop [environments](../ci/environments/index.md) | | | | ✓ | ✓ | ✓ | +| View a job with [debug logging](../ci/variables/index.md#debug-logging) | | | | ✓ | ✓ | ✓ | +| Use pipeline editor | | | | ✓ | ✓ | ✓ | +| Add specific runners to project | | | | | ✓ | ✓ | +| Clear runner caches manually | | | | | ✓ | ✓ | +| Enable shared runners in project | | | | | ✓ | ✓ | +| Manage CI/CD settings | | | | | ✓ | ✓ | +| Manage job triggers | | | | | ✓ | ✓ | +| Manage project-level CI/CD variables | | | | | ✓ | ✓ | +| Run [interactive web terminals](../ci/interactive_web_terminal/index.md) | | | | | ✓ | ✓ | +| Use [environment terminals](../ci/environments/index.md#web-terminals-deprecated) | | | | | ✓ | ✓ | +| Delete pipelines | | | | | | ✓ | + +<!-- markdownlint-disable MD029 --> + +1. If the project is public and **Public pipelines** is enabled in **Project Settings > CI/CD**. +2. If **Public pipelines** is enabled in **Project Settings > CI/CD**. +3. If the project is public. +4. Only if the job was both: + - Triggered by the user. + - [In GitLab 13.0](https://gitlab.com/gitlab-org/gitlab/-/issues/35069) and later, + run for a non-protected branch. +5. If the user is [allowed to merge or push to the protected branch](../ci/pipelines/index.md#pipeline-security-on-protected-branches). + +<!-- markdownlint-enable MD029 --> + +#### Job permissions + +This table shows granted privileges for jobs triggered by specific types of users: + +| Action | Guest, Reporter | Developer | Maintainer| Administrator | +|---------------------------------------------|-----------------|-----------|-----------|---------------| +| Run CI job | | ✓ | ✓ | ✓ | +| Clone source and LFS from current project | | ✓ | ✓ | ✓ | +| Clone source and LFS from public projects | | ✓ | ✓ | ✓ | +| Clone source and LFS from internal projects | | ✓ (*1*) | ✓ (*1*) | ✓ | +| Clone source and LFS from private projects | | ✓ (*2*) | ✓ (*2*) | ✓ (*2*) | +| Pull container images from current project | | ✓ | ✓ | ✓ | +| Pull container images from public projects | | ✓ | ✓ | ✓ | +| Pull container images from internal projects| | ✓ (*1*) | ✓ (*1*) | ✓ | +| Pull container images from private projects | | ✓ (*2*) | ✓ (*2*) | ✓ (*2*) | +| Push container images to current project | | ✓ | ✓ | ✓ | +| Push container images to other projects | | | | | +| Push source and LFS | | | | | + +1. Only if the triggering user is not an external one. +1. Only if the triggering user is a member of the project. + ### Wiki and issues Project features like [wikis](project/wiki/index.md) and issues can be hidden from users depending on @@ -257,9 +324,9 @@ Maintainers and Developers from pushing to a protected branch. Read through the [protected branches](project/protected_branches.md) to learn more. -### Value Stream Analytics permissions +### Value stream analytics permissions -Find the current permissions on the Value Stream Analytics dashboard, as described in +Find the current permissions on the value stream analytics dashboard, as described in [related documentation](analytics/value_stream_analytics.md#permissions). ### Issue board permissions @@ -281,7 +348,8 @@ read through the documentation on [permissions and access to confidential issues ### Container Registry visibility permissions -Find the visibility permissions for the Container Registry, as described in the +The ability to view the Container Registry and pull images is controlled by the Container Registry's +visibility permissions. Find these permissions for the Container Registry as described in the [related documentation](packages/container_registry/index.md#container-registry-visibility-permissions). ## Group members permissions @@ -293,67 +361,74 @@ The following table lists group permissions available for each role: <!-- Keep this table sorted: first, by minimum role, then alphabetically. --> -| Action | Guest | Reporter | Developer | Maintainer | Owner | -|--------------------------------------------------------|-------|----------|-----------|------------|-------| -| Browse group | ✓ | ✓ | ✓ | ✓ | ✓ | -| Edit SAML SSO Billing **(PREMIUM SAAS)** | ✓ | ✓ | ✓ | ✓ | ✓ (4) | -| Pull a container image using the dependency proxy | ✓ | ✓ | ✓ | ✓ | ✓ | -| View Contribution analytics | ✓ | ✓ | ✓ | ✓ | ✓ | -| View group epic **(PREMIUM)** | ✓ | ✓ | ✓ | ✓ | ✓ | -| View group wiki pages **(PREMIUM)** | ✓ (6) | ✓ | ✓ | ✓ | ✓ | -| View Insights **(ULTIMATE)** | ✓ | ✓ | ✓ | ✓ | ✓ | -| View Insights charts **(ULTIMATE)** | ✓ | ✓ | ✓ | ✓ | ✓ | -| View Issue analytics **(PREMIUM)** | ✓ | ✓ | ✓ | ✓ | ✓ | -| View Value Stream analytics | ✓ | ✓ | ✓ | ✓ | ✓ | -| Create/edit group epic **(PREMIUM)** | | ✓ | ✓ | ✓ | ✓ | -| Create/edit/delete epic boards **(PREMIUM)** | | ✓ | ✓ | ✓ | ✓ | -| Manage group labels | | ✓ | ✓ | ✓ | ✓ | -| Pull [packages](packages/index.md) | | ✓ | ✓ | ✓ | ✓ | -| View a container registry | | ✓ | ✓ | ✓ | ✓ | -| View Group DevOps Adoption **(ULTIMATE)** | | ✓ | ✓ | ✓ | ✓ | -| View metrics dashboard annotations | | ✓ | ✓ | ✓ | ✓ | -| View Productivity analytics **(PREMIUM)** | | ✓ | ✓ | ✓ | ✓ | -| Create and edit group wiki pages **(PREMIUM)** | | | ✓ | ✓ | ✓ | -| Create project in group | | | ✓ (3)(5) | ✓ (3) | ✓ (3) | -| Create/edit/delete group milestones | | | ✓ | ✓ | ✓ | -| Create/edit/delete iterations | | | ✓ | ✓ | ✓ | -| Create/edit/delete metrics dashboard annotations | | | ✓ | ✓ | ✓ | -| Enable/disable a dependency proxy | | | ✓ | ✓ | ✓ | -| Purge the dependency proxy for a group | | | | | ✓ | -| Publish [packages](packages/index.md) | | | ✓ | ✓ | ✓ | -| Use security dashboard **(ULTIMATE)** | | | ✓ | ✓ | ✓ | -| View group Audit Events | | | ✓ (7) | ✓ (7) | ✓ | -| Create subgroup | | | | ✓ (1) | ✓ | -| Delete group wiki pages **(PREMIUM)** | | | ✓ | ✓ | ✓ | -| Edit epic comments (posted by any user) **(ULTIMATE)** | | | | ✓ (2) | ✓ (2) | -| List group deploy tokens | | | | ✓ | ✓ | -| Manage [group push rules](group/index.md#group-push-rules) **(PREMIUM)** | | | | ✓ | ✓ | -| View/manage group-level Kubernetes cluster | | | | ✓ | ✓ | -| Administer project compliance frameworks | | | | | ✓ | -| Create/Delete group deploy tokens | | | | | ✓ | -| Change group visibility level | | | | | ✓ | -| Delete group | | | | | ✓ | -| Delete group epic **(PREMIUM)** | | | | | ✓ | -| Disable notification emails | | | | | ✓ | -| Edit group settings | | | | | ✓ | -| Filter members by 2FA status | | | | | ✓ | -| Manage group level CI/CD variables | | | | | ✓ | -| Manage group members | | | | | ✓ | -| Share (invite) groups with groups | | | | | ✓ | -| View 2FA status of members | | | | | ✓ | -| View Billing **(FREE SAAS)** | | | | | ✓ (4) | -| View Usage Quotas **(FREE SAAS)** | | | | | ✓ (4) | +| Action | Guest | Reporter | Developer | Maintainer | Owner | +|--------------------------------------------------------------------------|-------|----------|-----------|------------|-------| +| Browse group | ✓ | ✓ | ✓ | ✓ | ✓ | +| Pull a container image using the dependency proxy | ✓ | ✓ | ✓ | ✓ | ✓ | +| View Contribution analytics | ✓ | ✓ | ✓ | ✓ | ✓ | +| View group epic **(PREMIUM)** | ✓ | ✓ | ✓ | ✓ | ✓ | +| View group wiki pages **(PREMIUM)** | ✓ (6) | ✓ | ✓ | ✓ | ✓ | +| View Insights **(ULTIMATE)** | ✓ | ✓ | ✓ | ✓ | ✓ | +| View Insights charts **(ULTIMATE)** | ✓ | ✓ | ✓ | ✓ | ✓ | +| View Issue analytics **(PREMIUM)** | ✓ | ✓ | ✓ | ✓ | ✓ | +| View value stream analytics | ✓ | ✓ | ✓ | ✓ | ✓ | +| Create/edit group epic **(PREMIUM)** | | ✓ | ✓ | ✓ | ✓ | +| Create/edit/delete epic boards **(PREMIUM)** | | ✓ | ✓ | ✓ | ✓ | +| Manage group labels | | ✓ | ✓ | ✓ | ✓ | +| Publish [packages](packages/index.md) | | | ✓ | ✓ | ✓ | +| Pull [packages](packages/index.md) | | ✓ | ✓ | ✓ | ✓ | +| Delete [packages](packages/index.md | | | | ✓ | ✓ | +| Pull a Container Registry image | ✓ (7) | ✓ | ✓ | ✓ | ✓ | +| Remove a Container Registry image | | | ✓ | ✓ | ✓ | +| View Group DevOps Adoption **(ULTIMATE)** | | ✓ | ✓ | ✓ | ✓ | +| View metrics dashboard annotations | | ✓ | ✓ | ✓ | ✓ | +| View Productivity analytics **(PREMIUM)** | | ✓ | ✓ | ✓ | ✓ | +| Create and edit group wiki pages **(PREMIUM)** | | | ✓ | ✓ | ✓ | +| Create project in group | | | ✓ (3)(5) | ✓ (3) | ✓ (3) | +| Create/edit/delete group milestones | | | ✓ | ✓ | ✓ | +| Create/edit/delete iterations | | | ✓ | ✓ | ✓ | +| Create/edit/delete metrics dashboard annotations | | | ✓ | ✓ | ✓ | +| Enable/disable a dependency proxy | | | ✓ | ✓ | ✓ | +| Purge the dependency proxy for a group | | | | | ✓ | +| Use security dashboard **(ULTIMATE)** | | | ✓ | ✓ | ✓ | +| View group Audit Events | | | ✓ (7) | ✓ (7) | ✓ | +| Create subgroup | | | | ✓ (1) | ✓ | +| Delete group wiki pages **(PREMIUM)** | | | ✓ | ✓ | ✓ | +| Edit epic comments (posted by any user) **(ULTIMATE)** | | | | ✓ (2) | ✓ (2) | +| List group deploy tokens | | | | ✓ | ✓ | +| Manage [group push rules](group/index.md#group-push-rules) **(PREMIUM)** | | | | ✓ | ✓ | +| View/manage group-level Kubernetes cluster | | | | ✓ | ✓ | +| Administer project compliance frameworks | | | | | ✓ | +| Create/Delete group deploy tokens | | | | | ✓ | +| Change group visibility level | | | | | ✓ | +| Delete group | | | | | ✓ | +| Delete group epic **(PREMIUM)** | | | | | ✓ | +| Disable notification emails | | | | | ✓ | +| Edit group settings | | | | | ✓ | +| Edit SAML SSO **(PREMIUM SAAS)** | | | | | ✓ (4) | +| Filter members by 2FA status | | | | | ✓ | +| Manage group level CI/CD variables | | | | | ✓ | +| Manage group members | | | | | ✓ | +| Share (invite) groups with groups | | | | | ✓ | +| View 2FA status of members | | | | | ✓ | +| View Billing **(FREE SAAS)** | | | | | ✓ (4) | +| View Usage Quotas **(FREE SAAS)** | | | | | ✓ (4) | +| Manage runners | | | | | ✓ | + +<!-- markdownlint-disable MD029 --> 1. Groups can be set to [allow either Owners or Owners and Maintainers to create subgroups](group/subgroups/index.md#creating-a-subgroup) -1. Introduced in GitLab 12.2. -1. Default project creation role can be changed at: +2. Introduced in GitLab 12.2. +3. Default project creation role can be changed at: - The [instance level](admin_area/settings/visibility_and_access_controls.md#define-which-roles-can-create-projects). - The [group level](group/index.md#specify-who-can-add-projects-to-a-group). -1. Does not apply to subgroups. -1. Developers can push commits to the default branch of a new project only if the [default branch protection](group/index.md#change-the-default-branch-protection-of-a-group) is set to "Partially protected" or "Not protected". -1. In addition, if your group is public or internal, all users who can see the group can also see group wiki pages. -1. Users can only view events based on their individual actions. +4. Does not apply to subgroups. +5. Developers can push commits to the default branch of a new project only if the [default branch protection](group/index.md#change-the-default-branch-protection-of-a-group) is set to "Partially protected" or "Not protected". +6. In addition, if your group is public or internal, all users who can see the group can also see group wiki pages. +7. Users can only view events based on their individual actions. + +<!-- markdownlint-enable MD029 --> ### Subgroup permissions @@ -470,11 +545,16 @@ with the permissions described on the documentation on [auditor users permission > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/40942) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.4. -Owners can add members with a "minimal access" role to a parent group. Such users don't -automatically have access to projects and subgroups underneath. To support such access, owners must explicitly add these "minimal access" users to the specific subgroups/projects. +Owners can add members with a "minimal access" role to a parent group. Such users don't automatically have access to +projects and subgroups underneath. Owners must explicitly add these "minimal access" users to the specific subgroups and +projects. + +Because of an [outstanding issue](https://gitlab.com/gitlab-org/gitlab/-/issues/267996), when minimal access users: -Users with minimal access can list the group in the UI and through the API. However, they cannot see -details such as projects or subgroups. They do not have access to the group's page or list any of its subgroups or projects. +- Sign in with standard web authentication, they receive a `404` error when accessing the parent group. +- Sign in with Group SSO, they receive a `404` error immediately because they are redirected to the parent group page. + +To work around the issue, give these users the Guest role or higher to any project or subgroup within the parent group. ### Minimal access users take license seats @@ -492,66 +572,6 @@ which visibility level you select on project settings. - Everyone with access: everyone can see depending on your project visibility level. - Everyone: enabled for everyone (only available for GitLab Pages). -## GitLab CI/CD permissions - -GitLab CI/CD permissions rely on the role the user has in GitLab: - -- Maintainer -- Developer -- Guest/Reporter - -GitLab administrators can perform any action on GitLab CI/CD in scope of the GitLab -instance and project. - -| Action | Guest, Reporter | Developer |Maintainer| Administrator | -|---------------------------------------|-----------------|-------------|----------|---------------| -| See commits and jobs | ✓ | ✓ | ✓ | ✓ | -| Retry or cancel job | | ✓ | ✓ | ✓ | -| Erase job artifacts and job logs | | ✓ (*1*) | ✓ | ✓ | -| Delete project | | | ✓ | ✓ | -| Create project | | | ✓ | ✓ | -| Change project configuration | | | ✓ | ✓ | -| Add specific runners | | | ✓ | ✓ | -| Add shared runners | | | | ✓ | -| Clear runner caches manually | | | ✓ | ✓ | -| See events in the system | | | | ✓ | -| Admin Area | | | | ✓ | - -1. Only if the job was: - - Triggered by the user - - [In GitLab 13.0](https://gitlab.com/gitlab-org/gitlab/-/issues/35069) and later, run for a non-protected branch. - -### Job permissions - -This table shows granted privileges for jobs triggered by specific types of -users: - -| Action | Guest, Reporter | Developer |Maintainer| Administrator | -|---------------------------------------------|-----------------|-------------|----------|---------| -| Run CI job | | ✓ | ✓ | ✓ | -| Clone source and LFS from current project | | ✓ | ✓ | ✓ | -| Clone source and LFS from public projects | | ✓ | ✓ | ✓ | -| Clone source and LFS from internal projects | | ✓ (*1*) | ✓ (*1*) | ✓ | -| Clone source and LFS from private projects | | ✓ (*2*) | ✓ (*2*) | ✓ (*2*) | -| Pull container images from current project | | ✓ | ✓ | ✓ | -| Pull container images from public projects | | ✓ | ✓ | ✓ | -| Pull container images from internal projects| | ✓ (*1*) | ✓ (*1*) | ✓ | -| Pull container images from private projects | | ✓ (*2*) | ✓ (*2*) | ✓ (*2*) | -| Push container images to current project | | ✓ | ✓ | ✓ | -| Push container images to other projects | | | | | -| Push source and LFS | | | | | - -1. Only if the triggering user is not an external one -1. Only if the triggering user is a member of the project - -## Running pipelines on protected branches - -The permission to merge or push to protected branches is used to define if a user can -run CI/CD pipelines and execute actions on jobs that are related to those branches. - -See [Security on protected branches](../ci/pipelines/index.md#pipeline-security-on-protected-branches) -for details about the pipelines security model. - ## Release permissions with protected tags [The permission to create tags](project/protected_tags.md) is used to define if a user can diff --git a/doc/user/profile/account/create_accounts.md b/doc/user/profile/account/create_accounts.md index 32b8d2b33ee..7e1074aa50f 100644 --- a/doc/user/profile/account/create_accounts.md +++ b/doc/user/profile/account/create_accounts.md @@ -1,7 +1,7 @@ --- type: reference stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/profile/account/delete_account.md b/doc/user/profile/account/delete_account.md index 365f96b48b3..c116b1fc00d 100644 --- a/doc/user/profile/account/delete_account.md +++ b/doc/user/profile/account/delete_account.md @@ -1,7 +1,7 @@ --- type: howto stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -55,16 +55,17 @@ There are two options for deleting users: - **Delete user and contributions** When using the **Delete user** option, not all associated records are deleted with the user. -Here's a list of things that are **not** deleted: +Here's a list of things created by the user that are **not** deleted: -- Issues that the user created. -- Merge requests that the user created. -- Notes that the user created. -- Abuse reports that the user reported. -- Award emoji that the user created. +- Abuse reports +- Award emoji +- Epics +- Issues +- Merge requests +- Notes Instead of being deleted, these records are moved to a system-wide -user with the username "Ghost User", whose sole purpose is to act as a container +user with the username Ghost User, whose sole purpose is to act as a container for such records. Any commits made by a deleted user still display the username of the original user. @@ -81,14 +82,21 @@ is a sole owner of. Administrators can also request this behavior when deleting users from the [API](../../../api/users.md#user-deletion) or the Admin Area. -<!-- ## Troubleshooting +## Troubleshooting -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. +### Deleting a user results in a PostgreSQL null value error -Each scenario can be a third-level heading, e.g. `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +There is [a known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/349411) that results +in users not being deleted, and the following error generated: + +```plaintext +ERROR: null value in column "user_id" violates not-null constraint +``` + +The error can be found in the [PostgreSQL log](../../../administration/logs.md#postgresql-logs) and +in the **Retries** section of the [background jobs view](../../admin_area/index.md#background-jobs) in the Admin Area. + +If the user being deleted used the [iterations](../../group/iterations/index.md) feature, such +as adding an issue to an iteration, you must use +[the workaround documented in the issue](https://gitlab.com/gitlab-org/gitlab/-/issues/349411#workaround) +to delete the user. diff --git a/doc/user/profile/account/two_factor_authentication.md b/doc/user/profile/account/two_factor_authentication.md index 3af8c1c1b5a..a820cf150e9 100644 --- a/doc/user/profile/account/two_factor_authentication.md +++ b/doc/user/profile/account/two_factor_authentication.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/profile/img/personal_readme_setup_v14_5.png b/doc/user/profile/img/personal_readme_setup_v14_5.png Binary files differnew file mode 100644 index 00000000000..92d8e0ec936 --- /dev/null +++ b/doc/user/profile/img/personal_readme_setup_v14_5.png diff --git a/doc/user/profile/index.md b/doc/user/profile/index.md index 89e4ea6ea5b..f201e04183c 100644 --- a/doc/user/profile/index.md +++ b/doc/user/profile/index.md @@ -1,7 +1,7 @@ --- type: index, howto stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -104,16 +104,34 @@ user profiles are only visible to signed-in users. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/232157) in GitLab 14.5. -If you want to add more information to your profile page, you can create a README file. When you populate the README file with information, it's included on your profile page. +You can add more information to your profile page with a README file. When you populate +the README file with information, it's included on your profile page. -To add a README to your profile: +### From a new project -1. Create a new public project with the same project path as your GitLab username. +To create a new project and add its README to your profile: + +1. On the top bar, select **Menu > Project**. +1. Select **Create new project**. +1. Select **Create blank project**. +1. Enter the project details: + - In the **Project name** field, enter the name for your new project. + - In the **Project URL** field, select your GitLab username. + - In the **Project slug** field, enter your GitLab username. +1. For **Visibility Level**, select **Public**. +  +1. For **Project Configuration**, ensure **Initialize repository with a README** is selected. +1. Select **Create project**. 1. Create a README file inside this project. The file can be any valid [README or index file](../project/repository/index.md#readme-and-index-files). 1. Populate the README file with [Markdown](../markdown.md). -To use an existing project, [update the path](../project/settings/index.md#renaming-a-repository) of the project to match -your username. +GitLab displays the contents of your README below your contribution graph. + +### From an existing project + +To add the README from an existing project to your profile, +[update the path](../project/settings/index.md#renaming-a-repository) of the project +to match your username. ## Add external accounts to your user profile page diff --git a/doc/user/profile/personal_access_tokens.md b/doc/user/profile/personal_access_tokens.md index 45cff326332..10d718fdf1a 100644 --- a/doc/user/profile/personal_access_tokens.md +++ b/doc/user/profile/personal_access_tokens.md @@ -1,7 +1,7 @@ --- type: concepts, howto stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -97,7 +97,7 @@ A personal access token can perform actions based on the assigned scopes. | `read_user` | Read-only for endpoints under `/users`. Essentially, access to any of the `GET` requests in the [Users API](../../api/users.md). | | `read_api` | Read-only for the complete API, including all groups and projects, the Container Registry, and the Package Registry. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28944) in GitLab 12.10.) | | `read_repository` | Read-only (pull) for the repository through `git clone`. | -| `write_repository` | Read-write (pull, push) for the repository through `git clone`. Required for accessing Git repositories over HTTP when 2FA is enabled. | +| `write_repository` | Read-write (pull, push) for the repository through `git clone`. | | `read_registry` | Read-only (pull) for [Container Registry](../packages/container_registry/index.md) images if a project is private and authorization is required. | | `write_registry` | Read-write (push) for [Container Registry](../packages/container_registry/index.md) images if a project is private and authorization is required. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28958) in GitLab 12.10.) | | `sudo` | API actions as any user in the system (if the authenticated user is an administrator). | @@ -111,7 +111,7 @@ Personal access tokens expire on the date you define, at midnight UTC. - In GitLab Ultimate, administrators can [limit the lifetime of personal access tokens](../admin_area/settings/account_and_limit_settings.md#limit-the-lifetime-of-personal-access-tokens). - In GitLab Ultimate, administrators can choose whether or not to - [enforce personal access token expiration](../admin_area/settings/account_and_limit_settings.md#allow-expired-personal-access-tokens-to-be-used). + [enforce personal access token expiration](../admin_area/settings/account_and_limit_settings.md#allow-expired-personal-access-tokens-to-be-used-deprecated). ## Create a personal access token programmatically **(FREE SELF)** diff --git a/doc/user/profile/preferences.md b/doc/user/profile/preferences.md index 63be88f90d6..52baf5189e1 100644 --- a/doc/user/profile/preferences.md +++ b/doc/user/profile/preferences.md @@ -43,7 +43,7 @@ The default theme is Indigo. You can choose between 10 themes: GitLab has started work on dark mode! The dark mode Alpha release is available in the spirit of iteration and the lower expectations of -[Alpha versions](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha). +[Alpha versions](../../policy/alpha-beta-support.md#alpha-features). Progress on dark mode is tracked in the [Dark theme epic](https://gitlab.com/groups/gitlab-org/-/epics/2902). See the epic for: diff --git a/doc/user/profile/unknown_sign_in_notification.md b/doc/user/profile/unknown_sign_in_notification.md index 0ed2a11d363..37bde3ce846 100644 --- a/doc/user/profile/unknown_sign_in_notification.md +++ b/doc/user/profile/unknown_sign_in_notification.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/project/clusters/add_eks_clusters.md b/doc/user/project/clusters/add_eks_clusters.md index a5473629d4f..e14a71a7e10 100644 --- a/doc/user/project/clusters/add_eks_clusters.md +++ b/doc/user/project/clusters/add_eks_clusters.md @@ -34,7 +34,7 @@ Prerequisites: - An [Amazon Web Services](https://aws.amazon.com/) account. - Permissions to manage IAM resources. -For instance-level clusters, see [additional requirements for self-managed instances](#additional-requirements-for-self-managed-instances). **(FREE SELF)** +For instance-level clusters, see [additional requirements for self-managed instances](#additional-requirements-for-self-managed-instances). To create new Kubernetes clusters for your project, group, or instance through the certificate-based method: @@ -174,7 +174,7 @@ When you create a new cluster, you have the following settings: | Kubernetes cluster name | Your cluster's name. | | Environment scope | The [associated environment](multiple_kubernetes_clusters.md#setting-the-environment-scope). | | Service role | The **EKS IAM role** (**role A**). | -| Kubernetes version | The [Kubernetes version](../../infrastructure/clusters/connect/index.md#supported-cluster-versions) for your cluster. | +| Kubernetes version | The [Kubernetes version](../../clusters/agent/index.md#supported-cluster-versions) for your cluster. | | Key pair name | The [key pair](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html) that you can use to connect to your worker nodes. | | VPC | The [VPC](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html) to use for your EKS Cluster resources. | | Subnets | The [subnets](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Subnets.html) in your VPC where your worker nodes run. Two are required. | @@ -256,7 +256,7 @@ IAM user in the Amazon AWS console, and follow these steps: #### EKS access key and ID -> Instance profiles were [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/291015) in GitLab 13.7. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/291015) instance profiles in GitLab 13.7. If you're using GitLab 13.7 or later, you can use instance profiles to dynamically retrieve temporary credentials from AWS when needed. diff --git a/doc/user/project/clusters/add_existing_cluster.md b/doc/user/project/clusters/add_existing_cluster.md index acc5ed4cb30..0c3827bcbb1 100644 --- a/doc/user/project/clusters/add_existing_cluster.md +++ b/doc/user/project/clusters/add_existing_cluster.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Connect existing clusters through cluster certificates **(DEPRECATED)** +# Connect existing clusters through cluster certificates (DEPRECATED) **(FREE)** > [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. @@ -27,7 +27,7 @@ To add any cluster to GitLab, you need: - Either a GitLab.com account or an account for a self-managed installation running GitLab 12.5 or later. - The Maintainer role for group-level and project-level clusters. -- Access to the Admin area for instance-level clusters. **(FREE SELF)** +- Access to the Admin area for instance-level clusters. - A Kubernetes cluster. - Cluster administration access to the cluster with `kubectl`. diff --git a/doc/user/project/clusters/kubernetes_pod_logs.md b/doc/user/project/clusters/kubernetes_pod_logs.md index 19166a1ff8c..b5e2a1bad51 100644 --- a/doc/user/project/clusters/kubernetes_pod_logs.md +++ b/doc/user/project/clusters/kubernetes_pod_logs.md @@ -1,13 +1,13 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Kubernetes Logs (DEPRECATED) **(FREE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/4752) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.0. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26383) to [GitLab Free](https://about.gitlab.com/pricing/) 12.9. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/4752) in GitLab 11.0. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26383) from GitLab Ultimate to GitLab Free 12.9. > - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. WARNING: diff --git a/doc/user/project/clusters/multiple_kubernetes_clusters.md b/doc/user/project/clusters/multiple_kubernetes_clusters.md index 12527853b40..a56eaa9b953 100644 --- a/doc/user/project/clusters/multiple_kubernetes_clusters.md +++ b/doc/user/project/clusters/multiple_kubernetes_clusters.md @@ -6,8 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Multiple clusters per project with cluster certificates (DEPRECATED) **(FREE)** -> - Introduced in [GitLab Premium](https://about.gitlab.com/pricing/) 10.3 -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35094) to GitLab Free in 13.2. +> - Introduced in GitLab 10.3 +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35094) from GitLab Premium to GitLab Free in 13.2. > - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. WARNING: diff --git a/doc/user/project/clusters/protect/container_host_security/index.md b/doc/user/project/clusters/protect/container_host_security/index.md index 98ba4a1f84d..a0297a7a86f 100644 --- a/doc/user/project/clusters/protect/container_host_security/index.md +++ b/doc/user/project/clusters/protect/container_host_security/index.md @@ -6,11 +6,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Container Host Security **(FREE)** -NOTE: -In GitLab 14.5, using a certificate to connect GitLab to a Kubernetes cluster is [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8). -You can continue using Container Host Security, even though it relies on this certificate-based -method. The work to allow all aspects of Container Host Security to function through the [GitLab Agent](../../../../clusters/agent/index.md) -instead of the certificate-based method can be tracked [in this GitLab issue](https://gitlab.com/gitlab-org/gitlab/-/issues/299350). +> [Deprecated](https://gitlab.com/groups/gitlab-org/-/epics/7476) in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) in GitLab 15.0. + +WARNING: +Container Host Security is in its end-of-life process. It's [deprecated](https://gitlab.com/groups/gitlab-org/-/epics/7476) +for use in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) +in GitLab 15.0. Container Host Security in GitLab provides Intrusion Detection and Prevention capabilities that can monitor and (optionally) block activity inside the containers themselves. This is done by leveraging diff --git a/doc/user/project/clusters/protect/container_host_security/quick_start_guide.md b/doc/user/project/clusters/protect/container_host_security/quick_start_guide.md index 466bcb7916f..4e56b7e5140 100644 --- a/doc/user/project/clusters/protect/container_host_security/quick_start_guide.md +++ b/doc/user/project/clusters/protect/container_host_security/quick_start_guide.md @@ -6,6 +6,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Getting started with Container Host Security **(FREE)** +> [Deprecated](https://gitlab.com/groups/gitlab-org/-/epics/7476) in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) in GitLab 15.0. + +WARNING: +Container Host Security is in its end-of-life process. It's [deprecated](https://gitlab.com/groups/gitlab-org/-/epics/7476) +for use in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) +in GitLab 15.0. + The following steps are recommended for installing Container Host Security. ## Installation steps diff --git a/doc/user/project/clusters/protect/container_network_security/index.md b/doc/user/project/clusters/protect/container_network_security/index.md index 06dc6b24620..7176a1cf1b9 100644 --- a/doc/user/project/clusters/protect/container_network_security/index.md +++ b/doc/user/project/clusters/protect/container_network_security/index.md @@ -6,11 +6,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Container Network Security **(FREE)** -NOTE: -In GitLab 14.5, using a certificate to connect GitLab to a Kubernetes cluster is [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8). -You can continue using Container Network Security, even though it relies on this certificate-based -method. The work to allow all aspects of Container Network Security to function through the [GitLab Agent](../../../../clusters/agent/index.md) -instead of the certificate-based method can be tracked [in this GitLab issue](https://gitlab.com/gitlab-org/gitlab/-/issues/299350) and [this GitLab Epic](https://gitlab.com/groups/gitlab-org/-/epics/7057). +> [Deprecated](https://gitlab.com/groups/gitlab-org/-/epics/7476) in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) in GitLab 15.0. + +WARNING: +Container Network Security is in its end-of-life process. It's [deprecated](https://gitlab.com/groups/gitlab-org/-/epics/7476) +for use in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) +in GitLab 15.0. Container Network Security in GitLab provides basic firewall functionality by leveraging Cilium NetworkPolicies to filter traffic going in and out of the cluster as well as traffic between pods diff --git a/doc/user/project/clusters/protect/container_network_security/quick_start_guide.md b/doc/user/project/clusters/protect/container_network_security/quick_start_guide.md index 340c9397e9c..e6c91302d7b 100644 --- a/doc/user/project/clusters/protect/container_network_security/quick_start_guide.md +++ b/doc/user/project/clusters/protect/container_network_security/quick_start_guide.md @@ -6,6 +6,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Getting started with Container Network Security **(FREE)** +> [Deprecated](https://gitlab.com/groups/gitlab-org/-/epics/7476) in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) in GitLab 15.0. + +WARNING: +Container Network Security is in its end-of-life process. It's [deprecated](https://gitlab.com/groups/gitlab-org/-/epics/7476) +for use in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) +in GitLab 15.0. + The following steps are recommended for installing Container Network Security. ## Installation steps diff --git a/doc/user/project/clusters/protect/index.md b/doc/user/project/clusters/protect/index.md index 1314a1948d5..473195f4c17 100644 --- a/doc/user/project/clusters/protect/index.md +++ b/doc/user/project/clusters/protect/index.md @@ -6,6 +6,15 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Protecting your deployed applications **(FREE)** +> [Deprecated](https://gitlab.com/groups/gitlab-org/-/epics/7476) in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) in GitLab 15.0. + +WARNING: +The Container Network Security and Container Host Security features are in their end-of-life +processes. They're +[deprecated](https://gitlab.com/groups/gitlab-org/-/epics/7476) +for use in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) +in GitLab 15.0. + GitLab makes it straightforward to protect applications deployed in [connected Kubernetes clusters](index.md). These protections are available in the Kubernetes network layer and in the container itself. At the network layer, the Container Network Security capabilities in GitLab provide basic firewall diff --git a/doc/user/project/clusters/serverless/index.md b/doc/user/project/clusters/serverless/index.md index 265a60c6f2c..29164da307b 100644 --- a/doc/user/project/clusters/serverless/index.md +++ b/doc/user/project/clusters/serverless/index.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/6) in GitLab 14.3. WARNING: -Serverless is currently in [alpha](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha). +Serverless is currently in [alpha](../../../../policy/alpha-beta-support.md#alpha-features). ## Overview diff --git a/doc/user/project/code_owners.md b/doc/user/project/code_owners.md index 4068d8e056c..eb18834cc6b 100644 --- a/doc/user/project/code_owners.md +++ b/doc/user/project/code_owners.md @@ -281,7 +281,7 @@ README.md @docs ### Approvals shown as optional -A Code Owner approval rule is optional if these conditions are not met: +A Code Owner approval rule is optional if any of these conditions are true: - The user or group are not a member of the project or parent group. - [Code Owner approval on a protected branch](protected_branches.md#require-code-owner-approval-on-a-protected-branch) has not been set up. diff --git a/doc/user/project/deploy_keys/index.md b/doc/user/project/deploy_keys/index.md index 2e876b24b53..57f6efa3092 100644 --- a/doc/user/project/deploy_keys/index.md +++ b/doc/user/project/deploy_keys/index.md @@ -29,7 +29,7 @@ repository in automation, it's a simple solution. A drawback is that your repository could become vulnerable if a remote machine is compromised by a hacker. You should limit access to the remote machine before a deploy key is enabled on your repository. A good rule to follow is to provide access only to trusted users, -and make sure that the allowed users have at least the [Maintainer role](../../permissions.md) +and make sure that the allowed users have at least the Maintainer role in the GitLab project. If this security implication is a concern for your organization, diff --git a/doc/user/project/deploy_tokens/index.md b/doc/user/project/deploy_tokens/index.md index f57fa5aa57d..4d69209aafa 100644 --- a/doc/user/project/deploy_tokens/index.md +++ b/doc/user/project/deploy_tokens/index.md @@ -14,7 +14,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w Deploy tokens allow you to download (`git clone`) or push and pull packages and container registry images of a project without having a user and a password. -Deploy tokens can be managed by [maintainers only](../../permissions.md). +Deploy tokens can be managed only by users with the Maintainer role. Deploy tokens cannot be used with the GitLab API. diff --git a/doc/user/project/description_templates.md b/doc/user/project/description_templates.md index 6c17964f3a5..539f5230063 100644 --- a/doc/user/project/description_templates.md +++ b/doc/user/project/description_templates.md @@ -114,7 +114,9 @@ To re-use templates [you've created](../project/description_templates.md#create- You might also be interested in templates for various [file types in groups](../group/index.md#group-file-templates). -### Set a default template for merge requests and issues **(PREMIUM)** +### Set a default template for merge requests and issues + +> `Default.md` template [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78302) in GitLab 14.8. In a project, you can choose a default description template for new issues and merge requests. As a result, every time a new merge request or issue is created, it's pre-filled with the text you @@ -125,17 +127,29 @@ Prerequisites: - On your project's left sidebar, select **Settings > General** and expand **Visibility, project features, permissions**. Ensure issues or merge requests are set to either **Everyone with access** or **Only Project Members**. -To set a default description template for merge requests: +To set a default description template for merge requests, either: -1. Go to your project's **Settings**. -1. Select **Expand** under the **Merge requests** header. -1. Fill in the **Default description template for merge requests** text area. -1. Select **Save changes**. +- [Create a merge request template](#create-a-merge-request-template) named `Default.md` and save it in `.gitlab/merge_request_templates/`. + This will not overwrite the default template if one has been set in the project settings. +- Users on GitLab Premium and higher: set the default template in project settings: + + 1. On the top bar, select **Menu > Projects** and find your project. + 1. On the left sidebar, select **Settings**. + 1. Expand **Merge requests**. + 1. Fill in the **Default description template for merge requests** text area. + 1. Select **Save changes**. + +To set a default description template for issues, either: -To set a default description template for issues: +- [Create an issue template](#create-an-issue-template) named `Default.md` and save it in `.gitlab/issue_templates/`. + This will not overwrite the default template if one has been set in the project settings. +- Users on GitLab Premium and higher: set the default template in project settings: -1. Select **Expand** under **Default issue template**. -1. Fill in the **Default description template for issues** text area. + 1. On the top bar, select **Menu > Projects** and find your project. + 1. On the left sidebar, select **Settings**. + 1. Expand **Default issue template**. + 1. Fill in the **Default description template for issues** text area. + 1. Select **Save changes**. Because GitLab merge request and issues support [Markdown](../markdown.md), you can use it to format headings, lists, and so on. @@ -143,6 +157,16 @@ headings, lists, and so on. You can also provide `issues_template` and `merge_requests_template` attributes in the [Projects REST API](../../api/projects.md) to keep your default issue and merge request templates up to date. +#### Priority of description templates + +When you set [merge request and issue description templates](#set-a-default-template-for-merge-requests-and-issues) +in various places, they have the following priorities in a project. +The ones higher up override the ones below: + +1. Template selected in project settings. +1. `Default.md` from the parent group. +1. `Default.md` from the project repository. + ## Example description template We use description templates for issues and merge requests in the diff --git a/doc/user/project/file_lock.md b/doc/user/project/file_lock.md index 1d06b605aa9..9911c90863d 100644 --- a/doc/user/project/file_lock.md +++ b/doc/user/project/file_lock.md @@ -33,7 +33,7 @@ GitLab supports two different modes of file locking: ## Permissions Locks can be created by any person who has at least -[Developer role](../permissions.md) in the repository. +Developer role in the repository. Only the user who locked the file or directory can edit locked files. Other users are prevented from modifying locked files by pushing, merging, @@ -77,7 +77,7 @@ Check this document to learn more about [using Git LFS](../../topics/git/lfs/ind ### Configure Exclusive File Locks -You need the [Maintainer role](../permissions.md) to configure +You need the Maintainer role Exclusive File Locks for your project through the command line. The first thing to do before using File Locking is to tell Git LFS which @@ -226,4 +226,4 @@ To view and remove file locks: This list shows all the files locked either through LFS or GitLab UI. Locks can be removed by their author, or any user -with at least the [Maintainer role](../permissions.md). +with at least the Maintainer role. diff --git a/doc/user/project/img/time_tracking_example_v12_2.png b/doc/user/project/img/time_tracking_example_v12_2.png Binary files differdeleted file mode 100644 index 31d8c490ed1..00000000000 --- a/doc/user/project/img/time_tracking_example_v12_2.png +++ /dev/null diff --git a/doc/user/project/import/bitbucket.md b/doc/user/project/import/bitbucket.md index 0c50fc77e33..62495872659 100644 --- a/doc/user/project/import/bitbucket.md +++ b/doc/user/project/import/bitbucket.md @@ -42,7 +42,7 @@ When issues/pull requests are being imported, the Bitbucket importer tries to fi the Bitbucket author/assignee in the GitLab database using the Bitbucket `nickname`. For this to work, the Bitbucket author/assignee should have signed in beforehand in GitLab and **associated their Bitbucket account**. Their `nickname` must also match their Bitbucket -`username.`. If the user is not found in the GitLab database, the project creator +`username`. If the user is not found in the GitLab database, the project creator (most of the times the current user that started the import process) is set as the author, but a reference on the issue about the original Bitbucket author is kept. diff --git a/doc/user/project/import/bitbucket_server.md b/doc/user/project/import/bitbucket_server.md index da47b673c29..4e3642eb3bd 100644 --- a/doc/user/project/import/bitbucket_server.md +++ b/doc/user/project/import/bitbucket_server.md @@ -5,130 +5,110 @@ group: Import info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Import your project from Bitbucket Server to GitLab **(FREE)** +# Import your project from Bitbucket Server **(FREE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/20164) in GitLab 11.2. NOTE: -The Bitbucket Server importer does not work with [Bitbucket Cloud](https://bitbucket.org). -Use the [Bitbucket Cloud importer](bitbucket.md) for that. +This process is different than [importing from Bitbucket Cloud](bitbucket.md). -Import your projects from Bitbucket Server to GitLab with minimal effort. +From Bitbucket Server, you can import: -The Bitbucket importer can import: - -- Repository description (GitLab 11.2+) -- Git repository data (GitLab 11.2+) -- Pull requests (GitLab 11.2+) -- Pull request comments (GitLab 11.2+) +- Repository description +- Git repository data +- Pull requests +- Pull request comments When importing, repository public access is retained. If a repository is private in Bitbucket, it's created as private in GitLab as well. -## Limitations +## Import your Bitbucket repositories -- GitLab doesn't allow comments on arbitrary lines of code, so any Bitbucket comments out of bounds - are inserted as comments in the merge request. -- Bitbucket Server allows multiple levels of threading. GitLab import collapses this into one thread - and quote part of the original comment. -- Declined pull requests have unreachable commits, which prevents the GitLab importer from - generating a proper diff. These pull requests show up as empty changes. -- Pull request approvals are not imported. -- Attachments in Markdown are not imported. -- Task lists are not imported. -- Emoji reactions are not imported. -- Project filtering does not support fuzzy search (only `starts with` or `full match strings` are - supported). +Prerequisites: -## How it works +- An administrator must have enabled the **Bitbucket Server** in + **Admin > Settings > General > Visibility and access controls > Import sources**. +- Review the importer's [limitations](#limitations). -The Bitbucket Server importer works as follows: +To import your Bitbucket repositories: -1. The user is prompted to enter the URL, username, and password (or personal access token) to log in to Bitbucket. - These credentials are preserved only as long as the importer is running. -1. The importer attempts to list all the current repositories on the Bitbucket Server. -1. Upon selection, the importer clones the repository and import pull requests and comments. +1. Sign in to GitLab. +1. On the top bar, select **New** (**{plus}**). +1. Select **New project/repository**. +1. Select **Import project**. +1. Select **Bitbucket Server**. +1. Log in to Bitbucket and grant GitLab access to your Bitbucket account. +1. Select the projects to import, or import all projects. You can filter projects by name and select + the namespace for which to import each project. -### User assignment +## Limitations -When issues/pull requests are being imported, the Bitbucket importer tries to -find the author's email address with a confirmed email address in the GitLab -user database. If no such user is available, the project creator is set as -the author. The importer appends a note in the comment to mark the original -creator. +- GitLab doesn't allow comments on arbitrary lines of code. Any out-of-bounds Bitbucket comments are + inserted as comments in the merge request. +- Bitbucket Server allows multiple threading levels. The importer collapses this into one thread and + quotes part of the original comment. +- Declined pull requests have unreachable commits. This prevents the importer from generating a + proper diff. These pull requests show up as empty changes. +- Project filtering doesn't support fuzzy search. Only starts with or full match strings are + supported. -The importer creates any new namespaces (groups) if they don't exist or in -the case the namespace is taken, the repository is imported under the user's -namespace that started the import process. +The following aren't imported: -#### User assignment by username +- Pull request approvals +- Attachments in Markdown +- Task lists +- Emoji reactions -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218609) in GitLab 13.4. -> - It's [deployed behind a feature flag](../../feature_flags.md), disabled by default. -> - It's disabled on GitLab.com. -> - It's not recommended for production use. -> - To use it in GitLab self-managed instances, ask a GitLab administrator to enable it. +## User assignment -WARNING: -This feature might not be available to you. Check the **version history** note above for details. +When issues and pull requests are importing, the importer tries to find the author's email address +with a confirmed email address in the GitLab user database. If no such user is available, the +project creator is set as the author. The importer appends a note in the comment to mark the +original creator. -If you've enabled this feature, the importer tries to find a user in the GitLab user database with -the author's: +The importer creates any new namespaces (groups) if they don't exist. If the namespace is taken, the +repository imports under the namespace of the user who started the import process. -- `username` -- `slug` -- `displayName` +### User assignment by username -If the user is not found by any of these properties, the project creator is set as the author. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218609) in GitLab 13.4 [with a flag](../../../administration/feature_flags.md) named `bitbucket_server_user_mapping_by_username`. Disabled by default. +> - Not recommended for production use. -##### Enable or disable User assignment by username +FLAG: +On self-managed GitLab and GitLab.com, by default this feature is not available. To make it +available, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) +named `bitbucket_server_user_mapping_by_username`. This feature is not ready for production use. -User assignment by username is under development and not ready for production use. It is -deployed behind a feature flag that is **disabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) -can enable it. +With this feature enabled, the importer tries to find a user in the GitLab user database with the +author's: -To enable it: +- `username` +- `slug` +- `displayName` -```ruby -Feature.enable(:bitbucket_server_user_mapping_by_username) -``` +If no user matches these properties, the project creator is set as the author. -To disable it: +## Troubleshooting -```ruby -Feature.disable(:bitbucket_server_user_mapping_by_username) -``` +### General -## Import your Bitbucket repositories +If the GUI-based import tool does not work, you can try to: -Prerequisite: +- Use the [GitLab Import API](../../../api/import.md#import-repository-from-bitbucket-server) + Bitbucket Server endpoint. +- Set up [repository mirroring](../repository/mirror/index.md). + It provides verbose error output. -- An administrator must have enabled the importer in - **Admin > Application Settings > Visibility and access controls > Import sources**. +See the [troubleshooting section](bitbucket.md#troubleshooting) +for Bitbucket Cloud. -To import your Bitbucket repositories: +### LFS objects not imported -1. Sign in to GitLab. -1. On the top bar, select **New** (**{plus}**). -1. Select **New project/repository**. -1. Select **Import project**. -1. Select **Bitbucket Server**. -1. Log in to Bitbucket and grant GitLab access to your Bitbucket account. -1. Select the projects that you'd like to import or import all projects. - You can filter projects by name and select the namespace - each project will be imported for. +If the project import completes but LFS objects can't be downloaded or cloned, you may be using a +password or personal access token containing special characters. For more information, see +[this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/337769). -## Automate group and project import **(PREMIUM)** +## Related topics For information on automating user, group, and project import API calls, see [Automate group and project import](index.md#automate-group-and-project-import). - -## Troubleshooting - -If the GUI-based import tool does not work, you can try to: - -- Use the [GitLab Import API](../../../api/import.md#import-repository-from-bitbucket-server) Bitbucket server endpoint. -- Set up [Repository Mirroring](../repository/mirror/index.md), which provides verbose error output. - -See the [troubleshooting](bitbucket.md#troubleshooting) section for [Bitbucket](bitbucket.md). diff --git a/doc/user/project/import/github.md b/doc/user/project/import/github.md index d4cca723333..9f1c049045c 100644 --- a/doc/user/project/import/github.md +++ b/doc/user/project/import/github.md @@ -26,7 +26,7 @@ The following aspects of a project are imported: - Regular issue and pull request comments - [Git Large File Storage (LFS) Objects](../../../topics/git/lfs/index.md) - Pull request comments replies in discussions ([GitLab.com & 14.5+](https://gitlab.com/gitlab-org/gitlab/-/issues/336596)) -- Diff Notes suggestions ([GitLab.com & 14.7+](https://gitlab.com/gitlab-org/gitlab/-/issues/340624)) [with a flag](../../../administration/feature_flags.md) named `github_importer_use_diff_note_with_suggestions`. Enabled by default. +- Diff Notes suggestions ([GitLab.com & 14.7+](https://gitlab.com/gitlab-org/gitlab/-/issues/340624)) References to pull requests and issues are preserved (GitLab.com & 8.7+), and each imported repository maintains visibility level unless that [visibility @@ -181,8 +181,7 @@ Mirroring does not sync any new or updated pull requests from your GitHub projec ## Improve the speed of imports on self-managed instances -NOTE: -An administrator role on the GitLab server is required for this process. +Administrator access on the GitLab server is required for this process. For large projects it may take a while to import all data. To reduce the time necessary, you can increase the number of Sidekiq workers that process the following queues: diff --git a/doc/user/project/import/index.md b/doc/user/project/import/index.md index 001f0d56cc5..41ef15108ec 100644 --- a/doc/user/project/import/index.md +++ b/doc/user/project/import/index.md @@ -50,7 +50,7 @@ information, see the prerequisites and important notes in these sections: NOTE: When migrating to GitLab.com, you must create users manually unless [SCIM](../../../user/group/saml_sso/scim_setup.md) will be used. Creating users with the API is limited to self-managed instances as it requires -the Administrator role. +administrator access. To migrate all data from self-managed to GitLab.com, you can leverage the [API](../../../api/index.md). Migrate the assets in this order: diff --git a/doc/user/project/import/jira.md b/doc/user/project/import/jira.md index c6992422733..5f7475eac36 100644 --- a/doc/user/project/import/jira.md +++ b/doc/user/project/import/jira.md @@ -40,9 +40,8 @@ iterations of the GitLab Jira importer. ### Permissions -In order to be able to import issues from a Jira project you need to have read access on Jira -issues and a [Maintainer or higher](../../permissions.md#project-members-permissions) role in the -GitLab project that you wish to import into. +To be able to import issues from a Jira project you must have read access on Jira +issues and at least the Maintainer role in the GitLab project that you wish to import into. ### Jira integration diff --git a/doc/user/project/import/phabricator.md b/doc/user/project/import/phabricator.md index 80f2d6d7e62..96b38b49960 100644 --- a/doc/user/project/import/phabricator.md +++ b/doc/user/project/import/phabricator.md @@ -11,7 +11,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: The Phabricator task importer is in -[beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta) and is +[beta](../../../policy/alpha-beta-support.md#beta-features) and is [**not** complete](https://gitlab.com/gitlab-org/gitlab/-/issues/284406). It imports only an issue's title and description. The GitLab project created during the import process contains only issues, and the associated repository is disabled. diff --git a/doc/user/project/index.md b/doc/user/project/index.md index bee097cdcbe..801c2520bda 100644 --- a/doc/user/project/index.md +++ b/doc/user/project/index.md @@ -105,7 +105,6 @@ Projects include the following [features](https://about.gitlab.com/features/): - [License Compliance](../compliance/license_compliance/index.md): Approve and deny licenses for projects. **(ULTIMATE)** - [Dependency List](../application_security/dependency_list/index.md): View project dependencies. **(ULTIMATE)** - [Requirements](requirements/index.md): Create criteria to check your products against. **(ULTIMATE)** -- [Static Site Editor](static_site_editor/index.md): Edit content on static websites without prior knowledge of the codebase or Git commands. - [Code Intelligence](code_intelligence.md): Navigate code. ## Project integrations diff --git a/doc/user/project/integrations/bamboo.md b/doc/user/project/integrations/bamboo.md index 38de8d9f1af..bf343078634 100644 --- a/doc/user/project/integrations/bamboo.md +++ b/doc/user/project/integrations/bamboo.md @@ -46,6 +46,7 @@ integration in GitLab. 1. Select **Atlassian Bamboo**. 1. Ensure the **Active** checkbox is selected. 1. Enter the base URL of your Bamboo server. For example, `https://bamboo.example.com`. +1. Optional. Clear the **Enable SSL verification** checkbox to disable [SSL verification](overview.md#ssl-verification). 1. Enter the [build key](#identify-the-bamboo-build-plan-build-key) from your Bamboo build plan. 1. If necessary, enter a username and password for a Bamboo user that has diff --git a/doc/user/project/integrations/index.md b/doc/user/project/integrations/index.md index ac6e18e8e6a..9764c4d44a0 100644 --- a/doc/user/project/integrations/index.md +++ b/doc/user/project/integrations/index.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w You can find the available integrations under your project's **Settings > Integrations** page. You need to have at least -the [Maintainer role](../../permissions.md) on the project. +the Maintainer role on the project. ## Integrations diff --git a/doc/user/project/integrations/mattermost_slash_commands.md b/doc/user/project/integrations/mattermost_slash_commands.md index 1ff558b569b..b317e65bdf2 100644 --- a/doc/user/project/integrations/mattermost_slash_commands.md +++ b/doc/user/project/integrations/mattermost_slash_commands.md @@ -67,7 +67,7 @@ After you enable custom slash commands in Mattermost, you need configuration information from GitLab. To get this information: 1. In a different browser tab than your current Mattermost session, sign in to - GitLab as a user with [Administrator role](../../permissions.md). + GitLab as a user with administrator access. 1. On the top bar, select **Menu > Admin**. 1. In the left menu, select **Settings > Integrations**, then select **Mattermost slash commands**. diff --git a/doc/user/project/integrations/overview.md b/doc/user/project/integrations/overview.md index 5b83df9b22e..8ecc16050be 100644 --- a/doc/user/project/integrations/overview.md +++ b/doc/user/project/integrations/overview.md @@ -82,6 +82,15 @@ instance configuration or provide custom settings. Read more about [Project integration management](../../admin_area/settings/project_integration_management.md). +## SSL verification + +By default, the SSL certificate for outgoing HTTP requests is verified based on +an internal list of Certificate Authorities. This means the certificate cannot +be self-signed. + +You can turn off SSL verification in the configuration settings for [webhooks](webhooks.md#configure-a-webhook) +and some integrations. + ## Troubleshooting integrations Some integrations use service hooks for integration with external applications. To confirm which ones use service hooks, see the [integrations listing](#integrations-listing) above. Learn more about [troubleshooting service hooks](webhooks.md#troubleshoot-webhooks). diff --git a/doc/user/project/integrations/prometheus.md b/doc/user/project/integrations/prometheus.md index de7ac6782d6..760b5030416 100644 --- a/doc/user/project/integrations/prometheus.md +++ b/doc/user/project/integrations/prometheus.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/project/integrations/prometheus_library/cloudwatch.md b/doc/user/project/integrations/prometheus_library/cloudwatch.md index a07abf26fba..e8d611af30d 100644 --- a/doc/user/project/integrations/prometheus_library/cloudwatch.md +++ b/doc/user/project/integrations/prometheus_library/cloudwatch.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/project/integrations/prometheus_library/haproxy.md b/doc/user/project/integrations/prometheus_library/haproxy.md index 97f69d65412..76d13d5487c 100644 --- a/doc/user/project/integrations/prometheus_library/haproxy.md +++ b/doc/user/project/integrations/prometheus_library/haproxy.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/project/integrations/prometheus_library/index.md b/doc/user/project/integrations/prometheus_library/index.md index a5fc398e558..9bdd4945f5d 100644 --- a/doc/user/project/integrations/prometheus_library/index.md +++ b/doc/user/project/integrations/prometheus_library/index.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/project/integrations/prometheus_library/kubernetes.md b/doc/user/project/integrations/prometheus_library/kubernetes.md index 26d006adeb9..33a06958e0c 100644 --- a/doc/user/project/integrations/prometheus_library/kubernetes.md +++ b/doc/user/project/integrations/prometheus_library/kubernetes.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/project/integrations/prometheus_library/nginx.md b/doc/user/project/integrations/prometheus_library/nginx.md index ad89543e9a6..ecf75d7b17a 100644 --- a/doc/user/project/integrations/prometheus_library/nginx.md +++ b/doc/user/project/integrations/prometheus_library/nginx.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/project/integrations/prometheus_library/nginx_ingress.md b/doc/user/project/integrations/prometheus_library/nginx_ingress.md index 03bf9258659..e123000e0c5 100644 --- a/doc/user/project/integrations/prometheus_library/nginx_ingress.md +++ b/doc/user/project/integrations/prometheus_library/nginx_ingress.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md b/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md index 89c174f8fb9..fda7744e847 100644 --- a/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md +++ b/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/project/integrations/webex_teams.md b/doc/user/project/integrations/webex_teams.md index de152aabde5..dd4cdb632e6 100644 --- a/doc/user/project/integrations/webex_teams.md +++ b/doc/user/project/integrations/webex_teams.md @@ -13,7 +13,7 @@ You can configure GitLab to send notifications to a Webex Teams space: ## Create a webhook for the space -1. Go to the [Incoming Webhooks app page](https://apphub.webex.com/applications/incoming-webhooks-cisco-systems-38054). +1. Go to the [Incoming Webhooks app page](https://apphub.webex.com/applications/incoming-webhooks-cisco-systems-38054-23307). 1. Select **Connect** and log in to Webex Teams, if required. 1. Enter a name for the webhook and select the space to receive the notifications. 1. Select **ADD**. diff --git a/doc/user/project/integrations/webhook_events.md b/doc/user/project/integrations/webhook_events.md index ab70a2d43f4..9d8b98edba1 100644 --- a/doc/user/project/integrations/webhook_events.md +++ b/doc/user/project/integrations/webhook_events.md @@ -773,6 +773,7 @@ Merge request events are triggered when: - A new merge request is created. - An existing merge request is updated, approved, unapproved, merged, or closed. - A commit is added in the source branch. +- All threads are resolved on the merge request. The available values for `object_attributes.action` in the payload are: @@ -838,6 +839,7 @@ Payload example: "updated_at": "2013-12-03T17:23:34Z", "milestone_id": null, "state": "opened", + "blocking_discussions_resolved": true, "merge_status": "unchecked", "target_project_id": 14, "iid": 1, diff --git a/doc/user/project/integrations/webhooks.md b/doc/user/project/integrations/webhooks.md index 8bc2b51276a..ace5783c852 100644 --- a/doc/user/project/integrations/webhooks.md +++ b/doc/user/project/integrations/webhooks.md @@ -57,7 +57,7 @@ You can configure a webhook for a group or a project. The URL must be percent-encoded if it contains one or more special characters. 1. In **Secret token**, enter the [secret token](#validate-payloads-by-using-a-secret-token) to validate payloads. 1. In the **Trigger** section, select the [events](webhook_events.md) to trigger the webhook. -1. Optional. Clear the **Enable SSL verification** checkbox to disable [SSL verification](#verify-an-ssl-certificate). +1. Optional. Clear the **Enable SSL verification** checkbox to disable [SSL verification](overview.md#ssl-verification). 1. Select **Add webhook**. ## Test a webhook @@ -123,15 +123,6 @@ The token is sent with the hook request in the `X-Gitlab-Token` HTTP header. Your webhook endpoint can check the token to verify that the request is legitimate. -## Verify an SSL certificate - -By default, the SSL certificate of the webhook endpoint is verified based on -an internal list of Certificate Authorities. This means the certificate cannot -be self-signed. - -You can turn off SSL verification in the [webhook settings](#configure-a-webhook) -in your GitLab projects. - ## Filter push events by branch Push events can be filtered by branch using a branch name or wildcard pattern diff --git a/doc/user/project/issue_board.md b/doc/user/project/issue_board.md index 47a2d215024..71440298d85 100644 --- a/doc/user/project/issue_board.md +++ b/doc/user/project/issue_board.md @@ -194,7 +194,7 @@ card includes: ## Permissions -Users with the [Reporter and higher roles](../permissions.md) can use all the functionality of the +Users with at least the Reporter role can use all the functionality of the issue board feature to create or delete lists. They can also drag issues from one list to another. ## Ordering issues in a list @@ -402,7 +402,7 @@ To set a WIP limit for a list: > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34723) in GitLab 12.8. > - [View blocking issues when hovering over blocked icon](https://gitlab.com/gitlab-org/gitlab/-/issues/210452) in GitLab 13.10. -If an issue is blocked by another issue, an icon appears next to its title to indicate its blocked +If an issue is [blocked by another issue](issues/related_issues.md#blocking-issues), an icon appears next to its title to indicate its blocked status. When you hover over the blocked icon (**{issue-block}**), a detailed information popover is displayed. @@ -496,6 +496,9 @@ The steps depend on the scope of the list: ### Filter issues +> - Filtering by iteration [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118742) in GitLab 13.6. +> - Filtering by issue type [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/268152) in GitLab 14.6. + You can use the filters on top of your issue board to show only the results you want. It's similar to the filtering used in the [issue tracker](issues/index.md). @@ -504,11 +507,12 @@ You can filter by the following: - Assignee - Author - Epic -- Iteration ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118742) in GitLab 13.6) +- Iteration - Label - Milestone - My Reaction - Release +- Type (issue/incident) - Weight #### Filtering issues in a group board @@ -565,7 +569,7 @@ You can move issues and lists by dragging them. Prerequisites: -- You must have at least the Reporter [role](../permissions.md#project-members-permissions) for a project in GitLab. +- You must have at least the Reporter role for a project in GitLab. To move an issue, select the issue card and drag it to another position in its current list or into a different list. Learn about possible effects in [Dragging issues between lists](#dragging-issues-between-lists). diff --git a/doc/user/project/issues/associate_zoom_meeting.md b/doc/user/project/issues/associate_zoom_meeting.md index aba8c45699c..41de91d9bd7 100644 --- a/doc/user/project/issues/associate_zoom_meeting.md +++ b/doc/user/project/issues/associate_zoom_meeting.md @@ -25,7 +25,7 @@ In an issue, leave a comment using the `/zoom` quick action followed by a valid /zoom https://zoom.us/j/123456789 ``` -If the Zoom meeting URL is valid and you have at least the Reporter [role](../../permissions.md), +If the Zoom meeting URL is valid and you have at least the Reporter role, a system alert notifies you of its successful addition. The issue's description is automatically edited to include the Zoom link, and a button appears right under the issue's title. @@ -44,5 +44,5 @@ Similarly to adding a Zoom meeting, you can remove it with a quick action: /remove_zoom ``` -If you have at least the Reporter [role](../../permissions.md), +If you have at least the Reporter role, a system alert notifies you that the meeting URL was successfully removed. diff --git a/doc/user/project/issues/confidential_issues.md b/doc/user/project/issues/confidential_issues.md index e8c58f2feb9..15130523da6 100644 --- a/doc/user/project/issues/confidential_issues.md +++ b/doc/user/project/issues/confidential_issues.md @@ -11,19 +11,20 @@ Confidential issues are [issues](index.md) visible only to members of a project Confidential issues can be used by open source projects and companies alike to keep security vulnerabilities private or prevent surprises from leaking out. -## Making an issue confidential +## Make an issue confidential -You can make an issue confidential during issue creation or by editing -an existing one. +You can make an issue confidential when you create or edit an issue. When you create a new issue, a checkbox right below the text area is available to mark the issue as confidential. Check that box and hit the **Create issue** button to create the issue. For existing issues, edit them, check the confidential checkbox and hit **Save changes**. +When you create a confidential issue in a project, the project becomes listed in the **Contributed projects** section in your [profile](../../profile/index.md). **Contributed projects** does not show information about the confidential issue; it only shows the project name. +  -## Modifying issue confidentiality +## Modify issue confidentiality There are two ways to change an issue's confidentiality. @@ -42,15 +43,15 @@ system note in the issue's comments.  -When an issue is made confidential, only users with at least the [Reporter role](../../permissions.md) +When an issue is made confidential, only users with at least the Reporter role for the project have access to the issue. Users with Guest or [Minimal](../../permissions.md#users-with-minimal-access) roles can't access the issue even if they were actively participating before the change. -## Indications of a confidential issue +## Confidential issue indicators There are a few things that visually separate a confidential issue from a -regular one. In the issues index page view, you can see the eye-slash (**(eye-slash)**) icon +regular one. In the issues index page view, you can see the eye-slash (**{eye-slash}**) icon next to the issues that are marked as confidential:  @@ -74,19 +75,18 @@ There is also an indicator on the sidebar denoting confidentiality. ## Merge requests for confidential issues -Although you can make issues be confidential in public projects, you cannot make -confidential merge requests. Learn how to create [merge requests for confidential issues](../merge_requests/confidential.md) -that prevent leaks of private data. +Although you can create confidential issues (and make existing issues confidential) in a public project, you cannot make confidential merge requests. +Learn how to create [merge requests for confidential issues](../merge_requests/confidential.md) that prevent leaks of private data. ## Permissions and access to confidential issues There are two kinds of level access for confidential issues. The general rule is that confidential issues are visible only to members of a project with at -least the Reporter [role](../../permissions.md#project-members-permissions). However, a guest user can also create +least the Reporter role. However, a guest user can also create confidential issues, but can only view the ones that they created themselves. Confidential issues are also hidden in search results for unprivileged users. -For example, here's what a user with the [Maintainer role](../../permissions.md) and the Guest role +For example, here's what a user with the Maintainer role and the Guest role sees in the project's search results respectively. | Maintainer role | Guest role | diff --git a/doc/user/project/issues/csv_export.md b/doc/user/project/issues/csv_export.md index a9fca4f2b75..947fbdcc2d1 100644 --- a/doc/user/project/issues/csv_export.md +++ b/doc/user/project/issues/csv_export.md @@ -61,29 +61,29 @@ fields if needed, and newlines to separate rows. The first row contains the headers, which are listed in the following table along with a description of the values: -| Column | Description | -|-------------------|-------------| -| Issue ID | Issue `iid` | -| URL | A link to the issue on GitLab | -| Title | Issue `title` | -| State | `Open` or `Closed` | -| Description | Issue `description` | -| Author | Full name of the issue author | -| Author Username | Username of the author, with the `@` symbol omitted | -| Assignee | Full name of the issue assignee | -| Assignee Username | Username of the author, with the `@` symbol omitted | -| Confidential | `Yes` or `No` | -| Locked | `Yes` or `No` | -| Due Date | Formatted as `YYYY-MM-DD` | -| Created At (UTC) | Formatted as `YYYY-MM-DD HH:MM:SS` | -| Updated At (UTC) | Formatted as `YYYY-MM-DD HH:MM:SS` | -| Milestone | Title of the issue milestone | -| Weight | Issue weight | -| Labels | Title of any labels joined with a `,` | -| Time Estimate | [Time estimate](../time_tracking.md#estimates) in seconds | -| Time Spent | [Time spent](../time_tracking.md#time-spent) in seconds | -| Epic ID | ID of the parent epic **(ULTIMATE)**, introduced in 12.7 | -| Epic Title | Title of the parent epic **(ULTIMATE)**, introduced in 12.7 | +| Column | Description | +|------------------------------------------|-----------------------------------------------------------| +| Issue ID | Issue `iid` | +| URL | A link to the issue on GitLab | +| Title | Issue `title` | +| State | `Open` or `Closed` | +| Description | Issue `description` | +| Author | Full name of the issue author | +| Author Username | Username of the author, with the `@` symbol omitted | +| Assignee | Full name of the issue assignee | +| Assignee Username | Username of the author, with the `@` symbol omitted | +| Confidential | `Yes` or `No` | +| Locked | `Yes` or `No` | +| Due Date | Formatted as `YYYY-MM-DD` | +| Created At (UTC) | Formatted as `YYYY-MM-DD HH:MM:SS` | +| Updated At (UTC) | Formatted as `YYYY-MM-DD HH:MM:SS` | +| Milestone | Title of the issue milestone | +| Weight | Issue weight | +| Labels | Title of any labels joined with a `,` | +| Time Estimate | [Time estimate](../time_tracking.md#estimates) in seconds | +| Time Spent | [Time spent](../time_tracking.md#time-spent) in seconds | +| [Epic](../../group/epics/index.md) ID | ID of the parent epic, introduced in 12.7 | +| [Epic](../../group/epics/index.md) Title | Title of the parent epic, introduced in 12.7 | ## Limitations diff --git a/doc/user/project/issues/csv_import.md b/doc/user/project/issues/csv_import.md index 69dc0cbafd8..e4b8efd27ed 100644 --- a/doc/user/project/issues/csv_import.md +++ b/doc/user/project/issues/csv_import.md @@ -14,9 +14,7 @@ retain columns such as labels and milestones, consider the [Move Issue feature]( The user uploading the CSV file is set as the author of the imported issues. -NOTE: -A permission level of [Developer](../../permissions.md), or higher, is required -to import issues. +You must have at least the Developer role for a project to import issues. ## Prepare for the import diff --git a/doc/user/project/issues/design_management.md b/doc/user/project/issues/design_management.md index ecf35fc4dcf..e7bb5ad4eeb 100644 --- a/doc/user/project/issues/design_management.md +++ b/doc/user/project/issues/design_management.md @@ -6,9 +6,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Design Management **(FREE)** -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/660) in GitLab Premium 12.2. -> - Support for SVGs [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12771) in GitLab Premium 12.4. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212566) to GitLab Free in 13.0. +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/660) in GitLab 12.2. +> - Support for SVGs [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12771) in GitLab 12.4. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212566) from GitLab Premium to GitLab Free in 13.0. Design Management allows you to upload design assets (including wireframes and mockups) to GitLab issues and keep them stored in a single place, accessed by the Design @@ -84,10 +84,10 @@ You can find to the **Design Management** section in the issue description: ## Adding designs -> - Drag and drop uploads [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34353) in GitLab Premium 12.9. -> - New version creation on upload [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34353) in GitLab Premium 12.9. +> - Drag and drop uploads [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34353) in GitLab 12.9. +> - New version creation on upload [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34353) in GitLab 12.9. > - Copy and paste uploads [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202634) in GitLab 12.10. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212566) to GitLab Free in 13.0. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212566) from GitLab Premium to GitLab Free in 13.0. To upload Design images, drag files from your computer and drop them in the Design Management section, or select **click to upload** to select images from your file browser: @@ -142,9 +142,9 @@ to help summarize changes between versions. ### Exploring designs by zooming -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13217) in GitLab Premium 12.7. -> - Drag to move a zoomed image [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/197324) in GitLab 12.10. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212566) to GitLab Free in 13.0. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13217) in GitLab 12.7. +> - Ability to drag a zoomed image to move it [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/197324) in GitLab 12.10. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212566) from GitLab Premium to GitLab Free in 13.0. Designs can be explored in greater detail by zooming in and out of the image. Control the amount of zoom with the `+` and `-` buttons at the bottom of the image. @@ -155,8 +155,8 @@ While zoomed in, you can drag the image to move around it. ## Deleting designs -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11089) in GitLab Premium 12.4. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212566) to GitLab Free in 13.0. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11089) in GitLab 12.4. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212566) from GitLab Premium to GitLab Free in 13.0. There are two ways to delete designs: manually delete them individually, or select a few of them to delete at once, @@ -190,8 +190,8 @@ You can change the order of designs by dragging them to a new position. ## Starting discussions on designs -> - Adjusting a pin's position [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34353) in GitLab Premium 12.8. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212566) to GitLab Free in 13.0. +> - Adjusting a pin's position [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34353) adjusting a pin's position in GitLab 12.8. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212566) from GitLab Premium to GitLab Free in 13.0. When a design is uploaded, you can start a discussion by selecting the image on the exact location you would like the discussion to be focused on. diff --git a/doc/user/project/issues/due_dates.md b/doc/user/project/issues/due_dates.md index 2c20ccdcee0..2630052d806 100644 --- a/doc/user/project/issues/due_dates.md +++ b/doc/user/project/issues/due_dates.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Due dates **(FREE)** Due dates can be used in [issues](index.md) to keep track of deadlines and make sure features are -shipped on time. Users need at least the Reporter [role](../../permissions.md) +shipped on time. Users need at least the Reporter role to be able to edit the due date. All users with permission to view the issue can view the due date. diff --git a/doc/user/project/issues/issue_weight.md b/doc/user/project/issues/issue_weight.md index 8f17f399cb0..756fe9699f1 100644 --- a/doc/user/project/issues/issue_weight.md +++ b/doc/user/project/issues/issue_weight.md @@ -15,10 +15,8 @@ value, or complexity a given issue has or costs. You can set the weight of an issue during its creation, by changing the value in the dropdown menu. You can set it to a non-negative integer -value from 0, 1, 2, and so on. (The database stores a 4-byte value, so the -upper bound is essentially limitless.) -You can remove weight from an issue -as well. +value from 0, 1, 2, and so on. +You can remove weight from an issue as well. This value appears on the right sidebar of an individual issue, as well as in the issues page next to a weight icon (**{weight}**). diff --git a/doc/user/project/issues/managing_issues.md b/doc/user/project/issues/managing_issues.md index d120df82dbf..155d6260a5c 100644 --- a/doc/user/project/issues/managing_issues.md +++ b/doc/user/project/issues/managing_issues.md @@ -29,7 +29,7 @@ You can create an issue in many ways in GitLab: Prerequisites: -- You must have at least the [Guest role](../../permissions.md) for the project. +- You must have at least the Guest role for the project. To create an issue: @@ -52,7 +52,7 @@ to the projects in the group. Prerequisites: -- You must have at least the [Guest role](../../permissions.md) for the project in the group. +- You must have at least the Guest role for the project in the group. To create an issue from a group: @@ -78,7 +78,7 @@ You can create a new issue from an existing one. The two issues can then be mark Prerequisites: -- You must have at least the [Guest role](../../permissions.md) for the project. +- You must have at least the Guest role for the project. To create an issue from another issue: @@ -98,7 +98,7 @@ You can create a new issue from an [issue board](../issue_board.md). Prerequisites: -- You must have at least the [Guest role](../../permissions.md) for the project. +- You must have at least the Guest role for the project. To create an issue from a project issue board: @@ -133,7 +133,7 @@ Prerequisites: - Your GitLab instance must have [incoming email](../../../administration/incoming_email.md) configured. - There must be at least one issue in the issue list. -- You must have at least the [Guest role](../../permissions.md) for the project. +- You must have at least the Guest role for the project. To email an issue to a project: @@ -224,7 +224,7 @@ You can edit an issue's title and description. Prerequisites: -- You must have at least the [Reporter role](../../permissions.md) for the project. +- You must have at least the Reporter role for the project, be the author of the issue, or be assigned to the issue. To edit an issue: @@ -242,7 +242,7 @@ You can edit multiple issues at a time when you're in a project. Prerequisites: -- You must have at least the [Reporter role](../../permissions.md) for the project. +- You must have at least the Reporter role for the project. To edit multiple issues at the same time: @@ -275,7 +275,7 @@ You can edit multiple issues across multiple projects when you're in a group. Prerequisites: -- You must have at least the [Reporter role](../../permissions.md) for a group. +- You must have at least the Reporter role for a group. To edit multiple issues at the same time: @@ -300,9 +300,11 @@ When you move an issue, it's closed and copied to the target project. The original issue is not deleted. A system note, which indicates where it came from and went to, is added to both issues. +Be careful when moving an issue to a project with different access rules. Before moving the issue, make sure it does not contain sensitive data. + Prerequisites: -- You must have at least the [Reporter role](../../permissions.md) for the project. +- You must have at least the Reporter role for the project. To move an issue: @@ -317,7 +319,7 @@ You can move all open issues from one project to another. Prerequisites: -- You must have at least the [Reporter role](../../permissions.md) for the project. +- You must have access to the Rails console of the GitLab instance. To do it: @@ -351,7 +353,7 @@ The issue is marked as closed but is not deleted. Prerequisites: -- You must have at least the [Reporter role](../../permissions.md) for the project. +- You must have at least the Reporter role for the project, be the author of the issue, or be assigned to the issue. To close an issue, you can do the following: @@ -364,7 +366,7 @@ To close an issue, you can do the following: Prerequisites: -- You must have at least the [Reporter role](../../permissions.md) for the project. +- You must have at least the Reporter role for the project, be the author of the issue, or be assigned to the issue. To reopen a closed issue, at the top of the issue, select **Reopen issue**. A reopened issue is no different from any other open issue. @@ -440,7 +442,7 @@ in the [project's settings](../settings/index.md). Prerequisites: -- You must have at least the [Maintainer role](../../permissions.md) for the project. +- You must have at least the Maintainer role for the project. To disable automatic issue closing: @@ -466,7 +468,7 @@ Merge requests in other projects can still close another project's issues. Prerequisites: -- You must have the [administrator access level](../../../administration/index.md) for your GitLab instance. +- You must have [administrator access](../../../administration/index.md) to your GitLab instance. To change the default issue closing pattern, edit the [`gitlab.rb` or `gitlab.yml` file](../../../administration/issue_closing_pattern.md) @@ -476,7 +478,7 @@ of your installation. Prerequisites: -- You must be the issue author or have at least the [Reporter role](../../permissions.md) for the project. +- You must be the issue author or have at least the Reporter role for the project, be the author of the issue, or be assigned to the issue. To change issue type: @@ -494,7 +496,7 @@ To change issue type: Prerequisites: -- You must have the [Owner role](../../permissions.md) for a project. +- You must have the Owner role for a project. To delete an issue: @@ -508,9 +510,9 @@ Alternatively: ## Promote an issue to an epic **(PREMIUM)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3777) in GitLab Ultimate 11.6. -> - Moved to GitLab Premium in 12.8. -> - Promoting issues to epics via the UI [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/233974) in GitLab Premium 13.6. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3777) in GitLab 11.6. +> - Moved from GitLab Ultimate to GitLab Premium in 12.8. +> - Promoting issues to epics via the UI [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/233974) in GitLab 13.6. You can promote an issue to an [epic](../../group/epics/index.md) in the immediate parent group. @@ -622,7 +624,7 @@ This status marks issues as progressing as planned or needing attention to keep Prerequisites: -- You must have at least the [Reporter role](../../permissions.md) for the project. +- You must have at least the Reporter role for the project. To edit health status of an issue: diff --git a/doc/user/project/issues/multiple_assignees_for_issues.md b/doc/user/project/issues/multiple_assignees_for_issues.md index 98e940b6b51..f957d701a3b 100644 --- a/doc/user/project/issues/multiple_assignees_for_issues.md +++ b/doc/user/project/issues/multiple_assignees_for_issues.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Multiple Assignees for Issues **(PREMIUM)** -> - Moved to GitLab Premium in 13.9. +> Moved to GitLab Premium in 13.9. In large teams, where there is shared ownership of an issue, it can be difficult to track who is working on it, who already completed their contributions, who diff --git a/doc/user/project/issues/related_issues.md b/doc/user/project/issues/related_issues.md index 8a2a104c54d..f83ebc5e8a8 100644 --- a/doc/user/project/issues/related_issues.md +++ b/doc/user/project/issues/related_issues.md @@ -6,10 +6,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Linked issues **(FREE)** -> The simple "relates to" relationship [moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212329) to [GitLab Free](https://about.gitlab.com/pricing/) in 13.4. +> The simple "relates to" relationship [moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212329) from GitLab Premium to GitLab Free in 13.4. Linked issues are a bi-directional relationship between any two issues and appear in a block below -the issue description. Issues can be across groups and projects. +the issue description. You can link issues in different projects. The relationship only shows up in the UI if the user can see both issues. When you try to close an issue that has open blockers, a warning is displayed. @@ -23,13 +23,18 @@ To manage linked issues through our API, visit the [issue links API documentatio > - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/34239) to warn when attempting to close an issue that is blocked by others in GitLab 13.0. > When you try to close an issue with open blockers, you see a warning that you can dismiss. -1. Link one issue to another by selecting the add linked issue button (**{plus}**) in the - **Linked issues** section of an issue. +Prerequisites: +- You must have at least the Reporter role for both projects. + +To link one issue to another: + +1. In the **Linked issues** section of an issue, + select the add linked issue button (**{plus}**). 1. Select the relationship between the two issues. Either: - **relates to** - - **blocks** **(PREMIUM)** - - **is blocked by** **(PREMIUM)** + - **[blocks](#blocking-issues)** + - **[is blocked by](#blocking-issues)** 1. Input the issue number or paste in the full URL of the issue.  @@ -64,3 +69,10 @@ Due to the bi-directional relationship, the relationship no longer appears in ei  Access our [permissions](../../permissions.md) page for more information. + +## Blocking issues **(PREMIUM)** + +When you [add a linked issue](#add-a-linked-issue), you can show that it **blocks** or +**is blocked by** another issue. + +Issues that block other issues have an icon (**{issue-block}**) shown in the issue lists and [boards](../issue_board.md). diff --git a/doc/user/project/issues/sorting_issue_lists.md b/doc/user/project/issues/sorting_issue_lists.md index 0340f15c25c..329f65bfacd 100644 --- a/doc/user/project/issues/sorting_issue_lists.md +++ b/doc/user/project/issues/sorting_issue_lists.md @@ -27,7 +27,7 @@ The available sorting options can change based on the context of the list. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34247/) in GitLab 13.7. When you sort by **Blocking**, the issue list changes to sort descending by the -number of issues each issue is blocking. +number of issues each issue is [blocking](related_issues.md#blocking-issues). ## Sorting by created date diff --git a/doc/user/project/members/index.md b/doc/user/project/members/index.md index 2dc29f5d725..8be2ade3f2f 100644 --- a/doc/user/project/members/index.md +++ b/doc/user/project/members/index.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -12,20 +12,30 @@ Each member gets a role, which determines what they can do in the project. ## Add users to a project +> - [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/247208) in GitLab 13.11 from a form to a modal window [with a flag](../../feature_flags.md). Disabled by default. +> - Modal window [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/247208) in GitLab 14.8. + +FLAG: +On self-managed GitLab, by default the modal window feature is available. +To hide the feature, ask an administrator to [disable the feature flag](../../../administration/feature_flags.md) +named `invite_members_group_modal`. +On GitLab.com, this feature is available. + Add users to a project so they become members and have permission to perform actions. Prerequisite: -- You must have the [Maintainer or Owner role](../../permissions.md). +- You must have the Maintainer or Owner role. To add a user to a project: -1. Go to your project and select **Project information > Members**. -1. On the **Invite member** tab, under **GitLab member or Email address**, type the username or email address. - In GitLab 13.11 and later, you can [replace this form with a modal window](#add-a-member-modal-window). -1. Select a [role](../../permissions.md). -1. Optional. Choose an expiration date. On that date, the user can no longer access the project. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Project information > Members**. +1. Select **Invite members**. +1. Enter an email address and select a [role](../../permissions.md). +1. Optional. Select an **Access expiration date**. + On that date, the user can no longer access the project. 1. Select **Invite**. If the user has a GitLab account, they are added to the members list. @@ -40,6 +50,15 @@ using the email address the invitation was sent to. ## Add groups to a project +> - [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/247208) in GitLab 13.11 from a form to a modal window [with a flag](../../feature_flags.md). Disabled by default. +> - Modal window [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/247208) in GitLab 14.8. + +FLAG: +On self-managed GitLab, by default the modal window feature is available. +To hide the feature, ask an administrator to [disable the feature flag](../../../administration/feature_flags.md) +named `invite_members_group_modal`. +On GitLab.com, this feature is available. + When you add a group to a project, each user in the group gets access to the project. Each user's access is based on: @@ -48,15 +67,16 @@ Each user's access is based on: Prerequisite: -- You must have the [Maintainer or Owner role](../../permissions.md). +- You must have the Maintainer or Owner role. To add groups to a project: 1. On the top bar, select **Menu > Projects** and find your project. 1. On the left sidebar, select **Project information > Members**. -1. On the **Invite group** tab, under **Select a group to invite**, choose a group. -1. Select the highest max [role](../../permissions.md) for users in the group. -1. Optional. Choose an expiration date. On that date, the user can no longer access the project. +1. Select **Invite a group**. +1. Select a group. +1. Select the highest [role](../../permissions.md) for users in the group. +1. Optional. Select an **Access expiration date**. On that date, the group can no longer access the project. 1. Select **Invite**. The members of the group are not displayed on the **Members** tab. @@ -72,7 +92,7 @@ retain the same permissions as the project you import them from. Prerequisite: -- You must have the [Maintainer or Owner role](../../permissions.md). +- You must have the Maintainer or Owner role. To import users: @@ -111,7 +131,7 @@ group itself. Prerequisites: -- You must have the [Owner role](../../permissions.md). +- You must have the Owner role. - Optional. Unassign the member from all issues and merge requests that are assigned to them. @@ -203,44 +223,3 @@ Prerequisite: ## Share a project with a group Instead of adding users one by one, you can [share a project with an entire group](share_project_with_groups.md). - -### Add a member modal window - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/247208) in GitLab 13.11. -> - [Deployed behind a feature flag](../../feature_flags.md), disabled by default. -> - Enabled on GitLab.com. -> - Recommended for production use. -> - Replaces the existing form with buttons to open a modal window. -> - To use in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-modal-window). - -WARNING: -This feature might not be available to you. Check the **version history** note above for details. - -In GitLab 13.11, you can optionally replace the form to add a member with a modal window. -To add a member after enabling this feature: - -1. On the top bar, select **Menu > Projects** and find your project. -1. On the left sidebar, select **Project information > Members**. -1. Select **Invite members**. -1. Enter an email address and select a role. -1. Optional. Select an **Access expiration date**. -1. Select **Invite**. - -### Enable or disable modal window **(FREE SELF)** - -The modal window for adding a member is under development and is ready for production use. It is -deployed behind a feature flag that is **disabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) -can enable it. - -To enable it: - -```ruby -Feature.enable(:invite_members_group_modal) -``` - -To disable it: - -```ruby -Feature.disable(:invite_members_group_modal) -``` diff --git a/doc/user/project/merge_requests/accessibility_testing.md b/doc/user/project/merge_requests/accessibility_testing.md index e67af8dc936..612f499bb65 100644 --- a/doc/user/project/merge_requests/accessibility_testing.md +++ b/doc/user/project/merge_requests/accessibility_testing.md @@ -1,8 +1,7 @@ --- stage: Verify -group: Testing +group: Pipeline Insights info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference, howto --- # Accessibility testing **(FREE)** diff --git a/doc/user/project/merge_requests/allow_collaboration.md b/doc/user/project/merge_requests/allow_collaboration.md index b10d6597c1e..5826ebcab49 100644 --- a/doc/user/project/merge_requests/allow_collaboration.md +++ b/doc/user/project/merge_requests/allow_collaboration.md @@ -50,7 +50,7 @@ You can push directly to the branch of the forked repository if: - The author of the merge request has enabled contributions from upstream members. -- You have at least the [Developer role](../../permissions.md) in the +- You have at least the Developer role in the upstream project. In the following example: diff --git a/doc/user/project/merge_requests/approvals/index.md b/doc/user/project/merge_requests/approvals/index.md index dddd3925dbb..e940426dc67 100644 --- a/doc/user/project/merge_requests/approvals/index.md +++ b/doc/user/project/merge_requests/approvals/index.md @@ -8,7 +8,7 @@ disqus_identifier: 'https://docs.gitlab.com/ee/user/project/merge_requests/appro # Merge request approvals **(FREE)** -> Redesign [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1979) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8 and [feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/10685) in 12.0. +> Redesign [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1979) in GitLab 11.8 and [feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/10685) in 12.0. You can configure your merge requests so that they must be approved before they can be merged. While [GitLab Free](https://about.gitlab.com/pricing/) allows @@ -89,7 +89,7 @@ a merge request from merging without approval. ## Required approvals **(PREMIUM)** -> Moved to [GitLab Premium](https://about.gitlab.com/pricing/) in 13.9. +> Moved to GitLab Premium in 13.9. Required approvals enforce code reviews by the number and type of users you specify. Without the approvals, the work cannot merge. Required approvals enable multiple use cases: @@ -103,7 +103,7 @@ Without the approvals, the work cannot merge. Required approvals enable multiple to determine who should review the work. - Require an [approval before merging code that causes test coverage to decline](../../../../ci/pipelines/settings.md#coverage-check-approval-rule) - [Require approval from a security team](../../../application_security/index.md#security-approvals-in-merge-requests) - before merging code that could introduce a vulnerability. **(ULTIMATE)** + before merging code that could introduce a vulnerability. ## Related topics diff --git a/doc/user/project/merge_requests/approvals/rules.md b/doc/user/project/merge_requests/approvals/rules.md index 129010010e7..fa447ea35af 100644 --- a/doc/user/project/merge_requests/approvals/rules.md +++ b/doc/user/project/merge_requests/approvals/rules.md @@ -76,9 +76,9 @@ To edit a merge request approval rule: select **{remove}** **Remove**. 1. Select **Update approval rule**. -## Add multiple approval rules **(PREMIUM)** +## Add multiple approval rules -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1979) in GitLab Premium 11.10. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1979) in GitLab 11.10. In GitLab Premium and higher tiers, you can enforce multiple approval rules on a merge request, and multiple default approval rules for a project. If your tier @@ -127,8 +127,8 @@ users were not explicitly listed in the approval rules. ### Group approvers You can add a group of users as approvers, but those users count as approvers only if -they have direct membership to the group. In the future, group approvers may be -restricted to only groups [with share access to the project](https://gitlab.com/gitlab-org/gitlab/-/issues/2048). +they have direct membership to the group. Group approvers are +restricted to only groups [with share access to the project](../../members/share_project_with_groups.md). A user's membership in an approvers group affects their individual ability to approve in these ways: @@ -143,7 +143,7 @@ approve in these ways: [**Prevent committers approval**](settings.md#prevent-approvals-by-users-who-add-commits) project setting. -### Code owners as eligible approvers **(PREMIUM)** +### Code owners as eligible approvers > Moved to GitLab Premium in 13.9. @@ -158,14 +158,14 @@ become eligible approvers in the project. To enable this merge request approval You can also [require code owner approval](../../protected_branches.md#require-code-owner-approval-on-a-protected-branch) -for protected branches. **(PREMIUM)** +for protected branches. -## Merge request approval segregation of duties **(PREMIUM)** +## Merge request approval segregation of duties > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/40491) in GitLab 13.4. > - Moved to GitLab Premium in 13.9. -You may have to grant users with the Reporter [role](../../../permissions.md#project-members-permissions) +You may have to grant users with the Reporter role permission to approve merge requests before they can merge to a protected branch. Some users (like managers) may not need permission to push or merge code, but still need oversight on proposed work. To enable approval permissions for these users without @@ -202,7 +202,7 @@ on a merge request, you can either add or remove approvers: Administrators can change the [merge request approvals settings](settings.md#prevent-editing-approval-rules-in-merge-requests) to prevent users from overriding approval rules for merge requests. -## Configure optional approval rules **(PREMIUM)** +## Configure optional approval rules Merge request approvals can be optional for projects where approvals are appreciated, but not required. To make an approval rule optional: @@ -211,9 +211,9 @@ appreciated, but not required. To make an approval rule optional: - Use the [Merge requests approvals API](../../../../api/merge_request_approvals.md#update-merge-request-level-rule) to set the `approvals_required` attribute to `0`. -## Approvals for protected branches **(PREMIUM)** +## Approvals for protected branches -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/460) in GitLab Premium 12.8. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/460) in GitLab 12.8. Approval rules are often relevant only to specific branches, like your [default branch](../../repository/branches/default.md). To configure an @@ -229,3 +229,16 @@ approval rule for certain branches:  1. To enable this configuration, read [Code Owner's approvals for protected branches](../../protected_branches.md#require-code-owner-approval-on-a-protected-branch). + +## Troubleshooting + +### Approval rule name can't be blank + +As a workaround for this validation error, you can delete the approval rule through +the API. + +1. [GET a project-level rule](../../../../api/merge_request_approvals.md#get-a-single-project-level-rule). +1. [DELETE the rule](../../../../api/merge_request_approvals.md#delete-project-level-rule). + +For more information about this validation error, read +[issue 285129](https://gitlab.com/gitlab-org/gitlab/-/issues/285129). diff --git a/doc/user/project/merge_requests/authorization_for_merge_requests.md b/doc/user/project/merge_requests/authorization_for_merge_requests.md index 4ae59a76a9a..37ecc1b8d06 100644 --- a/doc/user/project/merge_requests/authorization_for_merge_requests.md +++ b/doc/user/project/merge_requests/authorization_for_merge_requests.md @@ -16,7 +16,7 @@ There are two main ways to have a merge request flow with GitLab: With the protected branch flow everybody works within the same GitLab project. -The project maintainers get the [Maintainer role](../../permissions.md) and the regular developers +The project maintainers get the Maintainer role and the regular developers get the Developer role. Maintainers mark the authoritative branches as 'Protected'. @@ -25,7 +25,7 @@ Developers push feature branches to the project and create merge requests to have their feature branches reviewed and merged into one of the protected branches. -By default, only users with the [Maintainer role](../../permissions.md) can merge changes into a +By default, only users with the Maintainer role can merge changes into a protected branch. **Advantages** @@ -39,7 +39,7 @@ protected branch. ## Forking workflow -With the forking workflow, maintainers get the [Maintainer role](../../permissions.md) and regular +With the forking workflow, maintainers get the Maintainer role and regular developers get the Reporter role on the authoritative repository, which prohibits them from pushing any changes to it. diff --git a/doc/user/project/merge_requests/browser_performance_testing.md b/doc/user/project/merge_requests/browser_performance_testing.md index 6668e1736cf..9c7d9e2bf19 100644 --- a/doc/user/project/merge_requests/browser_performance_testing.md +++ b/doc/user/project/merge_requests/browser_performance_testing.md @@ -1,13 +1,12 @@ --- stage: Verify -group: Testing +group: Pipeline Insights info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference, howto --- # Browser Performance Testing **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3507) in [GitLab Premium](https://about.gitlab.com/pricing/) 10.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3507) in GitLab 10.3. If your application offers a web interface and you're using [GitLab CI/CD](../../../ci/index.md), you can quickly determine the rendering performance diff --git a/doc/user/project/merge_requests/changes.md b/doc/user/project/merge_requests/changes.md index d348c00bdc2..8796ea0635b 100644 --- a/doc/user/project/merge_requests/changes.md +++ b/doc/user/project/merge_requests/changes.md @@ -5,149 +5,151 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: index, reference --- -# Changes tab in merge requests **(FREE)** +# Changes in merge requests **(FREE)** -The **Changes** tab on a [merge request](index.md), below the main merge request details and next to the discussion tab, -shows the changes to files between branches or commits. This view of changes to a -file is also known as a **diff**. By default, the diff view compares the file in the -merge request branch and the file in the target branch. +A [merge request](index.md) proposes a set of changes to files in a branch in your repository. These +changes are shown as a _diff_ (difference) between the current state and the proposed +changes. -The diff view includes the following: +By default, the diff view compares the versions of files in the merge request source branch +to the files in the target branch, and shows only the parts of a file that have changed. -- The file's name and path. -- The number of lines added and deleted. -- Buttons for the following options: - - Toggle comments for this file; useful for inline reviews. - - Edit the file in the merge request's branch. - - Show full file, in case you want to look at the changes in context with the rest of the file. - - View file at the current commit. - - Preview the changes with [Review Apps](../../../ci/review_apps/index.md). -- The changed lines, with the specific changes highlighted. + - +## Show all changes in a merge request -## Merge request diff file navigation +To view the diff of changes included in a merge request: -When reviewing changes in the **Changes** tab, the diff can be navigated using -the file tree or file list. As you scroll through large diffs with many -changes, you can quickly jump to any changed file using the file tree or file -list. +1. Go to your merge request. +1. Below the merge request title, select **Changes**. +1. If the merge request changes many files, you can jump directly to a specific file: + 1. Select **Show file browser** (**{file-tree}**) to display the file tree. + 1. Select the file you want to view. + 1. To hide the file browser, select **Show file browser** again. - +In [GitLab 13.4](https://gitlab.com/gitlab-org/gitlab/-/issues/232820) and later, files +with many changes are collapsed to improve performance. GitLab displays the message: +**Some changes are not shown**. To view the changes for that file, select **Expand file**. -## Collapsed files in the Changes view +## Show one file at a time -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/232820) in GitLab 13.4. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/222790) in GitLab 13.2. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/229848) in GitLab 13.7. -When you review changes in the **Changes** tab, files with a large number of changes are collapsed -to improve performance. When files are collapsed, a warning appears at the top of the changes. -Select **Expand file** on any file to view the changes for that file. +For larger merge requests, you can review one file at a time. You can change this setting +[temporarily in a merge request](#in-a-merge-request-show-only-one-file-at-a-time), or +so it [applies to all merge requests](#in-all-merge-requests-show-only-one-file-at-a-time). -## File-by-file diff navigation +### In a merge request, show only one file at a time -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/222790) in GitLab 13.2. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/229848) in GitLab 13.7. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/233898) in GitLab 13.7. + +To temporarily change your viewing preferences for a specific merge request: + +1. Go to your merge request, and below the merge request title, select **Changes**. +1. Select **Preferences** (**{settings}**). +1. Select or clear the **Show one file at a time** checkbox. -For larger merge requests, consider reviewing one file at a time. To enable this feature: +This change overrides your choice in your user preferences. It persists until you +clear your browser's cookies or change this behavior again. + +### In all merge requests, show only one file at a time + +To view one file at a time for all of your merge requests: 1. In the top-right corner, select your avatar. 1. Select **Preferences**. -1. Scroll to the **Behavior** section and select **Show one file at a time on merge request's Changes tab**. +1. Scroll to the **Behavior** section and select the **Show one file at a time on merge request's Changes tab** checkbox. 1. Select **Save changes**. -After you enable this setting, GitLab displays only one file at a time in the **Changes** tab when you review merge requests. You can select **Prev** and **Next** to view other changed files. +After you enable this setting, GitLab displays only one file at a time when you review +merge requests. To view other changed files, either: + +- Scroll to the end of the file and select either **Prev** or **Next**. +- Select **Show file browser** (**{file-tree}**) and select another file to view. + +## Compare changes inline + +You can view the changes inline: - +1. Go to your merge request, and below the title, select **Changes**. +1. Select **Preferences** (**{settings}**). +1. In the **Compare changes** area, select **Inline**. -In [GitLab 13.7](https://gitlab.com/gitlab-org/gitlab/-/issues/233898) and later, if you want to change -this behavior, you can do so from your **User preferences** (as explained above) or directly in a -merge request: +The changes are displayed after the original text. -1. Go to the merge request's **Changes** tab. -1. Select the cog icon (**{settings}**) to reveal the merge request's settings dropdown. -1. Select or clear the checkbox **Show one file at a time** to change the setting accordingly. + -This change overrides the choice you made in your user preferences and persists until you clear your -browser's cookies or change this behavior again. +## Compare changes side-by-side -## Incrementally expand merge request diffs +Depending on the length of the changes in your merge request, you may find it +easier to view the changes inline, or side-by-side: -By default, the diff shows only the parts of a file which are changed. -To view more unchanged lines above or below a change select the -**Expand up** or **Expand down** icons. You can also select **Show unchanged lines** -to expand the entire file. +1. Go to your merge request, and below the title, select **Changes**. +1. Select **Preferences** (**{settings}**). +1. In the **Compare changes** area, select **Side-by-side**. - +The changes are displayed across from one another. -In GitLab [versions 13.1 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/205401), when viewing a -merge request's **Changes** tab, if a certain file was only renamed, you can expand it to see the -entire content by selecting **Show file contents**. + -## Ignore whitespace changes in Merge Request diff view +## Expand or collapse comments + +When reviewing code changes, you can hide inline comments: + +1. Go to your merge request, and below the title, select **Changes**. +1. Scroll to the file that contains the comments you want to hide. +1. Scroll to the line the comment is attached to, and select **Collapse** (**{collapse}**): +  + +To expand inline comments and show them again: + +1. Go to your merge request, and below the title, select **Changes**. +1. Scroll to the file that contains the collapsed comments you want to show. +1. Scroll to the line the comment is attached to, and select the user avatar: +  + +## Ignore whitespace changes Whitespace changes can make it more difficult to see the substantive changes in a merge request. You can choose to hide or show whitespace changes: -1. Go to your merge request, and select the **Changes** tab. -1. Above the list of changed files, select **(settings)** **Preferences** to - display the preferences menu. -1. Select or deselect **Show whitespace changes**: +1. Go to your merge request, and below the title, select **Changes**. +1. Before the list of changed files, select **Preferences** (**{settings}**). +1. Select or clear the **Show whitespace changes** checkbox:  ## Mark files as viewed -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/51513) in GitLab 13.9 behind a feature flag, enabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/51513) in GitLab 13.9 [with a flag](../../../administration/feature_flags.md) named `local_file_reviews`. Enabled by default. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/296674) in GitLab 14.3. -When reviewing a merge request with many files multiple times, it may be useful to the reviewer -to focus on new changes and ignore the files that they have already reviewed and don't want to -see anymore unless they are changed again. +When reviewing a merge request with many files multiple times, you can ignore files +you've already reviewed. To hide files that haven't changed since your last review: -To mark a file as viewed: +1. Go to your merge request, and below the title, select **Changes**. +1. In the file's header, select the **Viewed** checkbox. -1. Go to the merge request's **Diffs** tab. -1. On the right-top of the file, locate the **Viewed** checkbox. -1. Select it to mark the file as viewed. +Files marked as viewed are not shown to you again unless either: -Once checked, the file remains marked for that reviewer unless there are newly introduced -changes to its content or the checkbox is unchecked. +- New changes are made to its content. +- You clear the **Viewed** checkbox. -## Show merge request conflicts in diff +## Show merge request conflicts in diff **(FREE SELF)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/232484) in GitLab 13.5. -> - [Deployed behind a feature flag](../../feature_flags.md), disabled by default. -> - Disabled on GitLab.com. -> - Not recommended for production use. -> - To use in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-merge-request-conflicts-in-diff). **(FREE SELF)** +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/232484) in GitLab 13.5 [with a flag](../../../administration/feature_flags.md) named `display_merge_conflicts_in_diff`. Disabled by default. -This in-development feature might not be available for your use. There can be -[risks when enabling features still in development](../../../administration/feature_flags.md#risks-when-enabling-features-still-in-development). -Refer to this feature's version history for more details. +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, +ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) +named `display_merge_conflicts_in_diff`. On GitLab.com, this feature is not available. +The feature is not ready for production use. To avoid displaying the changes that are already on target branch in the diff, we compare the merge request's source branch with HEAD of the target branch. When there are conflicts between the source and target branch, we show the -conflicts on the merge request diff as well: +conflicts on the merge request diff:  - -### Enable or disable merge request conflicts in diff **(FREE SELF)** - -Merge request conflicts in diff is under development and not ready for production use. It is -deployed behind a feature flag that is **disabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) -can enable it. - -To enable it: - -```ruby -Feature.enable(:display_merge_conflicts_in_diff) -``` - -To disable it: - -```ruby -Feature.disable(:display_merge_conflicts_in_diff) -``` diff --git a/doc/user/project/merge_requests/code_quality.md b/doc/user/project/merge_requests/code_quality.md index 30d463efa69..d735ce0ef91 100644 --- a/doc/user/project/merge_requests/code_quality.md +++ b/doc/user/project/merge_requests/code_quality.md @@ -53,7 +53,7 @@ See also the Code Climate list of [Supported Languages for Maintainability](http ## Code Quality in diff view **(ULTIMATE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267612) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.11, disabled by default behind the `codequality_mr_diff` [feature flag](../../../administration/feature_flags.md). +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267612) in GitLab 13.11, disabled by default behind the `codequality_mr_diff` [feature flag](../../../administration/feature_flags.md). > - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/284140) in GitLab 13.12. > - [Disabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/2526) in GitLab 14.0 due to [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/334116). > - [Inline annotation added](https://gitlab.com/gitlab-org/gitlab/-/issues/2526) and [feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/284140) in GitLab 14.1. @@ -248,9 +248,9 @@ This can be done: ### Using with merge request pipelines The configuration provided by the Code Quality template does not let the `code_quality` job -run on [pipelines for merge requests](../../../ci/pipelines/merge_request_pipelines.md). +run on [merge request pipelines](../../../ci/pipelines/merge_request_pipelines.md). -If pipelines for merge requests is enabled, the `code_quality:rules` must be redefined. +If merge request pipelines is enabled, the `code_quality:rules` must be redefined. The template has these [`rules`](../../../ci/yaml/index.md#rules) for the `code quality` job: @@ -379,7 +379,7 @@ at the beginning of the file. ## Code Quality reports **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/21527) in GitLab Premium 12.9. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/21527) in GitLab 12.9.  @@ -392,7 +392,7 @@ After the Code Quality job completes: [downloadable artifact](../../../ci/pipelines/job_artifacts.md#download-job-artifacts) for the `code_quality` job. - The full list of code quality violations generated by a pipeline is shown in the - Code Quality tab of the Pipeline Details page. **(PREMIUM)** + Code Quality tab of the Pipeline Details page. ## Generate an HTML report @@ -591,3 +591,22 @@ plugins: If your merge requests do not show any code quality changes when using a custom tool, ensure that the line property is an `integer`. + +### Code Quality CI job with Code Climate plugins enabled fails with error "engine <plugin_name> ran for 900 seconds and was killed" + +If you enabled any of the Code Climate plugins, and the Code Quality CI job fails with the error below, +it's likely the job takes longer than the default timeout of 900 seconds. + +```shell +error: (CC::CLI::Analyze::EngineFailure) engine pmd ran for 900 seconds and was killed +Could not analyze code quality for the repository at /code +``` + +To work around this problem, set `TIMEOUT_SECONDS` to a higher value in your `.gitlab.-ci.yml` file. + +For example: + +```yaml +variables: + TIMEOUT_SECONDS: 3600 +``` diff --git a/doc/user/project/merge_requests/commits.md b/doc/user/project/merge_requests/commits.md index 3d3032bb193..0014c1ba994 100644 --- a/doc/user/project/merge_requests/commits.md +++ b/doc/user/project/merge_requests/commits.md @@ -29,21 +29,17 @@ To seamlessly navigate among commits in a merge request: ## View merge request commits in context -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/29274) in GitLab 13.12. -> - [Deployed behind a feature flag](../../feature_flags.md), enabled by default. -> - Disabled on GitLab.com. -> - Not recommended for production use. -> - To use in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-viewing-merge-request-commits-in-context). **(FREE SELF)** +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/29274) in GitLab 13.12 [with a flag](../../../administration/feature_flags.md) named `context_commits`. Enabled by default. +> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/320757) in GitLab 14.8. WARNING: -This feature is in [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta) +This feature is in [beta](../../../policy/alpha-beta-support.md#beta-features) and is [incomplete](https://gitlab.com/groups/gitlab-org/-/epics/1192). -Previously merged commits can be added, but they can't be removed due to -[this bug](https://gitlab.com/gitlab-org/gitlab/-/issues/325538). -This in-development feature might not be available for your use. There can be -[risks when enabling features still in development](../../../administration/feature_flags.md#risks-when-enabling-features-still-in-development). -Refer to this feature's version history for more details. +FLAG: +On self-managed GitLab, by default this feature is available. To hide the feature, +ask an administrator to [disable the feature flag](../../../administration/feature_flags.md) named `context_commits`. +On GitLab.com, this feature is available. When reviewing a merge request, it helps to have more context about the changes made. That includes unchanged lines in unchanged files, and previous commits @@ -66,22 +62,3 @@ To view the changes done on those previously merged commits: 1. Scroll to **(file-tree)** **Compare** and select **previously merged commits**:  - -### Enable or disable viewing merge request commits in context **(FREE SELF)** - -Viewing merge request commits in context is under development and not ready for production use. It is -deployed behind a feature flag that is **disabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) -can enable it. - -To enable it: - -```ruby -Feature.enable(:context_commits) -``` - -To disable it: - -```ruby -Feature.disable(:context_commits) -``` diff --git a/doc/user/project/merge_requests/confidential.md b/doc/user/project/merge_requests/confidential.md index 10c63421876..6900880417f 100644 --- a/doc/user/project/merge_requests/confidential.md +++ b/doc/user/project/merge_requests/confidential.md @@ -4,7 +4,7 @@ group: Code Review info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Merge requests for confidential issues +# Merge requests for confidential issues **(FREE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/58583) in GitLab 12.1. diff --git a/doc/user/project/merge_requests/drafts.md b/doc/user/project/merge_requests/drafts.md index a91679ffd63..637b682d603 100644 --- a/doc/user/project/merge_requests/drafts.md +++ b/doc/user/project/merge_requests/drafts.md @@ -17,7 +17,8 @@ the **Merge** button until you remove the **Draft** flag: ## Mark merge requests as drafts -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32692) in GitLab 13.2, Work-In-Progress (WIP) merge requests were renamed to **Draft**. Support for using **WIP** is scheduled for removal in GitLab 14.0. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32692) in GitLab 13.2, Work-In-Progress (WIP) merge requests were renamed to **Draft**. +> - [Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/228685) all support for using **WIP** in GitLab 14.8. > - **Mark as draft** and **Mark as ready** buttons [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/227421) in GitLab 13.5. There are several ways to flag a merge request as a draft: @@ -35,16 +36,12 @@ There are several ways to flag a merge request as a draft: is not a toggle, and adding this text again in a later commit doesn't mark the merge request as ready. -WARNING: -Adding `WIP:` to the start of the merge request's title still marks a merge request -as a draft. This feature is scheduled for removal in GitLab 14.0. Use `Draft:` instead. - ## Mark merge requests as ready When a merge request is ready to be merged, you can remove the `Draft` flag in several ways: - **Viewing a merge request**: In the top right corner of the merge request, click **Mark as ready**. - Users with [Developer or greater permissions](../../permissions.md) + Users with at least the Developer role can also scroll to the bottom of the merge request description and click **Mark as ready**:  @@ -79,11 +76,11 @@ draft merge requests: ## Pipelines for drafts -When the [pipelines for merged results](../../../ci/pipelines/pipelines_for_merged_results.md) +When the [merged results pipelines](../../../ci/pipelines/merged_results_pipelines.md) feature is enabled, draft merge requests run [merge request pipelines](../../../ci/pipelines/merge_request_pipelines.md) only. -To run pipelines for merged results, you must +To run merged results pipelines, you must [mark the merge request as ready](#mark-merge-requests-as-ready). <!-- ## Troubleshooting diff --git a/doc/user/project/merge_requests/fail_fast_testing.md b/doc/user/project/merge_requests/fail_fast_testing.md index 3cb50195f5a..355661516a7 100644 --- a/doc/user/project/merge_requests/fail_fast_testing.md +++ b/doc/user/project/merge_requests/fail_fast_testing.md @@ -1,8 +1,7 @@ --- stage: Verify -group: Testing +group: Pipeline Insights info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference, howto --- # Fail Fast Testing **(PREMIUM)** @@ -42,8 +41,8 @@ This template requires: - A project built in Rails that uses RSpec for testing. - CI/CD configured to: - Use a Docker image with Ruby available. - - Use [Pipelines for merge requests](../../../ci/pipelines/merge_request_pipelines.md#prerequisites) -- [Pipelines for Merged Results](../../../ci/pipelines/pipelines_for_merged_results.md#enable-pipelines-for-merged-results) + - Use [Merge request pipelines](../../../ci/pipelines/merge_request_pipelines.md#prerequisites) +- [Merged results pipelines](../../../ci/pipelines/merged_results_pipelines.md#enable-merged-results-pipelines) enabled in the project settings. - A Docker image with Ruby available. The template uses `image: ruby:2.6` by default, but you [can override](../../../ci/yaml/includes.md#override-included-configuration-values) this. diff --git a/doc/user/project/merge_requests/getting_started.md b/doc/user/project/merge_requests/getting_started.md index ec509f58723..8699a42dd5d 100644 --- a/doc/user/project/merge_requests/getting_started.md +++ b/doc/user/project/merge_requests/getting_started.md @@ -56,7 +56,7 @@ request's page at the top-right side, or by using - [Assign](#assignee) the merge request to a colleague for review. With [multiple assignees](#multiple-assignees), you can assign it to more than one person at a time. - Set a [milestone](../milestones/index.md) to track time-sensitive changes. - Add [labels](../labels.md) to help contextualize and filter your merge requests over time. -- [Require approval](approvals/index.md#required-approvals) from your team. **(PREMIUM)** +- [Require approval](approvals/index.md#required-approvals) from your team. - [Close issues automatically](#merge-requests-to-close-issues) when they are merged. - Enable the [delete source branch when merge request is accepted](#deleting-the-source-branch) option to keep your repository clean. - Enable the [squash commits when merge request is accepted](squash_and_merge.md) option to combine all the commits into one before merging, thus keep a clean commit history in your repository. @@ -66,7 +66,7 @@ After you have created the merge request, you can also: - [Discuss](../../discussions/index.md) your implementation with your team in the merge request thread. - [Perform inline code reviews](reviews/index.md). -- Add [merge request dependencies](merge_request_dependencies.md) to restrict it to be merged only when other merge requests have been merged. **(PREMIUM)** +- Add [merge request dependencies](merge_request_dependencies.md) to restrict it to be merged only when other merge requests have been merged. - Preview continuous integration [pipelines on the merge request widget](widgets.md). - Preview how your changes look directly on your deployed application with [Review Apps](widgets.md#live-preview-with-review-apps). - [Allow collaboration on merge requests across forks](allow_collaboration.md). @@ -162,7 +162,7 @@ enabled by default for all new merge requests, enable it in the This option is also visible in an existing merge request next to the merge request button and can be selected or deselected before merging. -It is only visible to users with the [Maintainer role](../../permissions.md) +It is only visible to users with the Maintainer role in the source project. If the user viewing the merge request does not have the correct diff --git a/doc/user/project/merge_requests/img/changes-inline_v14_8.png b/doc/user/project/merge_requests/img/changes-inline_v14_8.png Binary files differnew file mode 100644 index 00000000000..f3b6515283c --- /dev/null +++ b/doc/user/project/merge_requests/img/changes-inline_v14_8.png diff --git a/doc/user/project/merge_requests/img/changes-sidebyside_v14_8.png b/doc/user/project/merge_requests/img/changes-sidebyside_v14_8.png Binary files differnew file mode 100644 index 00000000000..981274dbb86 --- /dev/null +++ b/doc/user/project/merge_requests/img/changes-sidebyside_v14_8.png diff --git a/doc/user/project/merge_requests/img/collapse-comment_v14_8.png b/doc/user/project/merge_requests/img/collapse-comment_v14_8.png Binary files differnew file mode 100644 index 00000000000..93a802454f2 --- /dev/null +++ b/doc/user/project/merge_requests/img/collapse-comment_v14_8.png diff --git a/doc/user/project/merge_requests/img/expand-comment_v14_8.png b/doc/user/project/merge_requests/img/expand-comment_v14_8.png Binary files differnew file mode 100644 index 00000000000..a800eab83e9 --- /dev/null +++ b/doc/user/project/merge_requests/img/expand-comment_v14_8.png diff --git a/doc/user/project/merge_requests/img/file_by_file_v13_2.png b/doc/user/project/merge_requests/img/file_by_file_v13_2.png Binary files differdeleted file mode 100644 index e3114ebabad..00000000000 --- a/doc/user/project/merge_requests/img/file_by_file_v13_2.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/incrementally_expand_merge_request_diffs_v12_2.png b/doc/user/project/merge_requests/img/incrementally_expand_merge_request_diffs_v12_2.png Binary files differdeleted file mode 100644 index e3a2ff7960c..00000000000 --- a/doc/user/project/merge_requests/img/incrementally_expand_merge_request_diffs_v12_2.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/merge_request_diff_file_navigation.png b/doc/user/project/merge_requests/img/merge_request_diff_file_navigation.png Binary files differdeleted file mode 100644 index 1cdac5ef573..00000000000 --- a/doc/user/project/merge_requests/img/merge_request_diff_file_navigation.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/mr-diff-example_v14_8.png b/doc/user/project/merge_requests/img/mr-diff-example_v14_8.png Binary files differnew file mode 100644 index 00000000000..1984defde9a --- /dev/null +++ b/doc/user/project/merge_requests/img/mr-diff-example_v14_8.png diff --git a/doc/user/project/merge_requests/index.md b/doc/user/project/merge_requests/index.md index 8222d696853..9872bd2e936 100644 --- a/doc/user/project/merge_requests/index.md +++ b/doc/user/project/merge_requests/index.md @@ -94,7 +94,7 @@ You cannot undo the deletion of a merge request. To delete a merge request: -1. Sign in to GitLab as a user with the project Owner [role](../../permissions.md). +1. Sign in to GitLab as a user with the project Owner role. Only users with this role can delete merge requests in a project. 1. Go to the merge request you want to delete, and select **Edit**. 1. Scroll to the bottom of the page, and select **Delete merge request**. @@ -137,3 +137,4 @@ For a web developer writing a webpage for your company's website: - [Suggest code changes](reviews/suggestions.md) - [Commits](commits.md) - [CI/CD pipelines](../../../ci/index.md) +- [Push options](../push_options.md) for merge requests diff --git a/doc/user/project/merge_requests/load_performance_testing.md b/doc/user/project/merge_requests/load_performance_testing.md index 40859c6b572..cc4ad186771 100644 --- a/doc/user/project/merge_requests/load_performance_testing.md +++ b/doc/user/project/merge_requests/load_performance_testing.md @@ -1,13 +1,12 @@ --- stage: Verify -group: Testing +group: Pipeline Insights info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference, howto --- # Load Performance Testing **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10683) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10683) in GitLab 13.2. With Load Performance Testing, you can test the impact of any pending code changes to your application's backend in [GitLab CI/CD](../../../ci/index.md). diff --git a/doc/user/project/merge_requests/merge_request_dependencies.md b/doc/user/project/merge_requests/merge_request_dependencies.md index aace1f58c62..6bfef6ab134 100644 --- a/doc/user/project/merge_requests/merge_request_dependencies.md +++ b/doc/user/project/merge_requests/merge_request_dependencies.md @@ -7,9 +7,9 @@ type: reference, concepts # Merge request dependencies **(PREMIUM)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9688) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.2. -> - [Renamed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17291) from "Cross-project dependencies" to "Merge request dependencies" in [GitLab Premium](https://about.gitlab.com/pricing/) 12.4. -> - Intra-project MR dependencies were [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/16799) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.4. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9688) in GitLab 12.2. +> - [Renamed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17291) from "Cross-project dependencies" to "Merge request dependencies" in GitLab 12.4. +> - Intra-project MR dependencies were [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/16799) in GitLab 12.4. Merge request dependencies allows a required order of merging between merge requests to be expressed. If a merge request "depends on" another, diff --git a/doc/user/project/merge_requests/reviews/index.md b/doc/user/project/merge_requests/reviews/index.md index c34a8116625..280ae07b401 100644 --- a/doc/user/project/merge_requests/reviews/index.md +++ b/doc/user/project/merge_requests/reviews/index.md @@ -23,8 +23,8 @@ review merge requests in Visual Studio Code. ## Review a merge request -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/4213) in GitLab Premium 11.4. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/28154) to GitLab Free in 13.1. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/4213) in GitLab 11.4. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/28154) from GitLab Premium to GitLab Free in 13.1. When you review a merge request, you can create comments that are visible only to you. When you're ready, you can publish them together in a single action. @@ -160,7 +160,7 @@ Multiline comments display the comment's line numbers above the body of the comm ## Bulk edit merge requests at the project level -Users with permission level of [Developer or higher](../../../permissions.md) can manage merge requests. +Users with at least the Developer role can manage merge requests. When bulk-editing merge requests in a project, you can edit the following attributes: @@ -179,11 +179,11 @@ To update multiple project merge requests at the same time: 1. Select the appropriate fields and their values from the sidebar. 1. Click **Update all**. -## Bulk edit merge requests at the group level +## Bulk edit merge requests at the group level **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12719) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.2. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12719) in GitLab 12.2. -Users with permission level of [Developer or higher](../../../permissions.md) can manage merge requests. +Users with at least the Developer role can manage merge requests. When bulk editing merge requests in a group, you can edit the following attributes: diff --git a/doc/user/project/merge_requests/reviews/suggestions.md b/doc/user/project/merge_requests/reviews/suggestions.md index 1b2a35ba139..9868f2619ba 100644 --- a/doc/user/project/merge_requests/reviews/suggestions.md +++ b/doc/user/project/merge_requests/reviews/suggestions.md @@ -42,7 +42,7 @@ After the author applies a suggestion: 1. The suggestion is marked as **Applied**. 1. The thread is resolved. 1. GitLab creates a new commit with the changes. -1. If the user has the [Developer role](../../../permissions.md), GitLab pushes +1. If the user has the Developer role, GitLab pushes the suggested change directly into the codebase in the merge request's branch. ## Multi-line suggestions @@ -114,7 +114,7 @@ introduced by [#25381](https://gitlab.com/gitlab-org/gitlab/-/issues/25381). ## Batch suggestions -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25486) in GitLab 13.1 as an [alpha feature](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha) with a flag named `batch_suggestions`, disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25486) in GitLab 13.1 as an [alpha feature](../../../../policy/alpha-beta-support.md#alpha-features) with a flag named `batch_suggestions`, disabled by default. > - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/227799) in GitLab 13.2. > - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/320755) in GitLab 13.11. [Feature flag `batch_suggestions`](https://gitlab.com/gitlab-org/gitlab/-/issues/320755) removed. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/326168) custom commit messages for batch suggestions in GitLab 14.4. diff --git a/doc/user/project/merge_requests/status_checks.md b/doc/user/project/merge_requests/status_checks.md index f5ebfb3668c..a952c0550bc 100644 --- a/doc/user/project/merge_requests/status_checks.md +++ b/doc/user/project/merge_requests/status_checks.md @@ -28,6 +28,32 @@ You can configure merge request status checks for each individual project. These To learn more about use cases, feature discovery, and development timelines, see the [external status checks epic](https://gitlab.com/groups/gitlab-org/-/epics/3869). +## Lifecycle + +External status checks have an **asynchronous** workflow. Merge requests emit a merge request webhook payload to an external service whenever: + +- The merge request is changed. For example, title or description. +- Code is pushed to the source branch of the merge request. +- A comment is made in the merge request discussion. + +```mermaid +sequenceDiagram + Merge request->>+External service: Merge request payload + External service-->>-Merge request: Status check response + Note over External service,Merge request: Response includes SHA at HEAD +``` + +When the payload is received, the external service can then run any required processes before posting its response back to the merge request [using the REST API](../../../api/status_checks.md#set-status-of-an-external-status-check). + +Merge requests return a `409 Conflict` error to any responses that do not refer to the current `HEAD` of the source branch. As a result, it's safe for the external service to process and respond to out-of-date commits. + +External status checks have the following states: + +- `pending` - The default state. No response can been received by the merge request from the external service. +- `response received` - A response from the external service has been received and approved by it. + +Support for adding a `failed` state is tracked [in this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/338827). + ## View the status checks on a project Within each project's settings, you can see a list of status checks added to the project: diff --git a/doc/user/project/merge_requests/test_coverage_visualization.md b/doc/user/project/merge_requests/test_coverage_visualization.md index 1f84964c619..bd54eda42f5 100644 --- a/doc/user/project/merge_requests/test_coverage_visualization.md +++ b/doc/user/project/merge_requests/test_coverage_visualization.md @@ -1,8 +1,7 @@ --- stage: Verify -group: Testing +group: Pipeline Insights info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference, howto --- # Test coverage visualization **(FREE)** @@ -40,6 +39,7 @@ Other coverage analysis frameworks support the format out of the box, for exampl - [Istanbul](https://istanbul.js.org/docs/advanced/alternative-reporters/#cobertura) (JavaScript) - [Coverage.py](https://coverage.readthedocs.io/en/coverage-5.0.4/cmd.html#xml-reporting) (Python) +- [PHPUnit](https://github.com/sebastianbergmann/phpunit-documentation-english/blob/master/src/textui.rst#command-line-options) (PHP) Once configured, if you create a merge request that triggers a pipeline which collects coverage reports, the coverage is shown in the diff view. This includes reports @@ -52,6 +52,8 @@ from any job in any stage in the pipeline. The coverage displays for each line: Hovering over the coverage bar provides further information, such as the number of times the line was checked by tests. +Uploading a test coverage report does not enable [test coverage results in Merge Requests](../../../ci/pipelines/settings.md#add-test-coverage-results-to-a-merge-request-deprecated) or [code coverage history](../../../ci/pipelines/settings.md#view-code-coverage-history). You must configure these separately. + ### Limits A limit of 100 `<source>` nodes for Cobertura format XML files applies. If your Cobertura report exceeds @@ -133,6 +135,9 @@ The `source` is ignored if the path does not follow this pattern. The parser ass ## Example test coverage configurations +This section provides test coverage configuration examples for different programming languages. You can also see a working example in +the [`coverage-report`](https://gitlab.com/gitlab-org/ci-sample-projects/coverage-report/) demonstration project. + ### JavaScript example The following [`.gitlab-ci.yml`](../../../ci/yaml/index.md) example uses [Mocha](https://mochajs.org/) @@ -236,7 +241,7 @@ run tests: image: python:3 script: - pip install pytest pytest-cov - - coverage run -m pytest + - coverage run -m pytest - coverage report - coverage xml artifacts: @@ -244,6 +249,42 @@ run tests: cobertura: coverage.xml ``` +### PHP example + +The following [`.gitlab-ci.yml`](../../../ci/yaml/index.md) example for PHP uses [PHPUnit](https://phpunit.readthedocs.io/) +to collect test coverage data and generate the report. + +With a minimal [`phpunit.xml`](https://phpunit.readthedocs.io/en/9.5/configuration.html) file (you may reference +[this example repository](https://gitlab.com/yookoala/code-coverage-visualization-with-php/)), you can run the test and +generate the coverage xml: + +```yaml +run tests: + stage: test + image: php:latest + variables: + XDEBUG_MODE: coverage + before_script: + - apt-get update && apt-get -yq install git unzip zip libzip-dev zlib1g-dev + - docker-php-ext-install zip + - pecl install xdebug && docker-php-ext-enable xdebug + - php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');" + - php composer-setup.php --install-dir=/usr/local/bin --filename=composer + - composer install + - composer require --dev phpunit/phpunit phpunit/php-code-coverage + script: + - php ./vendor/bin/phpunit --coverage-text --coverage-cobertura=coverage.cobertura.xml + artifacts: + reports: + cobertura: coverage.cobertura.xml +``` + +[Codeception](https://codeception.com/), through PHPUnit, also supports generating Cobertura report with +[`run`](https://codeception.com/docs/reference/Commands#run). The path for the generated file +depends on the `--coverage-cobertura` option and [`paths`](https://codeception.com/docs/reference/Configuration#paths) +configuration for the [unit test suite](https://codeception.com/docs/05-UnitTests). Configure `.gitlab-ci.yml` +to find Cobertura in the appropriate path. + ### C/C++ example The following [`.gitlab-ci.yml`](../../../ci/yaml/index.md) example for C/C++ with @@ -272,3 +313,62 @@ run tests: reports: cobertura: build/coverage.xml ``` + +### Go example + +The following [`.gitlab-ci.yml`](../../../ci/yaml/index.md) example for Go uses: + +- [`go test`](https://go.dev/doc/tutorial/add-a-test) to run tests. +- [`gocover-cobertura`](https://github.com/t-yuki/gocover-cobertura) to convert Go's coverage profile into the Cobertura XML format. + +This example assumes that [Go modules](https://go.dev/ref/mod) are being used. +Using Go modules causes paths within the coverage profile to be prefixed with your +project's module identifier, which can be found in the `go.mod` file. This +prefix must be removed for GitLab to parse the Cobertura XML file correctly. You can use the following `sed` command to remove the prefix: + +```shell +sed -i 's;filename=\"<YOUR_MODULE_ID>/;filename=\";g' coverage.xml +``` + +Replace the `gitlab.com/my-group/my-project` placeholder in the following example with your own module identifier to make it work. + +```yaml +run tests: + stage: test + image: golang:1.17 + script: + - go install + - go test . -coverprofile=coverage.txt -covermode count + - go run github.com/t-yuki/gocover-cobertura < coverage.txt > coverage.xml + - sed -i 's;filename=\"gitlab.com/my-group/my-project/;filename=\";g' coverage.xml + artifacts: + reports: + cobertura: coverage.xml +``` + +### Ruby example + +The following [`.gitlab-ci.yml`](../../../ci/yaml/index.md) example for Ruby uses + +- [`rspec`](https://rspec.info/) to run tests. +- [`simplecov`](https://github.com/simplecov-ruby/simplecov) and [`simplecov-cobertura`](https://github.com/dashingrocket/simplecov-cobertura) + to record the coverage profile and create a report in the Cobertura XML format. + +This example assumes: + +- That [`bundler`](https://bundler.io/) is being used for dependency management. + The `rspec`, `simplecov` and `simplecov-cobertura` gems have been added to your `Gemfile`. +- The `CoberturaFormatter` has been added to your `SimpleCov.formatters` + configuration within the `spec_helper.rb` file. + +```yaml +run tests: + stage: test + image: ruby:3.1 + script: + - bundle install + - bundle exec rspec + artifacts: + reports: + cobertura: coverage/coverage.xml +``` diff --git a/doc/user/project/merge_requests/testing_and_reports_in_merge_requests.md b/doc/user/project/merge_requests/testing_and_reports_in_merge_requests.md index 0a9a2a37bfe..741ac325cca 100644 --- a/doc/user/project/merge_requests/testing_and_reports_in_merge_requests.md +++ b/doc/user/project/merge_requests/testing_and_reports_in_merge_requests.md @@ -1,8 +1,7 @@ --- stage: Verify -group: Testing +group: Pipeline Insights info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: index description: "Test your code and display reports in merge requests" --- @@ -11,21 +10,21 @@ description: "Test your code and display reports in merge requests" GitLab has the ability to test the changes included in a feature branch and display reports or link to useful information directly from merge requests: -| Feature | Description | -|--------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [Accessibility Testing](accessibility_testing.md) | Automatically report A11y violations for changed pages in merge requests. | -| [Browser Performance Testing](browser_performance_testing.md) **(PREMIUM)** | Quickly determine the browser performance impact of pending code changes. | -| [Load Performance Testing](load_performance_testing.md) **(PREMIUM)** | Quickly determine the server performance impact of pending code changes. | -| [Code Quality](code_quality.md) | Analyze your source code quality using the [Code Climate](https://codeclimate.com/) analyzer and show the Code Climate report right in the merge request widget area. | -| [Display arbitrary job artifacts](../../../ci/yaml/index.md#artifactsexpose_as) | Configure CI pipelines with the `artifacts:expose_as` parameter to directly link to selected [artifacts](../../../ci/pipelines/job_artifacts.md) in merge requests. | -| [GitLab CI/CD](../../../ci/index.md) | Build, test, and deploy your code in a per-branch basis with built-in CI/CD. | -| [Unit test reports](../../../ci/unit_test_reports.md) | Configure your CI jobs to use Unit test reports, and let GitLab display a report on the merge request so that it's easier and faster to identify the failure without having to check the entire job log. | -| [License Compliance](../../compliance/license_compliance/index.md) **(ULTIMATE)** | Manage the licenses of your dependencies. | -| [Metrics Reports](../../../ci/metrics_reports.md) **(PREMIUM)** | Display the Metrics Report on the merge request so that it's fast and easy to identify changes to important metrics. | -| [Multi-Project pipelines](../../../ci/pipelines/multi_project_pipelines.md) **(PREMIUM)** | When you set up GitLab CI/CD across multiple projects, you can visualize the entire pipeline, including all cross-project interdependencies. | -| [Pipelines for merge requests](../../../ci/pipelines/merge_request_pipelines.md) | Customize a specific pipeline structure for merge requests in order to speed the cycle up by running only important jobs. | -| [Pipeline Graphs](../../../ci/pipelines/index.md#visualize-pipelines) | View the status of pipelines within the merge request, including the deployment process. | -| [Test Coverage visualization](test_coverage_visualization.md) | See test coverage results for merge requests, within the file diff. | +| Feature | Description | +|----------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| [Accessibility Testing](accessibility_testing.md) | Automatically report A11y violations for changed pages in merge requests. | +| [Browser Performance Testing](browser_performance_testing.md) | Quickly determine the browser performance impact of pending code changes. | +| [Load Performance Testing](load_performance_testing.md) | Quickly determine the server performance impact of pending code changes. | +| [Code Quality](code_quality.md) | Analyze your source code quality using the [Code Climate](https://codeclimate.com/) analyzer and show the Code Climate report right in the merge request widget area. | +| [Display arbitrary job artifacts](../../../ci/yaml/index.md#artifactsexpose_as) | Configure CI pipelines with the `artifacts:expose_as` parameter to directly link to selected [artifacts](../../../ci/pipelines/job_artifacts.md) in merge requests. | +| [GitLab CI/CD](../../../ci/index.md) | Build, test, and deploy your code in a per-branch basis with built-in CI/CD. | +| [Unit test reports](../../../ci/unit_test_reports.md) | Configure your CI jobs to use Unit test reports, and let GitLab display a report on the merge request so that it's easier and faster to identify the failure without having to check the entire job log. | +| [License Compliance](../../compliance/license_compliance/index.md) | Manage the licenses of your dependencies. | +| [Metrics Reports](../../../ci/metrics_reports.md) | Display the Metrics Report on the merge request so that it's fast and easy to identify changes to important metrics. | +| [Multi-Project pipelines](../../../ci/pipelines/multi_project_pipelines.md) | When you set up GitLab CI/CD across multiple projects, you can visualize the entire pipeline, including all cross-project interdependencies. | +| [Merge request pipelines](../../../ci/pipelines/merge_request_pipelines.md) | Customize a specific pipeline structure for merge requests in order to speed the cycle up by running only important jobs. | +| [Pipeline Graphs](../../../ci/pipelines/index.md#visualize-pipelines) | View the status of pipelines within the merge request, including the deployment process. | +| [Test Coverage visualization](test_coverage_visualization.md) | See test coverage results for merge requests, within the file diff. | ## Security Reports **(ULTIMATE)** diff --git a/doc/user/project/milestones/burndown_and_burnup_charts.md b/doc/user/project/milestones/burndown_and_burnup_charts.md index 7c22b271ec2..05cc424e5ee 100644 --- a/doc/user/project/milestones/burndown_and_burnup_charts.md +++ b/doc/user/project/milestones/burndown_and_burnup_charts.md @@ -11,7 +11,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w  -## Burndown charts **(PREMIUM)** +## Burndown charts > - [Added](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/6495) to GitLab 11.2 for group milestones. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/6903) [fixed burndown charts](#fixed-burndown-charts) in GitLab 13.6. @@ -73,10 +73,8 @@ The chart indicates the project's progress throughout that milestone (for issues In particular, it shows how many issues were or are still open for a given day in the milestone's corresponding period. -The burndown chart can also be toggled to display the cumulative open issue -weight for a given day. When using this feature, make sure issue weights have -been properly assigned, since an open issue with no weight adds zero to the -cumulative value. +You can also toggle the burndown chart to display the +[cumulative open issue weight](#switch-between-number-of-issues-and-issue-weight) for a given day. ### Fixed burndown charts @@ -100,7 +98,7 @@ Therefore, when the milestone start date is changed, the number of opened issues change. Reopened issues are considered as having been opened on the day after they were last closed. -## Burnup charts **(PREMIUM)** +## Burnup charts > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/6903) in GitLab 13.6. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/268350) in GitLab 13.7. @@ -123,12 +121,21 @@ To view a group's burnup chart: ### How burnup charts work Burnup charts have separate lines for total work and completed work. The total line -shows when scope is reduced or added to a milestone. The completed work is a count -of issues closed. +shows changes to the scope of a milestone. When an open issue is moved to another +milestone, the "total issues" goes down but the "completed issues" stays the same. +The completed work is a count of issues closed. When an issue is closed, the "total +issues" remains the same and "completed issues" goes up. -Burnup charts can show either the total number of issues or total weight for each -day of the milestone. Use the toggle above the charts to switch between total -and weight. +## Switch between number of issues and issue weight + +In both burndown or burnup charts you can view them +either by the total number of issues +or the total weight for each day of the milestone. + +To switch between the two settings, select either **Issues** or **Issue weight** above the charts. + +When sorting by weight, make sure all your issues +have weight assigned, because issues with no weight don't show on the chart. <!-- ## Troubleshooting diff --git a/doc/user/project/milestones/img/milestones_new_group_milestone_v13_5.png b/doc/user/project/milestones/img/milestones_new_group_milestone_v13_5.png Binary files differdeleted file mode 100644 index ffe1328b7d3..00000000000 --- a/doc/user/project/milestones/img/milestones_new_group_milestone_v13_5.png +++ /dev/null diff --git a/doc/user/project/milestones/index.md b/doc/user/project/milestones/index.md index fb61e123faf..4501cf500b0 100644 --- a/doc/user/project/milestones/index.md +++ b/doc/user/project/milestones/index.md @@ -53,35 +53,23 @@ If you're in a project and select **Issues > Milestones**, GitLab displays only ## Creating milestones -NOTE: -A permission level of [Developer or higher](../../permissions.md) is required to create milestones. - -### New project milestone - -To create a project milestone: - -1. In a project, go to **{issues}** **Issues > Milestones**. -1. Select **New milestone**. -1. Enter the title, an optional description, an optional start date, and an optional due date. -1. Select **New milestone**. - - +Users with at least the Developer role can create milestones. -### New group milestone +Milestones can be created either at project or group level. -To create a group milestone: +To create a milestone: -1. In a group, go to **{issues}** **Issues > Milestones**. +1. On the top bar, select **Menu > Projects** and find your project or **Menu > Groups** and find your group. +1. On the left sidebar, select **Issues > Milestones**. 1. Select **New milestone**. 1. Enter the title, an optional description, an optional start date, and an optional due date. 1. Select **New milestone**. - + ## Editing milestones -NOTE: -A permission level of [Developer or higher](../../permissions.md) is required to edit milestones. +Users with at least the Developer role can edit milestones. To edit a milestone: @@ -155,23 +143,18 @@ There are also tabs below these that show the following: - Completed Issues (closed) - **Merge Requests**: Shows all merge requests assigned to the milestone. These are displayed in four columns named: - Work in progress (open and unassigned) - - Waiting for merge (open and unassigned) + - Waiting for merge (open and assigned) - Rejected (closed) - Merged - **Participants**: Shows all assignees of issues assigned to the milestone. - **Labels**: Shows all labels that are used in issues assigned to the milestone. -### Project Burndown Charts +### Burndown Charts -For project milestones, a [burndown chart](burndown_and_burnup_charts.md) is in the milestone view, +The milestone view contains a [burndown and burnup chart](burndown_and_burnup_charts.md), showing the progress of completing a milestone. - - -### Group Burndown Charts - -For group milestones, a [burndown chart](burndown_and_burnup_charts.md) is in the milestone view, -showing the progress of completing a milestone. + ### Milestone sidebar diff --git a/doc/user/project/pages/index.md b/doc/user/project/pages/index.md index 283ed0b61b9..82b1a824f7a 100644 --- a/doc/user/project/pages/index.md +++ b/doc/user/project/pages/index.md @@ -81,7 +81,7 @@ GitLab Pages website. You can either use the GitLab [default domain for GitLab Pages websites](getting_started_part_one.md#gitlab-pages-default-domain-names), `*.gitlab.io`, or your own domain (`example.com`). In that case, you -must have the Administrator role in your domain's registrar (or control panel) to set it up with Pages. +must be an administrator in your domain's registrar (or control panel) to set it up with Pages. The following diagrams show the workflows you might follow to get started with Pages. diff --git a/doc/user/project/pages/introduction.md b/doc/user/project/pages/introduction.md index 10fbc57fa0b..65e1579001b 100644 --- a/doc/user/project/pages/introduction.md +++ b/doc/user/project/pages/introduction.md @@ -63,7 +63,7 @@ If the case of `404.html`, there are different scenarios. For example: You can configure redirects for your site using a `_redirects` file. To learn more, read the [redirects documentation](redirects.md). -## GitLab Pages Access Control **(FREE)** +## GitLab Pages Access Control To restrict access to your website, enable [GitLab Pages Access Control](pages_access_control.md). diff --git a/doc/user/project/protected_branches.md b/doc/user/project/protected_branches.md index 6c18fc158f5..8bb18905222 100644 --- a/doc/user/project/protected_branches.md +++ b/doc/user/project/protected_branches.md @@ -18,7 +18,7 @@ When a branch is protected, the default behavior enforces these restrictions on | Action | Who can do it | |:-------------------------|:------------------------------------------------------------------| -| Protect a branch | Maintainers only. | +| Protect a branch | At least the Maintainer role. | | Push to the branch | GitLab administrators and anyone with **Allowed** permission. (1) | | Force push to the branch | No one. | | Delete the branch | No one. | @@ -35,7 +35,7 @@ Administrators can set a default branch protection level in the Prerequisite: -- You must have at least the [Maintainer role](../permissions.md). +- You must have at least the Maintainer role. To protect a branch: @@ -52,7 +52,7 @@ The protected branch displays in the list of protected branches. Prerequisite: -- You must have at least the [Maintainer role](../permissions.md). +- You must have at least the Maintainer role. To protect multiple branches at the same time: @@ -75,7 +75,7 @@ The protected branch displays in the list of protected branches. ## Create a protected branch -Users with the Developer or higher [role](../permissions.md) can create a protected branch. +Users with at least the Developer role can create a protected branch. Prerequisites: @@ -217,16 +217,16 @@ for details about the pipelines security model. ## Delete a protected branch -Users with the [Maintainer role](../permissions.md) and greater can manually delete protected +Users with at least the Maintainer role can manually delete protected branches by using the GitLab web interface: 1. Go to **Repository > Branches**. 1. Next to the branch you want to delete, select the **Delete** button (**{remove}**). 1. On the confirmation dialog, type the branch name and select **Delete protected branch**. -You can delete a protected branch from the UI only. -This prevents you from accidentally deleting a branch -from the command line or from a Git client application. +Protected branches can only be deleted by using GitLab either from the UI or API. +This prevents accidentally deleting a branch through local Git commands or +third-party Git clients. <!-- ## Troubleshooting diff --git a/doc/user/project/protected_tags.md b/doc/user/project/protected_tags.md index e4743c82a3b..7d071a45d63 100644 --- a/doc/user/project/protected_tags.md +++ b/doc/user/project/protected_tags.md @@ -22,12 +22,12 @@ This feature evolved out of [protected branches](protected_branches.md) By default: -- To create tags, you must have the [Maintainer role](../permissions.md). +- To create tags, you must have the Maintainer role. - No one can update or delete tags. ## Configuring protected tags -To protect a tag, you need to have at least the [Maintainer role](../permissions.md). +To protect a tag, you need to have at least the Maintainer role. 1. Go to the project's **Settings > Repository**. @@ -66,6 +66,26 @@ all matching tags:  +## Prevent tag creation with the same name as branches + +A tag and a branch with identical names can contain different commits. If your +tags and branches use the same names, users running `git checkout` +commands might check out the _tag_ `qa` when they instead meant to check out +the _branch_ `qa`. + +To prevent this problem: + +1. Identify the branch names you do not want used as tags. +1. As described in [Configuring protected tags](#configuring-protected-tags), + create a protected tag: + + - For the **Name**, provide a name, such as `stable`. You can also create a wildcard + like `stable-*` to match multiple names, like `stable-v1` and `stable-v2`. + - For **Allowed to Create**, select **No one**. + - Select **Protect**. + +Users can still create branches, but not tags, with the protected names. + <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/user/project/push_options.md b/doc/user/project/push_options.md index 846d4732533..20dd37578fd 100644 --- a/doc/user/project/push_options.md +++ b/doc/user/project/push_options.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Push Options **(FREE)** GitLab supports using client-side [Git push options](https://git-scm.com/docs/git-push#Documentation/git-push.txt--oltoptiongt) -to perform various actions at the same time as pushing changes. Additionally, [Push Rules](../../push_rules/push_rules.md) offer server-side control and enforcement options. +to perform various actions at the same time as pushing changes. Additionally, [Push Rules](repository/push_rules.md) offer server-side control and enforcement options. Currently, there are push options available for: diff --git a/doc/user/project/quick_actions.md b/doc/user/project/quick_actions.md index 21a080de404..8070c37db78 100644 --- a/doc/user/project/quick_actions.md +++ b/doc/user/project/quick_actions.md @@ -49,7 +49,7 @@ threads. Some quick actions might not be available to all subscription tiers. | Command | Issue | Merge request | Epic | Action | |:-------------------------------------------------------------------------------------------------|:-----------------------|:-----------------------|:-----------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `/add_contacts email1 email2` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Add one or more [CRM contacts](../crm/index.md) ([introduced in GitLab 14.6](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73413)). | +| `/add_contacts [contact:email1@example.com] [contact:email2@example.com]` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Add one or more [CRM contacts](../crm/index.md) ([introduced in GitLab 14.6](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73413)). | | `/approve` | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Approve the merge request. | | `/assign @user1 @user2` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Assign one or more users. | | `/assign me` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Assign yourself. | @@ -78,7 +78,7 @@ threads. Some quick actions might not be available to all subscription tiers. | `/lock` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Lock the discussions. | | `/merge` | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Merge changes. Depending on the project setting, this may be [when the pipeline succeeds](merge_requests/merge_when_pipeline_succeeds.md), or adding to a [Merge Train](../../ci/pipelines/merge_trains.md). | | `/milestone %milestone` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Set milestone. | -| `/move <path/to/project>` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Move this issue to another project. | +| `/move <path/to/project>` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Move this issue to another project. Be careful when moving an issue to a project with different access rules. Before moving the issue, make sure it does not contain sensitive data. | | `/parent_epic <epic>` | **{dotted-circle}** No | **{dotted-circle}** No | **{check-circle}** Yes | Set parent epic to `<epic>`. The `<epic>` value should be in the format of `&epic`, `group&epic`, or a URL to an epic ([introduced in GitLab 12.1](https://gitlab.com/gitlab-org/gitlab/-/issues/10556)). | | `/promote` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Promote issue to epic. | | `/promote_to_incident` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Promote issue to incident ([introduced in GitLab 14.5](https://gitlab.com/gitlab-org/gitlab/-/issues/296787)). | @@ -89,7 +89,7 @@ threads. Some quick actions might not be available to all subscription tiers. | `/relabel ~label1 ~label2` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Replace current labels with those specified. | | `/relate #issue1 #issue2` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Mark issues as related. | | `/remove_child_epic <epic>` | **{dotted-circle}** No | **{dotted-circle}** No | **{check-circle}** Yes | Remove child epic from `<epic>`. The `<epic>` value should be in the format of `&epic`, `group&epic`, or a URL to an epic ([introduced in GitLab 12.0](https://gitlab.com/gitlab-org/gitlab/-/issues/7330)). | -| `/remove_contacts email1 email2` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Remove one or more [CRM contacts](../crm/index.md) ([introduced in GitLab 14.6](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73413)). | +| `/remove_contacts [contact:email1@example.com] [contact:email2@example.com]` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Remove one or more [CRM contacts](../crm/index.md) ([introduced in GitLab 14.6](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73413)). | | `/remove_due_date` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Remove due date. | | `/remove_epic` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Remove from epic. | | `/remove_estimate` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Remove time estimate. | diff --git a/doc/user/project/releases/index.md b/doc/user/project/releases/index.md index 747b41d07f2..8049ee9726d 100644 --- a/doc/user/project/releases/index.md +++ b/doc/user/project/releases/index.md @@ -393,7 +393,7 @@ deploy_to_production: To set a deploy freeze window in the UI, complete these steps: -1. Sign in to GitLab as a user with the [Maintainer role](../../permissions.md). +1. Sign in to GitLab as a user with the Maintainer role. 1. On the left sidebar, select **Project information**. 1. In the left navigation menu, navigate to **Settings > CI/CD**. 1. Scroll to **Deploy freezes**. @@ -480,7 +480,7 @@ to use the same URL. This is defined during [link creation](../../../api/release The format of the URL is: ```plaintext -https://host/namespace/project/releases/:release/downloads/:filepath +https://host/namespace/project/-/releases/:release/downloads/:filepath ``` If you have an asset for the `v11.9.0-rc2` release in the `gitlab-org` @@ -498,7 +498,7 @@ namespace and `gitlab-runner` project on `gitlab.com`, for example: This asset has a direct link of: ```plaintext -https://gitlab.com/gitlab-org/gitlab-runner/releases/v11.9.0-rc2/downloads/binaries/gitlab-runner-linux-amd64 +https://gitlab.com/gitlab-org/gitlab-runner/-/releases/v11.9.0-rc2/downloads/binaries/gitlab-runner-linux-amd64 ``` The physical location of the asset can change at any time and the direct link remains unchanged. @@ -706,7 +706,7 @@ Here is an example of a release evidence object: ### Collect release evidence **(PREMIUM SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/199065) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.10. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/199065) in GitLab 12.10. When a release is created, release evidence is automatically collected. To initiate evidence collection any other time, use an [API call](../../../api/releases/index.md#collect-release-evidence). You can collect release evidence multiple times for one release. @@ -714,7 +714,7 @@ Evidence collection snapshots are visible on the Releases page, along with the t ### Include report artifacts as release evidence **(ULTIMATE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32773) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.2. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32773) in GitLab 13.2. When you create a release, if [job artifacts](../../../ci/yaml/index.md#artifactsreports) are included in the last pipeline that ran, they are automatically included in the release as release evidence. @@ -767,22 +767,22 @@ In the API: > [Changes were made to the Guest role access](https://gitlab.com/gitlab-org/gitlab/-/issues/335209) in GitLab 14.5. -- Users with the [Reporter role or above](../../../user/permissions.md#project-members-permissions) +- Users with at least the Reporter role have read and download access to the project releases. -- Users with the [Guest role](../../../user/permissions.md#project-members-permissions) +- Users with the Guest role have read and download access to the project releases. This includes associated Git-tag-names, release description, author information of the releases. However, other repository-related information, such as [source code](#source-code), [release evidence](#release-evidence) are redacted. ### Create, update, and delete a release and its assets -- Users with [Developer role or above](../../../user/permissions.md#project-members-permissions) +- Users with at least the Developer role have write access to the project releases and assets. - If a release is associated with a [protected tag](../protected_tags.md), the user must be [allowed to create the protected tag](../protected_tags.md#configuring-protected-tags) too. As an example of release permission control, you can allow only -[Maintainer role or above](../../../user/permissions.md#project-members-permissions) +users with at least the Maintainer role to create, update, and delete releases by protecting the tag with a wildcard (`*`), and set **Maintainer** in the **Allowed to create** column. diff --git a/doc/user/project/repository/branches/default.md b/doc/user/project/repository/branches/default.md index 65b7c4a9881..d706a0e8f8a 100644 --- a/doc/user/project/repository/branches/default.md +++ b/doc/user/project/repository/branches/default.md @@ -38,7 +38,7 @@ the [Git commands you need](#update-the-default-branch-name-in-your-repository) To update the default branch name for an individual [project](../../index.md): -1. Sign in to GitLab with at least the [Maintainer](../../../permissions.md) role. +1. Sign in to GitLab with at least the Maintainer role. 1. In the left navigation menu, go to **Settings > Repository**. 1. Expand **Default branch**, and select a new default branch. 1. Optional. Select the **Auto-close referenced issues on default branch** checkbox to close @@ -57,8 +57,8 @@ GitLab administrators can configure a new default branch name at the ### Instance-level custom initial branch name **(FREE SELF)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/221013) in GitLab 13.2. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/325163) in GitLab 13.12. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/221013) in GitLab 13.2 [with a flag](../../../../administration/feature_flags.md) named `global_default_branch_name`. Enabled by default. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/325163) in GitLab 13.12. Feature flag `global_default_branch_name` removed. GitLab [administrators](../../../permissions.md) of self-managed instances can customize the initial branch for projects hosted on that instance. Individual @@ -129,7 +129,7 @@ renames a Git repository's (`example`) default branch. git symbolic-ref refs/remotes/origin/HEAD refs/remotes/origin/main ``` -1. Sign in to GitLab with at least the [Maintainer](../../../permissions.md) +1. Sign in to GitLab with at least the Maintainer role and follow the instructions to [change the default branch for this project](#change-the-default-branch-name-for-a-project). Select `main` as your new default branch. diff --git a/doc/user/project/repository/forking_workflow.md b/doc/user/project/repository/forking_workflow.md index 1ab21286a8e..b085372c8f2 100644 --- a/doc/user/project/repository/forking_workflow.md +++ b/doc/user/project/repository/forking_workflow.md @@ -27,16 +27,15 @@ To fork an existing project in GitLab: 1. Select the project to fork to: - Recommended method. Below **Select a namespace to fork the project**, identify - the project you want to fork to, and click **Select**. Only namespaces you have - Developer and higher [permissions](../../permissions.md) for are shown. + the project you want to fork to, and click **Select**. Only namespaces where you have + at least the Developer role for are shown.  - Experimental method. If your GitLab administrator has - [enabled the experimental fork project form](#enable-or-disable-the-fork-project-form), read + enabled the experimental fork project form, read [Create a fork with the fork project form](#create-a-fork-with-the-fork-project-form). - Only namespaces you have at least the Developer - [role](../../permissions.md) for are shown. + Only namespaces where you have at least the Developer role for are shown. NOTE: The project path must be unique in the namespace. @@ -85,14 +84,14 @@ You can unlink your fork from its upstream project in the [advanced settings](.. ## Create a fork with the fork project form **(FREE SELF)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15013) in GitLab 13.11. -> - It's [deployed behind a feature flag](../../../user/feature_flags.md), disabled by default. -> - It's disabled on GitLab.com. -> - It's not recommended for production use. -> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-the-fork-project-form). **(FREE SELF)** +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15013) in GitLab 13.11 [with a flag](../../../administration/feature_flags.md) named `fork_project_form`. Disabled by default. +> - [Enabled on self-managed and GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/64967) in GitLab 13.8. -This experimental version of the fork project form is available only if your GitLab -administrator has [enabled it](#enable-or-disable-the-fork-project-form): +FLAG: +On self-managed GitLab, by default this feature is available. To hide the feature, ask an administrator to [disable the feature flag](../../../administration/feature_flags.md) named `fork_project_form`. +On GitLab.com, this feature is available. + +This version of the fork project form is experimental:  @@ -103,23 +102,3 @@ To use it, follow the instructions at [Creating a fork](#creating-a-fork) and pr - The project slug. - Optional. The project description. - The visibility level for your fork. - -### Enable or disable the fork project form **(FREE SELF)** - -The new [fork project form](#create-a-fork-with-the-fork-project-form) is under -development and not ready for production use. It is deployed behind a feature flag -that is **disabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) -can enable it. - -To enable it: - -```ruby -Feature.enable(:fork_project_form) -``` - -To disable it: - -```ruby -Feature.disable(:fork_project_form) -``` diff --git a/doc/user/project/repository/gpg_signed_commits/index.md b/doc/user/project/repository/gpg_signed_commits/index.md index 0c5c0d5fa7c..f5cea8a8075 100644 --- a/doc/user/project/repository/gpg_signed_commits/index.md +++ b/doc/user/project/repository/gpg_signed_commits/index.md @@ -273,7 +273,7 @@ To remove a GPG key from your account: ## Rejecting commits that are not signed **(PREMIUM)** You can configure your project to reject commits that aren't GPG-signed -via [push rules](../../../../push_rules/push_rules.md). +via [push rules](../push_rules.md). ## GPG signing API diff --git a/doc/user/project/repository/mirror/index.md b/doc/user/project/repository/mirror/index.md index c8950d39f28..cddbc6fd891 100644 --- a/doc/user/project/repository/mirror/index.md +++ b/doc/user/project/repository/mirror/index.md @@ -37,7 +37,7 @@ Mirror a repository when: Prerequisite: -- You must have at least the [Maintainer role](../../../permissions.md) for the project. +- You must have at least the Maintainer role for the project. - If your mirror connects with `ssh://`, the host key must be detectable on the server, or you must have a local copy of the key. @@ -45,8 +45,8 @@ Prerequisite: 1. On the left sidebar, select **Settings > Repository**. 1. Expand **Mirroring repositories**. 1. Enter a **Git repository URL**. For security reasons, the URL to the original - repository is only displayed to users with the [Maintainer role](../../../permissions.md) - or the [Owner role](../../../permissions.md) for the mirrored project. + repository is only displayed to users with the Maintainer role + or the Owner role for the mirrored project. 1. Select a **Mirror direction**. 1. If you entered a `ssh://` URL, select either: - **Detect host keys**: GitLab fetches the host keys from the server and displays the fingerprints. @@ -87,7 +87,7 @@ While mirrors are scheduled to update automatically, you can force an immediate Prerequisite: -- You must have at least the [Maintainer role](../../../permissions.md) for the project. +- You must have at least the Maintainer role for the project. 1. On the top bar, select **Menu > Projects** and find your project. 1. On the left sidebar, select **Settings > Repository**. diff --git a/doc/user/project/repository/push_rules.md b/doc/user/project/repository/push_rules.md new file mode 100644 index 00000000000..bb473a2830b --- /dev/null +++ b/doc/user/project/repository/push_rules.md @@ -0,0 +1,288 @@ +--- +stage: Create +group: Source Code +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" +--- + +# Push rules **(PREMIUM)** + +Gain additional control over what can and can't be pushed to your repository by using +regular expressions to reject pushes based on commit contents, branch names or file details. + +GitLab already offers [protected branches](../protected_branches.md), but there are +cases when you need some specific rules. Some common scenarios: preventing Git tag removal, or +enforcing a special format for commit messages. + +Push rules are [pre-receive Git hooks](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks) you +can enable in a user-friendly interface. They are defined either: + +- Globally if you are an administrator. +- Per project, so you can have different rules applied to different + projects depending on your needs. + +## Use cases + +Every push rule could have its own use case, but let's consider some examples. + +### Commit messages with a specific reference + +Let's assume you have the following requirements for your workflow: + +- every commit should reference a Jira issue, for example: `Refactored css. Fixes JIRA-123.` +- users should not be able to remove Git tags with `git push` + +Write a regular expression that requires the mention +of a Jira issue in the commit message, like `JIRA\-\d+`. + +Now when a user tries to push a commit with a message `Bugfix`, their push is +declined. Only pushing commits with messages like `Bugfix according to JIRA-123` +is accepted. + +### Restrict branch names + +If your company has a strict policy for branch names, you may want the branches to start +with a certain name. This approach enables different +GitLab CI/CD jobs (such as `feature`, `hotfix`, `docker`, `android`) that rely on the +branch name. + +Your developers may not remember that policy, so they might push to +various branches, and CI pipelines might not work as expected. By restricting the +branch names globally in Push Rules, such mistakes are prevented. +All branch names that don't match your push rule are rejected. + +Note that the name of your default branch is always allowed, regardless of the branch naming +regular expression (regex) specified. GitLab is configured this way +because merges typically have the default branch as their target. +If you have other target branches, include them in your regex. (See [Enabling push rules](#enabling-push-rules)). + +The default branch also defaults to being a [protected branch](../protected_branches.md), +which already limits users from pushing directly. + +Some example regular expressions you can use in push rules: + +- `^JIRA-` Branches must start with `JIRA-`. +- `-JIRA$` Branches must end with `-JIRA`. +- `^[a-z0-9\\-]{4,15}$` Branches must be between `4` and `15` characters long, + accepting only lowercase letters, numbers and dashes. + +#### Default restricted branch names + +> Introduced in GitLab 12.10. + +By default, GitLab restricts certain formats of branch names for security purposes. +40-character hexadecimal names, similar to Git commit hashes, are prohibited. + +### Custom Push Rules **(PREMIUM SELF)** + +It's possible to create custom push rules rather than the push rules available in +**Admin Area > Push Rules** by using more advanced server hooks. + +See [server hooks](../../../administration/server_hooks.md) for more information. + +## Enabling push rules + +You can create push rules for all new projects to inherit, but they can be overridden +at the project level or the [group level](../../group/index.md#group-push-rules). + +To create global push rules: + +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Push Rules**. + +To override global push rules in a project's settings: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Repository**. +1. Expand **Push rules**. +1. Set the rule you want. +1. Select **Save push rules**. + +The following options are available: + +| Push rule | Description | +|---------------------------------|-------------| +| Removal of tags with `git push` | Forbid users to remove Git tags with `git push`. Tags can be deleted through the web UI. | +| Check whether the commit author is a GitLab user | Restrict commits to existing GitLab users (checked against their emails). <sup>1</sup> | +| Reject unverified users | GitLab rejects any commit that was not committed by the same user as the user who pushed it, or where the committer's email address is not [confirmed](../../../security/user_email_confirmation.md). | +| Check whether commit is signed through GPG | Reject commit when it is not signed through GPG. Read [signing commits with GPG](gpg_signed_commits/index.md). | +| Prevent pushing secret files | GitLab rejects any files that are likely to contain secrets. See the [forbidden file names](#prevent-pushing-secrets-to-the-repository). | +| Require expression in commit messages | Only commit messages that match this regular expression are allowed to be pushed. <sup>2</sup> Leave empty to allow any commit message. Uses multiline mode, which can be disabled using `(?-m)`. | +| Reject expression in commit messages | Only commit messages that do not match this regular expression are allowed to be pushed. <sup>2</sup> Leave empty to allow any commit message. Uses multiline mode, which can be disabled using `(?-m)`. | +| Restrict by branch name | Only branch names that match this regular expression are allowed to be pushed. <sup>2</sup> Leave empty to allow all branch names. | +| Restrict by commit author's email | Only commit author's email that match this regular expression are allowed to be pushed. <sup>1</sup> <sup>2</sup> Leave empty to allow any email. | +| Prohibited file names | Any committed filenames that match this regular expression and do not already exist in the repository are not allowed to be pushed. <sup>2</sup> Leave empty to allow any filenames. See [common examples](#prohibited-file-names). | +| Maximum file size | Pushes that contain added or updated files that exceed this file size (in MB) are rejected. Set to 0 to allow files of any size. Files tracked by Git LFS are exempted. | + +1. Checks both the commit author and committer. +1. GitLab uses [RE2 syntax](https://github.com/google/re2/wiki/Syntax) for regular expressions in push rules, and you can test them at the [regex101 regex tester](https://regex101.com/). + +### Caveat to "Reject unsigned commits" push rule + +This push rule ignores commits that are authenticated and created by GitLab +(either through the UI or API). When the **Reject unsigned commits** push rule is +enabled, unsigned commits may still show up in the commit history if a commit was +created **in** GitLab itself. As expected, commits created outside GitLab and +pushed to the repository are rejected. For more information about how GitLab +plans to fix this issue, read [issue #19185](https://gitlab.com/gitlab-org/gitlab/-/issues/19185). + +#### "Reject unsigned commits" push rule disables Web IDE + +In 13.10, if a project has the "Reject unsigned commits" push rule, the user is not allowed to +commit through GitLab Web IDE. + +To allow committing through the Web IDE on a project with this push rule, a GitLab administrator +must disable the feature flag `reject_unsigned_commits_by_gitlab`. This can be done through a +[rails console](../../../administration/operations/rails_console.md) and running: + +```ruby +Feature.disable(:reject_unsigned_commits_by_gitlab) +``` + +## Prevent pushing secrets to the repository + +> Moved to GitLab Premium in 13.9. + +Secrets, such as credential files and SSH private keys, should never be committed to a version control +system. In GitLab, you can use a predefined list of files to block those files from a +repository. Any merge request containing a file matching the list is blocked from being merged. +Files already committed to the repository are not restricted by this push rule. + +Files blocked by this rule are listed below. For a complete list of criteria, see +[`files_denylist.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/gitlab/checks/files_denylist.yml). + +- AWS CLI credential blobs: + + - `.aws/credentials` + - `aws/credentials` + - `homefolder/aws/credentials` + +- Private RSA SSH keys: + + - `/ssh/id_rsa` + - `/.ssh/personal_rsa` + - `/config/server_rsa` + - `id_rsa` + - `.id_rsa` + +- Private DSA SSH keys: + + - `/ssh/id_dsa` + - `/.ssh/personal_dsa` + - `/config/server_dsa` + - `id_dsa` + - `.id_dsa` + +- Private ED25519 SSH keys: + + - `/ssh/id_ed25519` + - `/.ssh/personal_ed25519` + - `/config/server_ed25519` + - `id_ed25519` + - `.id_ed25519` + +- Private ECDSA SSH keys: + + - `/ssh/id_ecdsa` + - `/.ssh/personal_ecdsa` + - `/config/server_ecdsa` + - `id_ecdsa` + - `.id_ecdsa` + +- Private ECDSA_SK SSH keys (GitLab 14.8 and later): + + - `/ssh/id_ecdsa_sk` + - `/.ssh/personal_ecdsa_sk` + - `/config/server_ecdsa_sk` + - `id_ecdsa_sk` + - `.id_ecdsa_sk` + +- Private ED25519_SK SSH keys (GitLab 14.8 and later): + + - `/ssh/id_ed25519_sk` + - `/.ssh/personal_ed25519_sk` + - `/config/server_ed25519_sk` + - `id_ed25519_sk` + - `.id_ed25519_sk` + +- Any files ending with these suffixes: + + - `*.pem` + - `*.key` + - `*.history` + - `*_history` + +### Prevent pushing secrets to all projects + +To set a global push rule to prevent pushing secrets to all projects: + +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Push Rules**. +1. Expand **Push rules**. +1. Select **Prevent pushing secret files**. +1. Select **Save push rules**. + +### Prevent pushing secrets to a project + +The push rule of a project overrides the global push rule. + +To prevent pushing secrets to a project: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Repository**. +1. Expand **Push rules**. +1. Select **Prevent pushing secret files**. +1. Select **Save push rules**. + +## Prohibited file names + +> Moved to GitLab Premium in 13.9. + +Each filename contained in a Git push is compared to the regular expression in this field. Filenames in Git consist of both the file's name and any directory that may precede it. A singular regular expression can contain multiple independent matches used as exclusions. File names can be broadly matched to any location in the repository, or restricted to specific locations. Filenames can also be partial matches used to exclude file types by extension. + +The following examples make use of regex string boundary characters which match the beginning of a string (`^`), and the end (`$`). They also include instances where either the directory path or the filename can include `.` or `/`. Both of these special regex characters have to be escaped with a backslash `\\` to be used as normal characters in a match condition. + +Example: prevent pushing any `.exe` files to any location in the repository. This uses a partial match, which matches any filename that contains `.exe` at the end: + +```plaintext +\.exe$ +``` + +Example: prevent a specific configuration file in the repository root from being pushed: + +```plaintext +^config\.yml$ +``` + +Example: prevent a specific configuration file in a known directory from being pushed: + +```plaintext +^directory-name\/config\.yml$ +``` + +Example: prevent the specific file named `install.exe` from being pushed to any +location in the repository. The parenthesized expression `(^|\/)` matches either +a file following a directory separator or a file in the root directory of the repository: + +```plaintext +(^|\/)install\.exe$ +``` + +Example: combining all of the above in a single expression. The preceding expressions rely +on the end-of-string character `$`. We can move that part of each expression to the +end of the grouped collection of match conditions where it is appended to all matches: + +```plaintext +(\.exe|^config\.yml|^directory-name\/config\.yml|(^|\/)install\.exe)$ +``` + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/project/repository/reducing_the_repo_size_using_git.md b/doc/user/project/repository/reducing_the_repo_size_using_git.md index 23cead7e8a7..37b16d90d93 100644 --- a/doc/user/project/repository/reducing_the_repo_size_using_git.md +++ b/doc/user/project/repository/reducing_the_repo_size_using_git.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Reduce repository size +# Reduce repository size **(FREE)** Git repositories become larger over time. When large files are added to a Git repository: @@ -193,7 +193,7 @@ When using repository cleanup, note: Repository size limits: - Can [be set by an administrator](../../admin_area/settings/account_and_limit_settings.md#account-and-limit-settings) - on self-managed instances. **(PREMIUM SELF)** +- Can [be set by an administrator](../../admin_area/settings/account_and_limit_settings.md) on self-managed instances. - Are [set for GitLab.com](../../gitlab_com/index.md#account-and-limit-settings). When a project has reached its size limit, you cannot: diff --git a/doc/user/project/repository/x509_signed_commits/index.md b/doc/user/project/repository/x509_signed_commits/index.md index f9d9af3e672..27ef14d31c5 100644 --- a/doc/user/project/repository/x509_signed_commits/index.md +++ b/doc/user/project/repository/x509_signed_commits/index.md @@ -1,8 +1,7 @@ --- stage: Create group: Source Code -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" -type: concepts, howto +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Sign commits and tags with X.509 certificates **(FREE)** @@ -335,7 +334,7 @@ step of the previous [main verification checks](#main-verification-checks). signature.__send__(:p7).verify([], signature.__send__(:cert_store), signature.__send__(:signed_text)) ``` - 1. If this fails, add the missing certificate(s) required to establish trust + 1. If this fails, add the missing certificates required to establish trust [to the GitLab certificate store](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). 1. After adding more certificates, (if these troubleshooting steps then pass) @@ -347,7 +346,7 @@ step of the previous [main verification checks](#main-verification-checks). pp signature.__send__(:p7).certificates ; nil ``` -Ensure any additional intermediate certificate(s) and the root certificate are added +Ensure any additional intermediate certificates and the root certificate are added to the certificate store. For consistency with how certificate chains are built on web servers: diff --git a/doc/user/project/requirements/index.md b/doc/user/project/requirements/index.md index 5fe0e0ef3a2..d549a22910a 100644 --- a/doc/user/project/requirements/index.md +++ b/doc/user/project/requirements/index.md @@ -40,7 +40,7 @@ can create a new requirement. Prerequisite: -- You must have at least the Reporter [role](../../permissions.md). +- You must have at least the Reporter role. To create a requirement: @@ -70,7 +70,7 @@ You can edit a requirement from the requirements list page. Prerequisite: -- You must have at least the Reporter [role](../../permissions.md). +- You must have at least the Reporter role. To edit a requirement: @@ -86,7 +86,7 @@ you're in the **Open** tab. Prerequisite: -- You must have at least the Reporter [role](../../permissions.md). +- You must have at least the Reporter role. To archive a requirement, select **Archive** (**{archive}**). @@ -98,7 +98,7 @@ You can view the list of archived requirements in the **Archived** tab. Prerequisite: -- You must have at least the Reporter [role](../../permissions.md). +- You must have at least the Reporter role.  @@ -217,7 +217,7 @@ requirements_confirmation: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/246857) in GitLab 13.7. -You must have at least the Reporter [role](../../permissions.md). +You must have at least the Reporter role. You can import requirements to a project by uploading a [CSV file](https://en.wikipedia.org/wiki/Comma-separated_values) with the columns `title` and `description`. @@ -291,7 +291,7 @@ audit and regulatory compliance tasks. Prerequisite: -- You must have at least the Reporter [role](../../permissions.md). +- You must have at least the Reporter role. To export requirements: diff --git a/doc/user/project/service_desk.md b/doc/user/project/service_desk.md index 1b30f14ac90..1e5b818c99d 100644 --- a/doc/user/project/service_desk.md +++ b/doc/user/project/service_desk.md @@ -87,7 +87,7 @@ To improve your project's security, we recommend the following: Unblocked email spam can result in many spam issues being created. The unique internal email address is visible to project members at least -the Reporter [role](../permissions.md) in your GitLab instance. +the Reporter role in your GitLab instance. An external user (issue creator) cannot see the internal email address displayed in the information note. @@ -333,3 +333,10 @@ Note that: Behind the scenes, Service Desk works by the special Support Bot user creating issues. This user does not count toward the license limit count. + +## Troubleshooting Service Desk + +### Emails to Service Desk do not create issues + +Your emails might be ignored because they contain one of the +[email headers that GitLab ignores](../../administration/incoming_email.md#rejected-headers). diff --git a/doc/user/project/settings/import_export.md b/doc/user/project/settings/import_export.md index 06bdf4ca14b..8cee567ae93 100644 --- a/doc/user/project/settings/import_export.md +++ b/doc/user/project/settings/import_export.md @@ -122,8 +122,7 @@ As described in the API documentation, the query may return an import error or e The following items are imported but changed slightly: -- Project members with the [Owner role](../../permissions.md) - are imported as Maintainers. +- Project members with the Owner role are imported as Maintainers. - If an imported project contains merge requests originating from forks, then new branches associated with such merge requests are created in a project during the import/export. Thus, the number of branches in the exported project might be bigger than in the original project. @@ -160,6 +159,8 @@ Imported users can be mapped by their public email addresses on self-managed ins for mapping to work correctly. - For contributions to be mapped correctly, users must be an existing member of the namespace, or they can be added as a member of the project. Otherwise, a supplementary comment is left to mention that the original author and the MRs, notes, or issues that are owned by the importer. +- Imported users are set as [direct members](../members/index.md) + in the imported project. For project migration imports performed over GitLab.com groups, preserving author information is possible through a [professional services engagement](https://about.gitlab.com/services/migration/). diff --git a/doc/user/project/settings/index.md b/doc/user/project/settings/index.md index 9df545b52ec..4c6be3511f2 100644 --- a/doc/user/project/settings/index.md +++ b/doc/user/project/settings/index.md @@ -16,7 +16,7 @@ In GitLab versions [13.10 and later](https://gitlab.com/groups/gitlab-org/-/epic GitLab displays a search box to help you find the settings you want to view. NOTE: -Only users who have the [Maintainer role](../../permissions.md) for the project and administrators can +Only users who have the Maintainer role for the project and administrators can access project settings. ## General settings @@ -93,7 +93,7 @@ executed in place of the local project's `gitlab-ci.yml` file. As part of this p files are merged together any time the pipeline runs. Jobs and variables defined in the compliance pipeline can't be changed by variables in the local project's `gitlab-ci.yml` file. -When used to enforce scan execution, this feature has some overlap with [scan execution policies](../../application_security/policies/#scan-execution-policies), +When used to enforce scan execution, this feature has some overlap with [scan execution policies](../../application_security/policies/scan-execution-policies.md), as we have not [unified the user experience for these two features](https://gitlab.com/groups/gitlab-org/-/epics/7312). For details on the similarities and differences between these features, see [Enforce scan execution](../../application_security/#enforce-scan-execution). @@ -249,7 +249,7 @@ Use the switches to enable or disable the following features: |:---------------------------------|:--------------------------|:--------------| | **Issues** | ✓ | Activates the GitLab issues tracker. | | **Repository** | ✓ | Enables [repository](../repository/) functionality | -| **Merge Requests** | ✓ | Enables [merge request](../merge_requests/) functionality; also see [Merge request settings](#merge-request-settings). | +| **Merge requests** | ✓ | Enables [merge request](../merge_requests/) functionality; also see [Merge request settings](#merge-request-settings). | | **Forks** | ✓ | Enables [forking](../repository/forking_workflow.md) functionality. | | **Git Large File Storage (LFS)** | | Enables the use of [large files](../../../topics/git/lfs/index.md#git-large-file-storage-lfs). | | **Packages** | | Supports configuration of a [package registry](../../../administration/packages/index.md#gitlab-package-registry-administration) functionality. | @@ -281,7 +281,7 @@ Some features depend on others: - If you disable **Repository** functionality, GitLab also disables the following features for your project: - - **Merge Requests** + - **Merge requests** - **CI/CD** - **Container Registry** - **Git Large File Storage** @@ -360,7 +360,7 @@ available in project listings. Only project owners and administrators have the To find an archived project: -1. Sign in to GitLab as the project owner or a user with the Administrator role. +1. Sign in to GitLab as the project owner or a user with administrator access. 1. If you: - Have the project's URL, open the project's page in your browser. - Don't have the project's URL: @@ -404,11 +404,17 @@ NOTE: Only project owners and administrators have the [permissions](../../permissions.md#project-members-permissions) to transfer a project. -You can transfer an existing project into a [group](../../group/index.md) if: +You can transfer an existing project into a [group](../../group/index.md). -- You have at least **Maintainer** [role](../../permissions.md#project-members-permissions) in that group. -- You're at least an **Owner** of the project to be transferred. +Prerequisites: + +- You must have at least the Maintainer role in that group. +- You must be the Owner of that project. - The group to which the project is being transferred to must allow creation of new projects. +- The project must not contain any [container images](../../packages/container_registry/index.md#limitations). + - If you transfer a project to a different root namespace, + the project must not contain any + [NPM packages](../../packages/npm_registry/index.md#limitations). To transfer a project: @@ -423,8 +429,8 @@ read what happens with the [redirects from the old project to the new one](../repository/index.md#what-happens-when-a-repository-path-changes). NOTE: -GitLab administrators can use the administration interface to move any project to any -namespace if needed. +GitLab administrators can use the [administration interface](../../admin_area/index.md#administering-projects) +to move any project to any namespace if needed. ##### Transferring a GitLab.com project to a different subscription tier diff --git a/doc/user/project/settings/project_access_tokens.md b/doc/user/project/settings/project_access_tokens.md index 3fcfe202d38..aa30c576c7c 100644 --- a/doc/user/project/settings/project_access_tokens.md +++ b/doc/user/project/settings/project_access_tokens.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication & Authorization +group: Authentication and Authorization info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" type: reference, howto --- @@ -42,7 +42,7 @@ To create a project access token: 1. On the top bar, select **Menu > Projects** and find your project. 1. On the left sidebar, select **Settings > Access Tokens**. -1. Enter a name. +1. Enter a name. The token name is visible to any user with permissions to view the project. 1. Optional. Enter an expiry date for the token. The token will expire on that date at midnight UTC. 1. Select a role for the token. 1. Select the [desired scopes](#scopes-for-a-project-access-token). @@ -84,16 +84,16 @@ To enable or disable project access token creation for all projects in a top-lev Even when creation is disabled, you can still use and revoke existing project access tokens. -## Project bot users +## Bot users for projects > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/210181) in GitLab 13.0. > - [Excluded from license seat use](https://gitlab.com/gitlab-org/gitlab/-/issues/223695) in GitLab 13.5. -Project bot users are [GitLab-created service accounts](../../../subscriptions/self_managed/index.md#billable-users). +Bot users for projects are [GitLab-created service accounts](../../../subscriptions/self_managed/index.md#billable-users). Each time you create a project access token, a bot user is created and added to the project. These bot users do not count as licensed seats. -The bot users have [permissions](../../permissions.md#project-members-permissions) that correspond with the +The bot users for projects have [permissions](../../permissions.md#project-members-permissions) that correspond with the selected role and [scope](#scopes-for-a-project-access-token) of the project access token. - The name is set to the name of the token. @@ -106,7 +106,7 @@ selected role and [scope](#scopes-for-a-project-access-token) of the project acc API calls made with a project access token are associated with the corresponding bot user. -Bot users: +Bot users for projects: - Are included in a project's member list but cannot be modified. - Cannot be added to any other project. @@ -114,5 +114,6 @@ Bot users: When the project access token is [revoked](#revoke-a-project-access-token): - The bot user is deleted. -- All records are moved to a system-wide user with the username `Ghost User`. For more information, see - [associated records](../../profile/account/delete_account.md#associated-records). +- All records are moved to a system-wide user with the username [Ghost User](../../profile/account/delete_account.md#associated-records). + +See also [Bot users for groups](../../group/settings/group_access_tokens.md#bot-users-for-groups). diff --git a/doc/user/project/static_site_editor/index.md b/doc/user/project/static_site_editor/index.md index 50b1a3a929d..072f5bf1927 100644 --- a/doc/user/project/static_site_editor/index.md +++ b/doc/user/project/static_site_editor/index.md @@ -12,6 +12,14 @@ description: "The static site editor enables users to edit content on static web > - WYSIWYG editor [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214559) in GitLab 13.0. > - Non-Markdown content blocks not editable on the WYSIWYG mode [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216836) in GitLab 13.3. > - Formatting Markdown [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/49052) in GitLab 13.7. +> - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/77246) in GitLab 14.7. + +WARNING: +This feature is in its end-of-life process. It is +[deprecated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/77246) +for use in GitLab 14.7, and is planned for +[removal](https://gitlab.com/groups/gitlab-org/-/epics/7351) in GitLab 15.0. +Users should instead use the [Web Editor](../repository/web_editor.md) or [Web IDE](../web_ide/index.md). Static Site Editor (SSE) enables users to edit content on static websites without prior knowledge of the underlying templating language, site architecture, or diff --git a/doc/user/project/time_tracking.md b/doc/user/project/time_tracking.md index 6ceb8c94934..5f747d99ce7 100644 --- a/doc/user/project/time_tracking.md +++ b/doc/user/project/time_tracking.md @@ -8,76 +8,81 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Time tracking **(FREE)** -With time tracking you can track estimates and time spent on issues and merge -requests in GitLab. +You can estimate and track the time you spend on [issues](issues/index.md) +and [merge requests](merge_requests/index.md). + +Then you can [view a report](#view-a-time-tracking-report) that shows totals over time. Use time tracking for these tasks: - Record the time spent working on an issue or a merge request. - Add or update an estimate of the total time to complete an issue or a merge -request. + request. - View a breakdown of time spent working on an issue or a merge request. You don't have to indicate an estimate to enter the time spent, and vice versa. -Data about time tracking shows up on the issue and merge request sidebar: +To enter and remove time tracking data, you must use [quick actions](quick_actions.md). +Type all quick actions on their own lines. +If you use any quick action more than once in a single comment, only its last occurrence is applied. + +You can see the data about time tracking on the right sidebar in issues and merge requests:  -## How to enter data +## Estimates -Time tracking uses two [quick actions](quick_actions.md): `/spend` and `/estimate`. +The estimate is designed to show the total time needed to complete an issue or merge request. -If you use either quick action more than once in a single comment, only the last occurrence is applied. +You can see the estimated time remaining when you hover over the time tracking information in the right sidebar. -Below is an example of how you can use those new quick actions inside a comment. + - +### Add an estimate -Adding time entries (time spent or estimates) is limited to project members -with [Reporter and higher permission levels](../permissions.md). +Prerequisites: -### Estimates +- You must have at least the Reporter role for the project. -To enter an estimate, type `/estimate`, followed by the time. +To enter an estimate, use the `/estimate` [quick action](quick_actions.md), followed by the time. For example, if you need to enter an estimate of 1 month, 2 weeks, 3 days, 4 hours, and 5 minutes, type `/estimate 1mo 2w 3d 4h 5m`. -Check the [time units you can use](#configuration). +Check the [time units you can use](#available-time-units). -The estimate is designed to show the total estimated time. The estimated -time remaining is automatically calculated and displayed when hovering over -the time tracking information in the right sidebar. +An issue or a merge request can have only one estimate. +Every time you enter a new time estimate, it overwrites the previous value. - +### Remove an estimate -An issue or a merge request can have only one estimate. Every time you enter a -new time estimate, it overwrites the previous value. +Prerequisites: -To remove an estimation entirely, use `/remove_estimate`. +- You must have at least the Reporter role for the project. -### Time spent +To remove an estimate entirely, use the `/remove_estimate` [quick action](quick_actions.md). -To enter time spent, type `/spend`, followed by the time. +## Time spent -For example, if you need -to log 1 month, 2 weeks, 3 days, 4 hours, and 5 minutes, type `/spend 1mo 2w 3d 4h 5m`. -Check the [time units you can use](#configuration). +As you work, you can log the time you've spent. Every new time spent entry is added to the current total time spent for the issue or the merge request. -To subtract time, enter a negative value. For example, `/spend -3d` removes three -days from the total time spent. You can't go below 0 minutes of time spent, -so if you remove more time than already entered, GitLab ignores the subtraction. +### Add time spent -You can log time in the past by providing a date after the time. -For example, if you want to log 1 hour of time spent on the 31 January 2021, -you would type `/spend 1h 2021-01-31`. If you supply a date in the future, the -command fails and no time is logged. +Prerequisites: + +- You must have at least the Reporter role for the project. + +To enter time spent, use the `/spend` [quick action](quick_actions.md), followed by the time. + +For example, if you need +to log 1 month, 2 weeks, 3 days, 4 hours, and 5 minutes, type `/spend 1mo 2w 3d 4h 5m`. +Check the [time units you can use](#available-time-units). -To add a timelog entry with a note, create a comment with a description and the quick action. -It then shows in the timelog "Summary/Notes" column. For example: +To add a [time tracking report](#view-a-time-tracking-report) entry with a note, create a comment +with a description and the quick action. +It then shows in the time tracking report **Summary/Notes** column. For example: ```plaintext Draft MR and respond to initial comments @@ -85,7 +90,29 @@ Draft MR and respond to initial comments /spend 30m ``` -To remove all the time spent at once, use `/remove_time_spent`. +### Add time spent on a specific date + +Prerequisites: + +- You must have at least the Reporter role for the project. + +You can log time in the past by providing a date after the time. +For example, to log 1 hour of time spent on 31 January 2021, +type `/spend 1h 2021-01-31`. + +If you type a future date, no time is logged. + +### Remove time spent + +Prerequisites: + +- You must have at least the Reporter role for the project. + +To subtract time, enter a negative value. For example, `/spend -3d` removes three +days from the total time spent. You can't go below 0 minutes of time spent, +so if you remove more time than already entered, GitLab ignores the subtraction. + +To remove all the time spent at once, use the `/remove_time_spent` [quick action](quick_actions.md). ## View a time tracking report @@ -93,35 +120,32 @@ To remove all the time spent at once, use `/remove_time_spent`. You can view a breakdown of time spent on an issue or merge request. -Prerequisites: - -- You must have at least the [Reporter role](../permissions.md#project-members-permissions) for a project. +To view a time tracking report: -To view a time tracking report, go to an issue or a merge request and select **Time tracking report** -in the right sidebar. +1. Go to an issue or a merge request. +1. In the right sidebar, select **Time tracking report**.  The breakdown of spent time is limited to a maximum of 100 entries. -## Configuration +## Available time units The following time units are available: -| Time unit | What to type | Default conversion rate | -| --------- | ------------ | ----------------------- | -| Month | `mo` | 4w | -| Week | `w` | 5d | -| Day | `d` | 8h | -| Hour | `h` | 60m | -| Minute | `m` | | +| Time unit | What to type | Conversion rate | +| --------- | --------------------------- | --------------- | +| Month | `mo`, `month`, or `months` | 4w (160h) | +| Week | `w`, `week`, or `weeks` | 5d (40h) | +| Day | `d`, `day`, or `days` | 8h | +| Hour | `h`, `hour`, or `hours` | 60m | +| Minute | `m`, `minute`, or `minutes` | | ### Limit displayed units to hours **(FREE SELF)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/29469/) in GitLab 12.1. -In GitLab self-managed instances, you can limit the display of time units to -hours. +In GitLab self-managed instances, you can limit the display of time units to hours. To do so: 1. On the top bar, select **Menu > Admin**. diff --git a/doc/user/project/wiki/group.md b/doc/user/project/wiki/group.md index 731fed2595a..2c499af123c 100644 --- a/doc/user/project/wiki/group.md +++ b/doc/user/project/wiki/group.md @@ -22,7 +22,7 @@ Group wikis are similar to [project wikis](index.md), with a few limitations: For updates, follow [the epic that tracks feature parity with project wikis](https://gitlab.com/groups/gitlab-org/-/epics/2782). -Similar to project wikis, group members with the [Developer role](../../permissions.md#group-members-permissions) +Similar to project wikis, group members with at least the Developer role and higher can edit group wikis. Group wiki repositories can be moved using the [Group repository storage moves API](../../../api/group_repository_storage_moves.md). @@ -40,7 +40,7 @@ To access a group wiki: > Introduced in [GitLab 13.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53247). -Users with the [Owner role](../../permissions.md) in a group can +Users with the Owner role in a group can [import and export group wikis](../../group/settings/import_export.md) when importing or exporting a group. diff --git a/doc/user/project/wiki/index.md b/doc/user/project/wiki/index.md index 95ea1b4d0bc..33f6c32d334 100644 --- a/doc/user/project/wiki/index.md +++ b/doc/user/project/wiki/index.md @@ -74,7 +74,7 @@ to be used as your wiki's home page. To create it: ## Create a new wiki page -Users with the [Developer role](../../permissions.md) can create new wiki pages: +Users with at least the Developer role can create new wiki pages: 1. On the top bar, select **Menu**. - For project wikis, select **Projects** and find your project. @@ -137,7 +137,7 @@ may not be able to check out the wiki locally afterward. ## Edit a wiki page -You need at least the [Developer role](../../permissions.md) to edit a wiki page: +You need at least the Developer role to edit a wiki page: 1. On the top bar, select **Menu**. - For project wikis, select **Projects** and find your project. @@ -156,7 +156,7 @@ For an example, read [Table of contents](../../markdown.md#table-of-contents). ## Delete a wiki page -You need at least the [Developer role](../../permissions.md) to delete a wiki page: +You need at least the Developer role to delete a wiki page: 1. On the top bar, select **Menu**. - For project wikis, select **Projects** and find your project. @@ -169,7 +169,7 @@ You need at least the [Developer role](../../permissions.md) to delete a wiki pa ## Move a wiki page -You need at least the [Developer role](../../permissions.md) to move a wiki page: +You need at least the Developer role to move a wiki page: 1. On the top bar, select **Menu**. - For project wikis, select **Projects** and find your project. @@ -239,7 +239,7 @@ Commits to wikis are not counted in [repository analytics](../../analytics/repos > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/23109) in GitLab 13.8, the sidebar can be customized by selecting the **Edit sidebar** button. -You need at least the [Developer role](../../permissions.md) to customize the wiki +You need at least the Developer role to customize the wiki navigation sidebar. This process creates a wiki page named `_sidebar` which fully replaces the default sidebar navigation: @@ -323,9 +323,17 @@ Previously added wiki pages are preserved in case you want to re-enable the wiki. To re-enable it, repeat the process to disable the wiki but toggle it on (in blue). -## Content Editor **(FREE)** +## Content Editor -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5643) in GitLab 14.0. +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5643) in GitLab 14.0. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345398) switching between editing experiences in GitLab 14.7 [with a flag](../../../administration/feature_flags.md) named `wiki_switch_between_content_editor_raw_markdown`. Enabled by default. + +FLAG: +On self-managed GitLab, by default this feature is available. +To hide the feature, ask an administrator to +[disable the feature flag](../../../administration/feature_flags.md) named +`wiki_switch_between_content_editor_raw_markdown`. +On GitLab.com, this feature is available. GitLab version 14.0 introduces a WYSIWYG editing experience for GitLab Flavored Markdown in Wikis through the [Content Editor](../../../development/fe_guide/content_editor.md). diff --git a/doc/user/project/working_with_projects.md b/doc/user/project/working_with_projects.md index 61eca19a67b..9cfcaf4ee81 100644 --- a/doc/user/project/working_with_projects.md +++ b/doc/user/project/working_with_projects.md @@ -99,6 +99,8 @@ To create a blank project: slug as the URL path to the project. To change the slug, first enter the project name, then change the slug. - In the **Project description (optional)** field, enter the description of your project's dashboard. + - In the **Project target (optional)** field, select your project's deployment target. + This information helps GitLab better understand its users and their deployment requirements. - To modify the project's [viewing and access rights](../../public_access/public_access.md) for users, change the **Visibility Level**. - To create README file so that the Git repository is initialized, has a default branch, and @@ -297,6 +299,29 @@ To delete a project: 1. Select **Delete project** 1. Confirm this action by completing the field. +## Projects pending deletion **(PREMIUM)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37014) in GitLab 13.3 for Administrators. +> - [Tab renamed](https://gitlab.com/gitlab-org/gitlab/-/issues/347468) from **Deleted projects** in GitLab 14.6. +> - [Available to all users](https://gitlab.com/gitlab-org/gitlab/-/issues/346976) in GitLab 14.8 [with a flag](../../administration/feature_flags.md) named `project_owners_list_project_pending_deletion`. Enabled by default. + +FLAG: +On self-managed GitLab, by default this feature is available to all users. To make it available for administrators only, +ask an administrator to [disable the feature flag](../../administration/feature_flags.md) named `project_owners_list_project_pending_deletion`. +On GitLab.com, this feature is available to all users. + +When delayed project deletion is [enabled for a group](../group/index.md#enable-delayed-project-deletion), +projects within that group are not deleted immediately, but only after a delay. To access a list of all projects that are pending deletion: + +1. On the top bar, select **Menu > Projects > Explore projects**. +1. Select the **Pending deletion** tab (in GitLab 14.6 and later) or the **Deleted projects** tab (GitLab 14.5 and earlier). + +Listed for each project is: + +- The time the project was marked for deletion. +- The time the project is scheduled for final deletion. +- A **Restore** link to stop the project being eventually deleted. + ## View project activity To view the activity of a project: diff --git a/doc/user/search/advanced_search.md b/doc/user/search/advanced_search.md index 13fba126169..05579696d35 100644 --- a/doc/user/search/advanced_search.md +++ b/doc/user/search/advanced_search.md @@ -140,3 +140,28 @@ its performance: | Issues | `global_search_issues_tab` | When enabled, the global search includes issues as part of the search. | | Merge Requests | `global_search_merge_requests_tab` | When enabled, the global search includes merge requests as part of the search. | | Wiki | `global_search_wiki_tab` | When enabled, the global search includes wiki as part of the search. [Group wikis](../project/wiki/group.md) are not included. | + +## Global Search validation + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/346263) in GitLab 14.6 [with a flag](../../administration/feature_flags.md) named `prevent_abusive_searches`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, + ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `prevent_abusive_searches`. + The feature is not ready for production use. + +To prevent abusive searches, such as searches that may result in a Distributed Denial of Service (DDoS), Global Search ignores, logs, and +doesn't return any results for searches considered abusive according to the following criteria, if `prevent_abusive_searches` feature flag is enabled: + +- Searches with less than 2 characters. +- Searches with any term greater than 100 characters. URL search terms have a maximum of 200 characters. +- Searches with a stop word as the only term (ie: "the", "and", "if", etc.). +- Searches with a `group_id` or `project_id` parameter that is not completely numeric. +- Searches with a `repository_ref` or `project_ref` parameter that has special characters not allowed by [Git refname](https://git-scm.com/docs/git-check-ref-format). +- Searches with a `scope` that is unknown. + +Regardless of the status of the `prevent_abusive_searches` feature flag, searches that don't +comply with the criteria described below aren't logged as abusive but are flagged with an error: + +- Searches with more than 4096 characters. +- Searches with more than 64 terms. diff --git a/doc/user/search/index.md b/doc/user/search/index.md index 0e2be455a0c..0a799843d3c 100644 --- a/doc/user/search/index.md +++ b/doc/user/search/index.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Search issues and merge requests -To search through issues and merge requests in multiple projects, on the top bar, select the **Issues** or **Merge Requests** links. +To search through issues and merge requests in multiple projects, on the top bar, select the **Issues** or **Merge requests** links. The numbers indicate how many issues, merge requests, and to-do items are assigned to you: @@ -32,24 +32,33 @@ in the search field in the upper right corner: ### Filter issue and merge request lists -> - Filter by Epics was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/195704) in GitLab Ultimate 12.9. -> - Filter by child Epics was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9029) in GitLab Ultimate 13.0. -> - Filter by Iterations was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118742) in GitLab 13.6. Moved to GitLab Premium in 13.9. +> - Filtering by epics was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/195704) in GitLab 12.9. +> - Filtering by child epics was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9029) in GitLab 13.0. +> - Filtering by iterations was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118742) in GitLab 13.6. Moved from GitLab Ultimate to Premium in 13.9. +> - Filtering by type was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322755) in GitLab 13.10 [with a flag](../../administration/feature_flags.md) named `vue_issues_list`. Disabled by default. -Follow these steps to filter the **Issues** and **Merge Requests** list pages in projects and +Follow these steps to filter the **Issues** and **Merge requests** list pages in projects and groups: 1. Click in the field **Search or filter results...**. 1. In the dropdown menu that appears, select the attribute you wish to filter by: - - Author - Assignee - - [Milestone](../project/milestones/index.md) + - Author + - Confidential + - [Epic and child Epic](../group/epics/index.md) (available only for the group the Epic was created, not for [higher group levels](https://gitlab.com/gitlab-org/gitlab/-/issues/233729)). - [Iteration](../group/iterations/index.md) - - Release - [Label](../project/labels.md) + - [Milestone](../project/milestones/index.md) - My-reaction - - Confidential - - [Epic and child Epic](../group/epics/index.md) (available only for the group the Epic was created, not for [higher group levels](https://gitlab.com/gitlab-org/gitlab/-/issues/233729)). + - Release + - Type + + FLAG: + On self-managed GitLab, by default filtering by type is not available. + To make it available per group, ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `vue_issues_list`. + On GitLab.com, this feature is not available. + + - Weight - Search for this text 1. Select or type the operator to use for filtering the attribute. The following operators are available: @@ -95,7 +104,7 @@ You can add this URL to your feed reader. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/39908) in GitLab 12.1. -You can filter the **Issues** list to individual instances by their ID. For example, enter filter `#10` to return only issue 10. The same applies to the **Merge Requests** list. Enter filter `#30` to return only merge request 30. +You can filter the **Issues** list to individual instances by their ID. For example, enter filter `#10` to return only issue 10. The same applies to the **Merge requests** list. Enter filter `#30` to return only merge request 30.  @@ -270,8 +279,13 @@ search, or choose a specific group or project. To search through code or other documents in a single project, you can use the search field on the top-right of your screen while the project page is open. -Code search shows only the first result in the file. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/327052) -in GitLab 14.7, you can access Git blame from any line that returned a result from the code search: +Code search shows only the first result in the file. + +#### Git blame from code search **(PREMIUM)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/327052) in GitLab 14.7. + +You can access Git blame from any line that returned a result from the code search:  diff --git a/doc/user/shortcuts.md b/doc/user/shortcuts.md index d6cbbf352fc..ebaa5da8e05 100644 --- a/doc/user/shortcuts.md +++ b/doc/user/shortcuts.md @@ -1,6 +1,6 @@ --- -stage: none -group: unassigned +stage: Create +group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference disqus_identifier: 'https://docs.gitlab.com/ee/workflow/shortcuts.html' @@ -149,6 +149,57 @@ This shortcut is available when viewing a [wiki page](project/wiki/index.md): |-------------------|-------------| | <kbd>e</kbd> | Edit wiki page. | +### Content editor + +These shortcuts are available when editing a file with the [Content Editor](https://about.gitlab.com/direction/create/editor/content_editor/): + +| Keyboard shortcut | Description | +|-------------------|-------------| +| <kbd>⌘</kbd> + <kbd>C</kbd> (Mac) / <kbd>Control</kbd> + <kbd>C</kbd> | Copy | +| <kbd>⌘</kbd> + <kbd>X</kbd> (Mac) / <kbd>Control</kbd> + <kbd>X</kbd> | Cut | +| <kbd>⌘</kbd> + <kbd>V</kbd> (Mac) / <kbd>Control</kbd> + <kbd>V</kbd> | Paste | +| <kbd>⌘</kbd> + <kbd>Shift</kbd> + <kbd>V</kbd> (Mac) / <kbd>Control</kbd> + <kbd>Shift</kbd> + <kbd>V</kbd> | Paste without formatting | +| <kbd>⌘</kbd> + <kbd>Z</kbd> (Mac) / <kbd>Control</kbd> + <kbd>Z</kbd> | Undo | +| <kbd>⌘</kbd> + <kbd>Shift</kbd> + <kbd>V</kbd> (Mac) / <kbd>Control</kbd> + <kbd>Shift</kbd> + <kbd>V</kbd> | Redo | +| <kbd>Shift</kbd> + <kbd>Enter</kbd> | Add a line break | + +#### Formatting + +| Mac | Windows/Linux | Description | +|-----|---------------|-------------| +| <kbd>⌘</kbd> + <kbd>b</kbd> | <kbd>Control</kbd> + <kbd>b</kbd> | Bold | +| <kbd>⌘</kbd> + <kbd>i</kbd> | <kbd>Control</kbd> + <kbd>i</kbd> | Italic | +| <kbd>⌘</kbd> + <kbd>Shift</kbd> + <kbd>s</kbd> | <kbd>Control</kbd> + <kbd>Shift</kbd> + <kbd>s</kbd> | Strikethrough | +| <kbd>⌘</kbd> + <kbd>e</kbd> | <kbd>Control</kbd> + <kbd>e</kbd> | Code | +| <kbd>⌘</kbd> + <kbd>Alt</kbd> + <kbd>0</kbd> | <kbd>Control</kbd> + <kbd>Alt</kbd> + <kbd>0</kbd> | Apply normal text style | +| <kbd>⌘</kbd> + <kbd>Alt</kbd> + <kbd>1</kbd> | <kbd>Control</kbd> + <kbd>Alt</kbd> + <kbd>1</kbd> | Apply heading style 1 | +| <kbd>⌘</kbd> + <kbd>Alt</kbd> + <kbd>2</kbd> | <kbd>Control</kbd> + <kbd>Alt</kbd> + <kbd>2</kbd> | Apply heading style 2 | +| <kbd>⌘</kbd> + <kbd>Alt</kbd> + <kbd>3</kbd> | <kbd>Control</kbd> + <kbd>Alt</kbd> + <kbd>3</kbd> | Apply heading style 3 | +| <kbd>⌘</kbd> + <kbd>Alt</kbd> + <kbd>4</kbd> | <kbd>Control</kbd> + <kbd>Alt</kbd> + <kbd>4</kbd> | Apply heading style 4 | +| <kbd>⌘</kbd> + <kbd>Alt</kbd> + <kbd>5</kbd> | <kbd>Control</kbd> + <kbd>Alt</kbd> + <kbd>5</kbd> | Apply heading style 5 | +| <kbd>⌘</kbd> + <kbd>Alt</kbd> + <kbd>6</kbd> | <kbd>Control</kbd> + <kbd>Alt</kbd> + <kbd>6</kbd> | Apply heading style 6 | +| <kbd>⌘</kbd> + <kbd>Shift</kbd> + <kbd>7</kbd> | <kbd>Control</kbd> + <kbd>Shift</kbd> + <kbd>7</kbd> | Ordered list | +| <kbd>⌘</kbd> + <kbd>Shift</kbd> + <kbd>8</kbd> | <kbd>Control</kbd> + <kbd>Shift</kbd> + <kbd>7</kbd> | Bullet list | +| <kbd>⌘</kbd> + <kbd>Shift</kbd> + <kbd>9</kbd> | <kbd>Control</kbd> + <kbd>Shift</kbd> + <kbd>7</kbd> | Task list | +| <kbd>⌘</kbd> + <kbd>Shift</kbd> + <kbd>b</kbd> | <kbd>Control</kbd> + <kbd>Shift</kbd> + <kbd>b</kbd> | Blockquote | +| <kbd>⌘</kbd> + <kbd>Alt</kbd> + <kbd>c</kbd> | <kbd>Control</kbd> + <kbd>Shift</kbd> + <kbd>c</kbd> | Code block | +| <kbd>⌘</kbd> + <kbd>,</kbd> | <kbd>Control</kbd> + <kbd>,</kbd> | Subscript | +| <kbd>⌘</kbd> + <kbd>.</kbd> | <kbd>Control</kbd> + <kbd>,</kbd> | Superscript | +| <kbd>Tab</kbd> | | Indent list | +| <kbd>Shift</kbd> + <kbd>Tab</kbd> | | Outdent list | + +#### Text selection + +| Keyboard shortcut | Description | +|-------------------|-------------| +| <kbd>⌘</kbd> + <kbd>a</kbd> (Mac) / <kbd>Control</kbd> + <kbd>a</kbd> | Select all | +| <kbd>Shift</kbd> + <kbd>←</kbd> | Extend selection one character to left | +| <kbd>Shift</kbd> + <kbd>→</kbd> | Extend selection one character to right | +| <kbd>Shift</kbd> + <kbd>↑</kbd> | Extend selection one line up | +| <kbd>Shift</kbd> + <kbd>↓</kbd> | Extend selection one line down | +| <kbd>⌘</kbd> + <kbd>Shift</kbd> + <kbd>↑</kbd> (Mac) / <kbd>Control</kbd> + <kbd>Shift</kbd> + <kbd>↑</kbd> | Extend selection to the beginning of the document | +| <kbd>⌘</kbd> + <kbd>Shift</kbd> + <kbd>↓</kbd> (Mac) / <kbd>Control</kbd> + <kbd>Shift</kbd> + <kbd>↓</kbd> | Extend selection to the end of the document | + ### Filtered search These shortcuts are available when using a [filtered search input](search/index.md): @@ -158,7 +209,7 @@ These shortcuts are available when using a [filtered search input](search/index. | <kbd>⌘</kbd> (Mac) + <kbd>⌫</kbd> | Clear entire search filter. | | <kbd>⌥</kbd> (Mac) / <kbd>Control</kbd> + <kbd>⌫</kbd> | Clear one token at a time. | -## Epics **(ULTIMATE)** +## Epics **(PREMIUM)** These shortcuts are available when viewing [epics](group/epics/index.md): diff --git a/doc/user/upgrade_email_bypass.md b/doc/user/upgrade_email_bypass.md index 3272ff30e72..8cd9a47f3e6 100644 --- a/doc/user/upgrade_email_bypass.md +++ b/doc/user/upgrade_email_bypass.md @@ -58,7 +58,7 @@ User.where(confirmed_at: nil).where('LENGTH(confirmation_token) = 32') ## What does it look like when a user is blocked? -A regular user might receive a message that says "You have to confirm your email address before continuing". This message could includes a 404 or 422 error code, when the user tries to sign in. +A user might receive a message that says "You have to confirm your email address before continuing". This message could includes a 404 or 422 error code, when the user tries to sign in. NOTE: We hope to improve the [sign-in experience for an unverified user](https://gitlab.com/gitlab-org/gitlab/-/issues/29279) in a future release. diff --git a/doc/user/usage_quotas.md b/doc/user/usage_quotas.md index 16b6424b232..75d10084328 100644 --- a/doc/user/usage_quotas.md +++ b/doc/user/usage_quotas.md @@ -49,7 +49,7 @@ The following storage usage statistics are available to an owner: Excess storage usage is the amount that a project's repository exceeds the free storage quota. If no purchased storage is available the project is locked. You cannot push changes to a locked project. -To unlock a project you must [purchase more storage](../subscriptions/gitlab_com/index.md#purchase-more-storage) +To unlock a project you must [purchase more storage](../subscriptions/gitlab_com/index.md#purchase-more-storage-and-transfer) for the namespace. When the purchase is completed, locked projects are automatically unlocked. The amount of purchased storage available must always be greater than zero. diff --git a/doc/user/workspace/img/1.1-Instance_overview.png b/doc/user/workspace/img/1.1-Instance_overview.png Binary files differdeleted file mode 100644 index a3c642cda3f..00000000000 --- a/doc/user/workspace/img/1.1-Instance_overview.png +++ /dev/null diff --git a/doc/user/workspace/img/1.2-Groups_overview.png b/doc/user/workspace/img/1.2-Groups_overview.png Binary files differdeleted file mode 100644 index 7771e5f4c3c..00000000000 --- a/doc/user/workspace/img/1.2-Groups_overview.png +++ /dev/null diff --git a/doc/user/workspace/img/1.3-Admin.png b/doc/user/workspace/img/1.3-Admin.png Binary files differdeleted file mode 100644 index 5f531e4ef5e..00000000000 --- a/doc/user/workspace/img/1.3-Admin.png +++ /dev/null diff --git a/doc/user/workspace/img/Admin_Settings.png b/doc/user/workspace/img/Admin_Settings.png Binary files differdeleted file mode 100644 index b0d13f43ccb..00000000000 --- a/doc/user/workspace/img/Admin_Settings.png +++ /dev/null diff --git a/doc/user/workspace/img/hardware_settings.png b/doc/user/workspace/img/hardware_settings.png Binary files differdeleted file mode 100644 index 919ff46f8c8..00000000000 --- a/doc/user/workspace/img/hardware_settings.png +++ /dev/null diff --git a/doc/user/workspace/index.md b/doc/user/workspace/index.md index cf35f082880..0befcbe25d2 100644 --- a/doc/user/workspace/index.md +++ b/doc/user/workspace/index.md @@ -14,6 +14,9 @@ As with all projects, the items mentioned on this page are subject to change or The development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc. +NOTE: +Workspace is currently in development. + Workspace will be above the [top-level namespaces](../group/index.md#namespaces) for you to manage everything you do as a GitLab administrator, including: @@ -27,33 +30,12 @@ Our goal is to reach feature parity between SaaS and self-managed installations, - Hardware Controls. For functionality that does not apply to groups, Hardware Controls are only applicable to self-managed installations. There is one Hardware Controls section per installation. -NOTE: -Workspace is currently in development. - -## Demo: New hierarchy concept for groups and projects for epics - -The following demo introduces the new hierarchy concept for groups and projects for epics. - -<div class="video-fallback"> - See the video: <a href="https://www.youtube.com/embed/fE74lsG_8yM">Consolidating groups and projects update (2021-08-23)</a>. -</div> -<figure class="video-container"> - <iframe src="https://www.youtube.com/embed/fE74lsG_8yM" frameborder="0" allowfullscreen="true"> </iframe> -</figure> - -## Concept previews - -The following provide a preview to the Workspace concept. - - - - - - - - +To learn about the current state of workspace development, +see [epic 4257](https://gitlab.com/groups/gitlab-org/-/epics/4257). - +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For a video introduction to the new hierarchy concept for groups and projects for epics, see +[Consolidating groups and projects update (August 2021)](https://www.youtube.com/watch?v=fE74lsG_8yM). ## Related topics |
